Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20251025 07:52:20.675590 416 test_util.cc:276] Using random seed: -232315940
2025-10-25T07:52:20Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-25T07:52:20Z Disabled control of system clock
I20251025 07:52:20.709496 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRBHlYt/build/asan/bin/kudu
/tmp/dist-test-taskRBHlYt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:37421
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:40091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:37421,127.0.104.61:42203,127.0.104.60:37611
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251025 07:52:21.046180 424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 07:52:21.046514 424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 07:52:21.046594 424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 07:52:21.054584 424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 07:52:21.054682 424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 07:52:21.054734 424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 07:52:21.054788 424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 07:52:21.065081 424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:40091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:37421,127.0.104.61:42203,127.0.104.60:37611
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:37421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 25 Oct 2025 07:43:22 UTC on 24a791456cd2
build id 8708
ASAN enabled
I20251025 07:52:21.066797 424 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 07:52:21.068574 424 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 07:52:21.081383 430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 07:52:21.081763 432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 07:52:21.081410 429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 07:52:21.083763 424 server_base.cc:1047] running on GCE node
I20251025 07:52:21.084873 424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 07:52:21.086686 424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 07:52:21.087891 424 hybrid_clock.cc:648] HybridClock initialized: now 1761378741087830 us; error 60 us; skew 500 ppm
I20251025 07:52:21.088258 424 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 07:52:21.090505 424 webserver.cc:492] Webserver started at http://127.0.104.62:45315/ using document root <none> and password file <none>
I20251025 07:52:21.091251 424 fs_manager.cc:362] Metadata directory not provided
I20251025 07:52:21.091356 424 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 07:52:21.091722 424 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 07:52:21.094045 424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-0/data/instance:
uuid: "b2dfbd0b44a441c9be9254f660974944"
format_stamp: "Formatted at 2025-10-25 07:52:21 on dist-test-slave-tm48"
I20251025 07:52:21.094873 424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-0/wal/instance:
uuid: "b2dfbd0b44a441c9be9254f660974944"
format_stamp: "Formatted at 2025-10-25 07:52:21 on dist-test-slave-tm48"
I20251025 07:52:21.100737 424 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20251025 07:52:21.104040 438 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 07:52:21.105383 424 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20251025 07:52:21.105538 424 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-0/wal
uuid: "b2dfbd0b44a441c9be9254f660974944"
format_stamp: "Formatted at 2025-10-25 07:52:21 on dist-test-slave-tm48"
I20251025 07:52:21.105726 424 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 07:52:21.119027 424 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 07:52:21.119858 424 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 07:52:21.120122 424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 07:52:21.142709 424 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:37421
I20251025 07:52:21.142793 489 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:37421 every 8 connection(s)
I20251025 07:52:21.144367 424 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-0/data/info.pb
I20251025 07:52:21.146481 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRBHlYt/build/asan/bin/kudu as pid 424
I20251025 07:52:21.146816 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-0/wal/instance
I20251025 07:52:21.150084 490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:21.150621 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRBHlYt/build/asan/bin/kudu
/tmp/dist-test-taskRBHlYt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:42203
--webserver_interface=127.0.104.61
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:40091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:37421,127.0.104.61:42203,127.0.104.60:37611
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251025 07:52:21.156556 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } has no permanent_uuid. Determining permanent_uuid...
I20251025 07:52:21.176103 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } has no permanent_uuid. Determining permanent_uuid...
W20251025 07:52:21.177110 439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:42203: connect: Connection refused (error 111)
W20251025 07:52:21.178634 490 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:42203: Network error: Client connection negotiation failed: client connection to 127.0.104.61:42203: connect: Connection refused (error 111)
I20251025 07:52:21.228020 490 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } attempt: 1
W20251025 07:52:21.229966 490 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:42203: Network error: Client connection negotiation failed: client connection to 127.0.104.61:42203: connect: Connection refused (error 111)
I20251025 07:52:21.298231 490 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } attempt: 2
W20251025 07:52:21.300204 490 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:42203: Network error: Client connection negotiation failed: client connection to 127.0.104.61:42203: connect: Connection refused (error 111)
I20251025 07:52:21.391486 490 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } attempt: 3
W20251025 07:52:21.393318 490 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:42203: Network error: Client connection negotiation failed: client connection to 127.0.104.61:42203: connect: Connection refused (error 111)
W20251025 07:52:21.499538 492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 07:52:21.499852 492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 07:52:21.499984 492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 07:52:21.508061 492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 07:52:21.508157 492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 07:52:21.508224 492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 07:52:21.508291 492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 07:52:21.518662 492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:40091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:37421,127.0.104.61:42203,127.0.104.60:37611
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:42203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 25 Oct 2025 07:43:22 UTC on 24a791456cd2
build id 8708
ASAN enabled
I20251025 07:52:21.520428 492 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 07:52:21.522188 492 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 07:52:21.530572 501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 07:52:21.530761 500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 07:52:21.530682 503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 07:52:21.531901 492 server_base.cc:1047] running on GCE node
I20251025 07:52:21.532614 492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 07:52:21.533936 492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 07:52:21.535130 492 hybrid_clock.cc:648] HybridClock initialized: now 1761378741535070 us; error 53 us; skew 500 ppm
I20251025 07:52:21.535468 492 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 07:52:21.536549 490 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } attempt: 4
I20251025 07:52:21.537866 492 webserver.cc:492] Webserver started at http://127.0.104.61:45311/ using document root <none> and password file <none>
W20251025 07:52:21.538424 490 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:42203: Network error: Client connection negotiation failed: client connection to 127.0.104.61:42203: connect: Connection refused (error 111)
I20251025 07:52:21.538782 492 fs_manager.cc:362] Metadata directory not provided
I20251025 07:52:21.538877 492 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 07:52:21.539177 492 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 07:52:21.541388 492 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/data/instance:
uuid: "8603bddd8a084ff7a73a25641768dfa9"
format_stamp: "Formatted at 2025-10-25 07:52:21 on dist-test-slave-tm48"
I20251025 07:52:21.542063 492 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/wal/instance:
uuid: "8603bddd8a084ff7a73a25641768dfa9"
format_stamp: "Formatted at 2025-10-25 07:52:21 on dist-test-slave-tm48"
I20251025 07:52:21.547113 492 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20251025 07:52:21.550369 509 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 07:52:21.551620 492 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20251025 07:52:21.551770 492 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/data,/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/wal
uuid: "8603bddd8a084ff7a73a25641768dfa9"
format_stamp: "Formatted at 2025-10-25 07:52:21 on dist-test-slave-tm48"
I20251025 07:52:21.551946 492 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 07:52:21.572522 492 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 07:52:21.573434 492 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 07:52:21.573709 492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 07:52:21.596354 492 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:42203
I20251025 07:52:21.598711 492 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/data/info.pb
I20251025 07:52:21.599053 560 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:42203 every 8 connection(s)
I20251025 07:52:21.601496 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRBHlYt/build/asan/bin/kudu as pid 492
I20251025 07:52:21.601686 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/wal/instance
I20251025 07:52:21.603277 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRBHlYt/build/asan/bin/kudu
/tmp/dist-test-taskRBHlYt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:37611
--webserver_interface=127.0.104.60
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:40091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:37421,127.0.104.61:42203,127.0.104.60:37611
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251025 07:52:21.605057 561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:21.614732 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } has no permanent_uuid. Determining permanent_uuid...
I20251025 07:52:21.630923 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } has no permanent_uuid. Determining permanent_uuid...
I20251025 07:52:21.641259 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } has no permanent_uuid. Determining permanent_uuid...
W20251025 07:52:21.642367 511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:37611: connect: Connection refused (error 111)
W20251025 07:52:21.643826 561 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:37611: Network error: Client connection negotiation failed: client connection to 127.0.104.60:37611: connect: Connection refused (error 111)
I20251025 07:52:21.693107 561 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } attempt: 1
W20251025 07:52:21.694908 561 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:37611: Network error: Client connection negotiation failed: client connection to 127.0.104.60:37611: connect: Connection refused (error 111)
I20251025 07:52:21.763147 561 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } attempt: 2
W20251025 07:52:21.764843 561 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:37611: Network error: Client connection negotiation failed: client connection to 127.0.104.60:37611: connect: Connection refused (error 111)
I20251025 07:52:21.837723 490 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } attempt: 5
I20251025 07:52:21.844198 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } has no permanent_uuid. Determining permanent_uuid...
W20251025 07:52:21.846031 490 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:37611: Network error: Client connection negotiation failed: client connection to 127.0.104.60:37611: connect: Connection refused (error 111)
I20251025 07:52:21.856071 561 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } attempt: 3
W20251025 07:52:21.857812 561 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:37611: Network error: Client connection negotiation failed: client connection to 127.0.104.60:37611: connect: Connection refused (error 111)
I20251025 07:52:21.897325 490 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } attempt: 1
W20251025 07:52:21.899225 490 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:37611: Network error: Client connection negotiation failed: client connection to 127.0.104.60:37611: connect: Connection refused (error 111)
W20251025 07:52:21.957651 563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 07:52:21.957999 563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 07:52:21.958158 563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 07:52:21.966856 563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 07:52:21.966959 563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 07:52:21.967020 563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 07:52:21.967089 563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 07:52:21.967448 490 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } attempt: 2
W20251025 07:52:21.969321 490 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:37611: Network error: Client connection negotiation failed: client connection to 127.0.104.60:37611: connect: Connection refused (error 111)
I20251025 07:52:21.977245 563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:40091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:37421,127.0.104.61:42203,127.0.104.60:37611
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:37611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 25 Oct 2025 07:43:22 UTC on 24a791456cd2
build id 8708
ASAN enabled
I20251025 07:52:21.979389 563 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 07:52:21.981217 563 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 07:52:21.989853 571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 07:52:21.989784 572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 07:52:21.990275 574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 07:52:21.991082 563 server_base.cc:1047] running on GCE node
I20251025 07:52:21.991840 563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 07:52:21.993224 563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 07:52:21.994451 563 hybrid_clock.cc:648] HybridClock initialized: now 1761378741994399 us; error 40 us; skew 500 ppm
I20251025 07:52:21.994832 563 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 07:52:21.997548 563 webserver.cc:492] Webserver started at http://127.0.104.60:38531/ using document root <none> and password file <none>
I20251025 07:52:21.998337 563 fs_manager.cc:362] Metadata directory not provided
I20251025 07:52:21.998448 563 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 07:52:21.998797 563 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 07:52:22.001049 561 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } attempt: 4
I20251025 07:52:22.001240 563 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-2/data/instance:
uuid: "b4a999d754c941ec850c143e4e24c407"
format_stamp: "Formatted at 2025-10-25 07:52:21 on dist-test-slave-tm48"
I20251025 07:52:22.002197 563 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-2/wal/instance:
uuid: "b4a999d754c941ec850c143e4e24c407"
format_stamp: "Formatted at 2025-10-25 07:52:21 on dist-test-slave-tm48"
W20251025 07:52:22.002897 561 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:37611: Network error: Client connection negotiation failed: client connection to 127.0.104.60:37611: connect: Connection refused (error 111)
I20251025 07:52:22.007556 563 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20251025 07:52:22.010875 580 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 07:52:22.012224 563 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251025 07:52:22.012362 563 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-2/data,/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-2/wal
uuid: "b4a999d754c941ec850c143e4e24c407"
format_stamp: "Formatted at 2025-10-25 07:52:21 on dist-test-slave-tm48"
I20251025 07:52:22.012550 563 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 07:52:22.035686 563 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 07:52:22.036559 563 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 07:52:22.036790 563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 07:52:22.057662 563 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:37611
I20251025 07:52:22.057696 631 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:37611 every 8 connection(s)
I20251025 07:52:22.059214 563 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-2/data/info.pb
I20251025 07:52:22.062422 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRBHlYt/build/asan/bin/kudu as pid 563
I20251025 07:52:22.062597 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-2/wal/instance
I20251025 07:52:22.063366 632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:22.069393 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } has no permanent_uuid. Determining permanent_uuid...
I20251025 07:52:22.075555 490 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } attempt: 3
I20251025 07:52:22.083658 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } has no permanent_uuid. Determining permanent_uuid...
I20251025 07:52:22.091895 490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Bootstrap starting.
I20251025 07:52:22.093408 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } has no permanent_uuid. Determining permanent_uuid...
I20251025 07:52:22.096561 490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:22.097913 490 log.cc:826] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Log is configured to *not* fsync() on all Append() calls
I20251025 07:52:22.101226 490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: No bootstrap required, opened a new log
I20251025 07:52:22.104126 632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407: Bootstrap starting.
I20251025 07:52:22.107473 632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:22.107421 490 raft_consensus.cc:359] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } }
I20251025 07:52:22.107748 490 raft_consensus.cc:385] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:22.107861 490 raft_consensus.cc:740] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2dfbd0b44a441c9be9254f660974944, State: Initialized, Role: FOLLOWER
I20251025 07:52:22.108530 632 log.cc:826] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407: Log is configured to *not* fsync() on all Append() calls
I20251025 07:52:22.108448 490 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } }
I20251025 07:52:22.109346 641 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } } }
I20251025 07:52:22.109606 641 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 07:52:22.110530 490 sys_catalog.cc:565] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 07:52:22.110976 632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407: No bootstrap required, opened a new log
I20251025 07:52:22.116909 632 raft_consensus.cc:359] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } }
I20251025 07:52:22.117453 632 raft_consensus.cc:385] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:22.117573 632 raft_consensus.cc:740] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4a999d754c941ec850c143e4e24c407, State: Initialized, Role: FOLLOWER
I20251025 07:52:22.118316 632 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } }
W20251025 07:52:22.123210 444 tablet.cc:2378] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251025 07:52:22.126083 653 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251025 07:52:22.126372 653 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251025 07:52:22.120764 424 master_runner.cc:186] Error getting master registration for 127.0.104.61:42203: OK, instance_id { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" instance_seqno: 1761378741590039 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251025 07:52:22.127149 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 07:52:22.127182 654 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } } }
I20251025 07:52:22.128126 654 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 07:52:22.128538 632 sys_catalog.cc:565] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [sys.catalog]: configured and running, proceeding with master startup.
W20251025 07:52:22.144433 666 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251025 07:52:22.144577 666 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251025 07:52:22.140764 563 master_runner.cc:186] Error getting master registration for 127.0.104.61:42203: OK, instance_id { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" instance_seqno: 1761378741590039 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251025 07:52:22.149955 563 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251025 07:52:22.288838 586 tablet.cc:2378] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251025 07:52:22.302115 561 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } attempt: 5
I20251025 07:52:22.316257 561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Bootstrap starting.
I20251025 07:52:22.320483 561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:22.322435 561 log.cc:826] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Log is configured to *not* fsync() on all Append() calls
I20251025 07:52:22.325691 561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: No bootstrap required, opened a new log
I20251025 07:52:22.334163 561 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } }
I20251025 07:52:22.334511 561 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:22.334623 561 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8603bddd8a084ff7a73a25641768dfa9, State: Initialized, Role: FOLLOWER
I20251025 07:52:22.335422 561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } }
I20251025 07:52:22.337356 669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } } }
I20251025 07:52:22.337704 669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 07:52:22.338723 561 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 07:52:22.364403 492 master_runner.cc:263] No leader master found from master 8603bddd8a084ff7a73a25641768dfa9
I20251025 07:52:22.364555 492 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251025 07:52:22.367467 680 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251025 07:52:22.367599 680 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251025 07:52:22.391920 669 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:22.392230 669 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } }
I20251025 07:52:22.394055 669 leader_election.cc:290] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2dfbd0b44a441c9be9254f660974944 (127.0.104.62:37421), b4a999d754c941ec850c143e4e24c407 (127.0.104.60:37611)
I20251025 07:52:22.395061 465 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8603bddd8a084ff7a73a25641768dfa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2dfbd0b44a441c9be9254f660974944" is_pre_election: true
I20251025 07:52:22.395673 465 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8603bddd8a084ff7a73a25641768dfa9 in term 0.
I20251025 07:52:22.394979 607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8603bddd8a084ff7a73a25641768dfa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4a999d754c941ec850c143e4e24c407" is_pre_election: true
I20251025 07:52:22.396054 607 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8603bddd8a084ff7a73a25641768dfa9 in term 0.
I20251025 07:52:22.396318 513 leader_election.cc:304] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8603bddd8a084ff7a73a25641768dfa9, b2dfbd0b44a441c9be9254f660974944; no voters:
I20251025 07:52:22.396763 669 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:22.396905 669 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:22.396992 669 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:22.400377 669 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } }
I20251025 07:52:22.401680 465 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8603bddd8a084ff7a73a25641768dfa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2dfbd0b44a441c9be9254f660974944"
I20251025 07:52:22.401887 465 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:22.403414 669 leader_election.cc:290] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [CANDIDATE]: Term 1 election: Requested vote from peers b2dfbd0b44a441c9be9254f660974944 (127.0.104.62:37421), b4a999d754c941ec850c143e4e24c407 (127.0.104.60:37611)
I20251025 07:52:22.404028 607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8603bddd8a084ff7a73a25641768dfa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4a999d754c941ec850c143e4e24c407"
I20251025 07:52:22.404245 607 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:22.406210 465 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8603bddd8a084ff7a73a25641768dfa9 in term 1.
I20251025 07:52:22.406714 513 leader_election.cc:304] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8603bddd8a084ff7a73a25641768dfa9, b2dfbd0b44a441c9be9254f660974944; no voters:
I20251025 07:52:22.407016 669 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:22.407843 607 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8603bddd8a084ff7a73a25641768dfa9 in term 1.
I20251025 07:52:22.420302 669 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [term 1 LEADER]: Becoming Leader. State: Replica: 8603bddd8a084ff7a73a25641768dfa9, State: Running, Role: LEADER
I20251025 07:52:22.421010 669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } }
I20251025 07:52:22.432991 682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8603bddd8a084ff7a73a25641768dfa9. Latest consensus state: current_term: 1 leader_uuid: "8603bddd8a084ff7a73a25641768dfa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } } }
I20251025 07:52:22.433346 682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [sys.catalog]: This master's current role is: LEADER
I20251025 07:52:22.443212 684 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 07:52:22.445549 684 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 07:52:22.465764 607 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [term 1 FOLLOWER]: Refusing update from remote peer 8603bddd8a084ff7a73a25641768dfa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 07:52:22.467077 465 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [term 1 FOLLOWER]: Refusing update from remote peer 8603bddd8a084ff7a73a25641768dfa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 07:52:22.467306 682 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:22.468006 682 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:22.495960 641 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8603bddd8a084ff7a73a25641768dfa9. Latest consensus state: current_term: 1 leader_uuid: "8603bddd8a084ff7a73a25641768dfa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } } }
I20251025 07:52:22.496251 641 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 07:52:22.499676 685 mvcc.cc:204] Tried to move back new op lower bound from 7214607329127354368 to 7214607328962772992. Current Snapshot: MvccSnapshot[applied={T|T < 7214607329127354368}]
I20251025 07:52:22.501341 686 mvcc.cc:204] Tried to move back new op lower bound from 7214607329127354368 to 7214607328962772992. Current Snapshot: MvccSnapshot[applied={T|T < 7214607329127354368}]
I20251025 07:52:22.506644 682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8603bddd8a084ff7a73a25641768dfa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } } }
I20251025 07:52:22.506870 682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [sys.catalog]: This master's current role is: LEADER
I20251025 07:52:22.499102 654 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8603bddd8a084ff7a73a25641768dfa9. Latest consensus state: current_term: 1 leader_uuid: "8603bddd8a084ff7a73a25641768dfa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } } }
I20251025 07:52:22.508119 654 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 07:52:22.509358 641 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8603bddd8a084ff7a73a25641768dfa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } } }
I20251025 07:52:22.509567 641 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 07:52:22.510535 684 catalog_manager.cc:1357] Generated new cluster ID: aee99068aba246df9094aaec3350d6ff
I20251025 07:52:22.510630 684 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 07:52:22.524813 669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8603bddd8a084ff7a73a25641768dfa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } } }
I20251025 07:52:22.525095 669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [sys.catalog]: This master's current role is: LEADER
I20251025 07:52:22.539142 654 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8603bddd8a084ff7a73a25641768dfa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } } }
I20251025 07:52:22.539348 654 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 07:52:22.581104 684 catalog_manager.cc:1380] Generated new certificate authority record
I20251025 07:52:22.582943 684 catalog_manager.cc:1514] Loading token signing keys...
I20251025 07:52:22.643193 684 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Generated new TSK 0
I20251025 07:52:22.644304 684 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251025 07:52:22.662302 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRBHlYt/build/asan/bin/kudu
/tmp/dist-test-taskRBHlYt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:37421,127.0.104.61:42203,127.0.104.60:37611
--builtin_ntp_servers=127.0.104.20:40091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
W20251025 07:52:23.005769 694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 07:52:23.006078 694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 07:52:23.006239 694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 07:52:23.014946 694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 07:52:23.015159 694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251025 07:52:23.025197 694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:40091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:37421,127.0.104.61:42203,127.0.104.60:37611
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 25 Oct 2025 07:43:22 UTC on 24a791456cd2
build id 8708
ASAN enabled
I20251025 07:52:23.026875 694 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 07:52:23.028712 694 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 07:52:23.037899 700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 07:52:23.037860 699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 07:52:23.038587 702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 07:52:23.039172 694 server_base.cc:1047] running on GCE node
I20251025 07:52:23.039857 694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 07:52:23.040844 694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 07:52:23.042073 694 hybrid_clock.cc:648] HybridClock initialized: now 1761378743042015 us; error 49 us; skew 500 ppm
I20251025 07:52:23.042402 694 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 07:52:23.044992 694 webserver.cc:492] Webserver started at http://127.0.104.1:36611/ using document root <none> and password file <none>
I20251025 07:52:23.045722 694 fs_manager.cc:362] Metadata directory not provided
I20251025 07:52:23.045823 694 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 07:52:23.046216 694 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 07:52:23.048594 694 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/data/instance:
uuid: "23eeffd3de254ce18db5febfde971c6b"
format_stamp: "Formatted at 2025-10-25 07:52:23 on dist-test-slave-tm48"
I20251025 07:52:23.049291 694 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/instance:
uuid: "23eeffd3de254ce18db5febfde971c6b"
format_stamp: "Formatted at 2025-10-25 07:52:23 on dist-test-slave-tm48"
I20251025 07:52:23.055099 694 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20251025 07:52:23.058612 708 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 07:52:23.059964 694 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20251025 07:52:23.060102 694 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal
uuid: "23eeffd3de254ce18db5febfde971c6b"
format_stamp: "Formatted at 2025-10-25 07:52:23 on dist-test-slave-tm48"
I20251025 07:52:23.060292 694 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 07:52:23.089937 694 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 07:52:23.090927 694 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 07:52:23.091207 694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 07:52:23.092206 694 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 07:52:23.094426 694 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 07:52:23.095127 694 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 07:52:23.095253 694 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 07:52:23.095329 694 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 07:52:23.128543 653 catalog_manager.cc:1269] Loaded cluster ID: aee99068aba246df9094aaec3350d6ff
I20251025 07:52:23.128665 653 catalog_manager.cc:1562] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: loading cluster ID for follower catalog manager: success
I20251025 07:52:23.132157 653 catalog_manager.cc:1584] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: acquiring CA information for follower catalog manager: success
I20251025 07:52:23.133226 694 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:46231
I20251025 07:52:23.133289 820 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:46231 every 8 connection(s)
I20251025 07:52:23.134258 653 catalog_manager.cc:1612] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251025 07:52:23.134722 694 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/data/info.pb
I20251025 07:52:23.142416 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRBHlYt/build/asan/bin/kudu as pid 694
I20251025 07:52:23.142649 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/instance
I20251025 07:52:23.146767 666 catalog_manager.cc:1269] Loaded cluster ID: aee99068aba246df9094aaec3350d6ff
I20251025 07:52:23.146875 666 catalog_manager.cc:1562] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407: loading cluster ID for follower catalog manager: success
I20251025 07:52:23.148159 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRBHlYt/build/asan/bin/kudu
/tmp/dist-test-taskRBHlYt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:37421,127.0.104.61:42203,127.0.104.60:37611
--builtin_ntp_servers=127.0.104.20:40091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251025 07:52:23.150755 666 catalog_manager.cc:1584] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407: acquiring CA information for follower catalog manager: success
I20251025 07:52:23.153148 666 catalog_manager.cc:1612] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251025 07:52:23.160840 831 raft_consensus.cc:493] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8603bddd8a084ff7a73a25641768dfa9)
I20251025 07:52:23.161123 831 raft_consensus.cc:515] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } }
I20251025 07:52:23.163336 821 heartbeater.cc:344] Connected to a master server at 127.0.104.60:37611
I20251025 07:52:23.163448 823 heartbeater.cc:344] Connected to a master server at 127.0.104.61:42203
I20251025 07:52:23.163775 823 heartbeater.cc:461] Registering TS with master...
I20251025 07:52:23.163782 821 heartbeater.cc:461] Registering TS with master...
I20251025 07:52:23.165046 821 heartbeater.cc:507] Master 127.0.104.60:37611 requested a full tablet report, sending...
I20251025 07:52:23.165019 823 heartbeater.cc:507] Master 127.0.104.61:42203 requested a full tablet report, sending...
I20251025 07:52:23.166697 831 leader_election.cc:290] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8603bddd8a084ff7a73a25641768dfa9 (127.0.104.61:42203), b4a999d754c941ec850c143e4e24c407 (127.0.104.60:37611)
I20251025 07:52:23.167224 607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2dfbd0b44a441c9be9254f660974944" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b4a999d754c941ec850c143e4e24c407" is_pre_election: true
I20251025 07:52:23.167342 536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2dfbd0b44a441c9be9254f660974944" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8603bddd8a084ff7a73a25641768dfa9" is_pre_election: true
I20251025 07:52:23.167582 526 ts_manager.cc:194] Registered new tserver with Master: 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:23.167811 597 ts_manager.cc:194] Registered new tserver with Master: 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:23.169111 440 leader_election.cc:304] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b2dfbd0b44a441c9be9254f660974944; no voters: 8603bddd8a084ff7a73a25641768dfa9, b4a999d754c941ec850c143e4e24c407
I20251025 07:52:23.169627 831 raft_consensus.cc:2749] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 07:52:23.169701 526 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:35633
I20251025 07:52:23.171240 822 heartbeater.cc:344] Connected to a master server at 127.0.104.62:37421
I20251025 07:52:23.171415 822 heartbeater.cc:461] Registering TS with master...
I20251025 07:52:23.171926 822 heartbeater.cc:507] Master 127.0.104.62:37421 requested a full tablet report, sending...
I20251025 07:52:23.173662 454 ts_manager.cc:194] Registered new tserver with Master: 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
W20251025 07:52:23.499990 832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 07:52:23.500322 832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 07:52:23.500464 832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 07:52:23.508846 832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 07:52:23.509065 832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251025 07:52:23.519285 832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:40091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:37421,127.0.104.61:42203,127.0.104.60:37611
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 25 Oct 2025 07:43:22 UTC on 24a791456cd2
build id 8708
ASAN enabled
I20251025 07:52:23.520902 832 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 07:52:23.522632 832 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 07:52:23.531934 837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 07:52:23.531934 838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 07:52:23.533170 840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 07:52:23.533485 832 server_base.cc:1047] running on GCE node
I20251025 07:52:23.533991 832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 07:52:23.534876 832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 07:52:23.536070 832 hybrid_clock.cc:648] HybridClock initialized: now 1761378743535999 us; error 63 us; skew 500 ppm
I20251025 07:52:23.536345 832 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 07:52:23.538367 832 webserver.cc:492] Webserver started at http://127.0.104.2:36725/ using document root <none> and password file <none>
I20251025 07:52:23.539064 832 fs_manager.cc:362] Metadata directory not provided
I20251025 07:52:23.539160 832 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 07:52:23.539486 832 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 07:52:23.541786 832 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/data/instance:
uuid: "3aea2869ff814cc7b142012faa274382"
format_stamp: "Formatted at 2025-10-25 07:52:23 on dist-test-slave-tm48"
I20251025 07:52:23.542526 832 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/instance:
uuid: "3aea2869ff814cc7b142012faa274382"
format_stamp: "Formatted at 2025-10-25 07:52:23 on dist-test-slave-tm48"
I20251025 07:52:23.547843 832 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20251025 07:52:23.551137 846 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 07:52:23.552469 832 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20251025 07:52:23.552613 832 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal
uuid: "3aea2869ff814cc7b142012faa274382"
format_stamp: "Formatted at 2025-10-25 07:52:23 on dist-test-slave-tm48"
I20251025 07:52:23.552802 832 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 07:52:23.586604 832 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 07:52:23.587486 832 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 07:52:23.587744 832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 07:52:23.588676 832 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 07:52:23.590360 832 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 07:52:23.590461 832 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 07:52:23.590566 832 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 07:52:23.590639 832 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 07:52:23.637017 832 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:43563
I20251025 07:52:23.637071 958 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:43563 every 8 connection(s)
I20251025 07:52:23.638496 832 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/data/info.pb
I20251025 07:52:23.645578 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRBHlYt/build/asan/bin/kudu as pid 832
I20251025 07:52:23.645792 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/instance
I20251025 07:52:23.650833 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRBHlYt/build/asan/bin/kudu
/tmp/dist-test-taskRBHlYt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:37421,127.0.104.61:42203,127.0.104.60:37611
--builtin_ntp_servers=127.0.104.20:40091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251025 07:52:23.657406 959 heartbeater.cc:344] Connected to a master server at 127.0.104.60:37611
I20251025 07:52:23.657368 961 heartbeater.cc:344] Connected to a master server at 127.0.104.61:42203
I20251025 07:52:23.657475 960 heartbeater.cc:344] Connected to a master server at 127.0.104.62:37421
I20251025 07:52:23.657853 959 heartbeater.cc:461] Registering TS with master...
I20251025 07:52:23.657850 960 heartbeater.cc:461] Registering TS with master...
I20251025 07:52:23.657850 961 heartbeater.cc:461] Registering TS with master...
I20251025 07:52:23.659098 961 heartbeater.cc:507] Master 127.0.104.61:42203 requested a full tablet report, sending...
I20251025 07:52:23.659185 960 heartbeater.cc:507] Master 127.0.104.62:37421 requested a full tablet report, sending...
I20251025 07:52:23.659144 959 heartbeater.cc:507] Master 127.0.104.60:37611 requested a full tablet report, sending...
I20251025 07:52:23.660746 526 ts_manager.cc:194] Registered new tserver with Master: 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:23.660924 454 ts_manager.cc:194] Registered new tserver with Master: 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:23.661778 526 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:59239
I20251025 07:52:23.661893 597 ts_manager.cc:194] Registered new tserver with Master: 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:23.672721 823 heartbeater.cc:499] Master 127.0.104.61:42203 was elected leader, sending a full tablet report...
I20251025 07:52:23.703011 967 raft_consensus.cc:493] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8603bddd8a084ff7a73a25641768dfa9)
I20251025 07:52:23.703276 967 raft_consensus.cc:515] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } }
I20251025 07:52:23.704111 967 leader_election.cc:290] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8603bddd8a084ff7a73a25641768dfa9 (127.0.104.61:42203), b4a999d754c941ec850c143e4e24c407 (127.0.104.60:37611)
I20251025 07:52:23.704625 607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2dfbd0b44a441c9be9254f660974944" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b4a999d754c941ec850c143e4e24c407" is_pre_election: true
I20251025 07:52:23.704651 536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2dfbd0b44a441c9be9254f660974944" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8603bddd8a084ff7a73a25641768dfa9" is_pre_election: true
I20251025 07:52:23.705366 439 leader_election.cc:304] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b2dfbd0b44a441c9be9254f660974944; no voters: 8603bddd8a084ff7a73a25641768dfa9, b4a999d754c941ec850c143e4e24c407
I20251025 07:52:23.705668 967 raft_consensus.cc:2749] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251025 07:52:23.995748 966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 07:52:23.996060 966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 07:52:23.996201 966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 07:52:24.004762 966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 07:52:24.004966 966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251025 07:52:24.015765 966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:40091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:37421,127.0.104.61:42203,127.0.104.60:37611
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 25 Oct 2025 07:43:22 UTC on 24a791456cd2
build id 8708
ASAN enabled
I20251025 07:52:24.017493 966 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 07:52:24.019425 966 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251025 07:52:24.028978 973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 07:52:24.028999 972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 07:52:24.030742 975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 07:52:24.031026 966 server_base.cc:1047] running on GCE node
I20251025 07:52:24.031575 966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 07:52:24.032459 966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 07:52:24.033664 966 hybrid_clock.cc:648] HybridClock initialized: now 1761378744033617 us; error 43 us; skew 500 ppm
I20251025 07:52:24.033983 966 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 07:52:24.036199 966 webserver.cc:492] Webserver started at http://127.0.104.3:40599/ using document root <none> and password file <none>
I20251025 07:52:24.036926 966 fs_manager.cc:362] Metadata directory not provided
I20251025 07:52:24.037029 966 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 07:52:24.037384 966 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251025 07:52:24.039801 966 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/data/instance:
uuid: "db42be7ad3c741a0aa427f27e245993b"
format_stamp: "Formatted at 2025-10-25 07:52:24 on dist-test-slave-tm48"
I20251025 07:52:24.040570 966 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/instance:
uuid: "db42be7ad3c741a0aa427f27e245993b"
format_stamp: "Formatted at 2025-10-25 07:52:24 on dist-test-slave-tm48"
I20251025 07:52:24.045818 966 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20251025 07:52:24.049021 981 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 07:52:24.050319 966 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20251025 07:52:24.050480 966 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal
uuid: "db42be7ad3c741a0aa427f27e245993b"
format_stamp: "Formatted at 2025-10-25 07:52:24 on dist-test-slave-tm48"
I20251025 07:52:24.050658 966 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 07:52:24.065048 966 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 07:52:24.065877 966 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 07:52:24.066154 966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 07:52:24.067021 966 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251025 07:52:24.068611 966 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251025 07:52:24.068712 966 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251025 07:52:24.068842 966 ts_tablet_manager.cc:616] Registered 0 tablets
I20251025 07:52:24.068913 966 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251025 07:52:24.106194 966 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:45417
I20251025 07:52:24.106258 1093 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:45417 every 8 connection(s)
I20251025 07:52:24.107583 966 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/data/info.pb
I20251025 07:52:24.112564 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRBHlYt/build/asan/bin/kudu as pid 966
I20251025 07:52:24.112795 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/instance
I20251025 07:52:24.128758 1096 heartbeater.cc:344] Connected to a master server at 127.0.104.61:42203
I20251025 07:52:24.129191 1096 heartbeater.cc:461] Registering TS with master...
I20251025 07:52:24.130465 1096 heartbeater.cc:507] Master 127.0.104.61:42203 requested a full tablet report, sending...
I20251025 07:52:24.132623 526 ts_manager.cc:194] Registered new tserver with Master: db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:24.134096 526 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:33577
I20251025 07:52:24.134315 1095 heartbeater.cc:344] Connected to a master server at 127.0.104.62:37421
I20251025 07:52:24.134469 1095 heartbeater.cc:461] Registering TS with master...
I20251025 07:52:24.135013 1095 heartbeater.cc:507] Master 127.0.104.62:37421 requested a full tablet report, sending...
I20251025 07:52:24.136328 454 ts_manager.cc:194] Registered new tserver with Master: db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:24.138351 1094 heartbeater.cc:344] Connected to a master server at 127.0.104.60:37611
I20251025 07:52:24.138458 1094 heartbeater.cc:461] Registering TS with master...
I20251025 07:52:24.138806 1094 heartbeater.cc:507] Master 127.0.104.60:37611 requested a full tablet report, sending...
I20251025 07:52:24.139931 596 ts_manager.cc:194] Registered new tserver with Master: db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:24.153717 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251025 07:52:24.163890 961 heartbeater.cc:499] Master 127.0.104.61:42203 was elected leader, sending a full tablet report...
I20251025 07:52:24.189155 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_eb8bbd390e29466e875081ce562d1bf8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:24.191735 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_5abeba9d640f40509583df01dc62749c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:24.191196 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_ae419ef915754fb2a3b38e861f3a865a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:24.192409 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_ab346bb1a526451a9c25ed99a62ed870"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:24.192699 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_850337fc1e7d447d961162f0510838bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:24.190459 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_050e4eed437740489b6a1c7553677f69"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:24.190371 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_dbcea20d468d400f908f0a8f46d94597"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:24.192147 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_e1ca1209c59f456bb4b1ed1b3729f684"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:24.197057 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5abeba9d640f40509583df01dc62749c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:24.197057 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb8bbd390e29466e875081ce562d1bf8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:24.197059 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_050e4eed437740489b6a1c7553677f69 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:24.189042 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_dfba56557b4e42b99167a77ea28e2c7c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:24.198832 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dfba56557b4e42b99167a77ea28e2c7c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:24.197835 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dbcea20d468d400f908f0a8f46d94597 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:24.189509 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_4ef8509650004dc9a3097f1d5004e03b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:24.200994 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_850337fc1e7d447d961162f0510838bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:24.202826 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae419ef915754fb2a3b38e861f3a865a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:24.201009 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab346bb1a526451a9c25ed99a62ed870 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:24.203680 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1ca1209c59f456bb4b1ed1b3729f684 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:24.203992 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ef8509650004dc9a3097f1d5004e03b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:24.239617 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_eb8bbd390e29466e875081ce562d1bf8" } new_table_name: "default.Table_33eaf5d540914e43aad9bc6e22eb698a" modify_external_catalogs: true
I20251025 07:52:24.267230 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskRBHlYt/build/asan/bin/kudu pid 563
I20251025 07:52:24.268810 627 generic_service.cc:196] Checking for leaks (request via RPC)
I20251025 07:52:24.279765 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_ae419ef915754fb2a3b38e861f3a865a" } modify_external_catalogs: true
I20251025 07:52:24.279901 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_ab346bb1a526451a9c25ed99a62ed870" } modify_external_catalogs: true
I20251025 07:52:24.280021 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_050e4eed437740489b6a1c7553677f69" } new_table_name: "default.Table_485ce9a08de848209ce36bf5eb89ac0f" modify_external_catalogs: true
I20251025 07:52:24.280125 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_ab346bb1a526451a9c25ed99a62ed870" } modify_external_catalogs: true
I20251025 07:52:24.280295 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_dbcea20d468d400f908f0a8f46d94597" } new_table_name: "default.Table_e29bb32f37634fbc8b3a87440b608151" modify_external_catalogs: true
I20251025 07:52:24.280635 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_e1ca1209c59f456bb4b1ed1b3729f684" } modify_external_catalogs: true
I20251025 07:52:24.280853 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_e1ca1209c59f456bb4b1ed1b3729f684" } modify_external_catalogs: true
I20251025 07:52:24.281805 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_850337fc1e7d447d961162f0510838bd" } new_table_name: "default.Table_02b9c6449126452b9694503a3d6ad253" modify_external_catalogs: true
I20251025 07:52:24.282243 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_5abeba9d640f40509583df01dc62749c" } modify_external_catalogs: true
I20251025 07:52:24.282428 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_5abeba9d640f40509583df01dc62749c" } modify_external_catalogs: true
I20251025 07:52:24.282528 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_4ef8509650004dc9a3097f1d5004e03b" } modify_external_catalogs: true
I20251025 07:52:24.282722 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_4ef8509650004dc9a3097f1d5004e03b" } modify_external_catalogs: true
I20251025 07:52:24.283936 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_dfba56557b4e42b99167a77ea28e2c7c" } new_table_name: "default.Table_7c051fe4fcce4b96a48330196dbf2ae1" modify_external_catalogs: true
W20251025 07:52:24.290552 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ef3820c3d739476fad3d83d3aa658742 consensus state has no leader
I20251025 07:52:24.280025 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_ae419ef915754fb2a3b38e861f3a865a" } modify_external_catalogs: true
I20251025 07:52:24.318082 894 tablet_service.cc:1505] Processing CreateTablet for tablet ef3820c3d739476fad3d83d3aa658742 (DEFAULT_TABLE table=default.Table_eb8bbd390e29466e875081ce562d1bf8 [id=768d2afbbcb24c26818cf1f64b330428]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.319782 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef3820c3d739476fad3d83d3aa658742. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.321264 517 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 5ff751061bff4474a7dc5d236e94cf0e
W20251025 07:52:24.321734 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 668f7073de1c41c6882cf15e65d90e46 has no consensus state
I20251025 07:52:24.321828 519 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet c73e5c78331249c0948c56d4c9913568
I20251025 07:52:24.322166 520 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 9f33db90ba1e4c778f41388c2567f13f
W20251025 07:52:24.322196 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aac844f429fc4328a261a326181f6c35 has no consensus state
W20251025 07:52:24.322518 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 52a8fd3c834a4137956a86ad4f198f0f has no consensus state
I20251025 07:52:24.322548 521 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 17c51b2adc614835afb870b5f5d00556
W20251025 07:52:24.322892 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 503d0732dd094099889fb56a747bbd33 has no consensus state
I20251025 07:52:24.325857 524 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 798aa2d7bc0749faaf2e8802fcbe100e
I20251025 07:52:24.335803 756 tablet_service.cc:1505] Processing CreateTablet for tablet ef3820c3d739476fad3d83d3aa658742 (DEFAULT_TABLE table=default.Table_eb8bbd390e29466e875081ce562d1bf8 [id=768d2afbbcb24c26818cf1f64b330428]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.337450 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef3820c3d739476fad3d83d3aa658742. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.345490 893 tablet_service.cc:1505] Processing CreateTablet for tablet 503d0732dd094099889fb56a747bbd33 (DEFAULT_TABLE table=default.Table_7c051fe4fcce4b96a48330196dbf2ae1 [id=a5c73cb1e5a349ca8675f4ba4cb8a82d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.345986 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 503d0732dd094099889fb56a747bbd33. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.347086 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 503d0732dd094099889fb56a747bbd33 (DEFAULT_TABLE table=default.Table_7c051fe4fcce4b96a48330196dbf2ae1 [id=a5c73cb1e5a349ca8675f4ba4cb8a82d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.348487 524 master_service.cc:954] ReplaceTablet: received request to replace tablet ef3820c3d739476fad3d83d3aa658742 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:24.348649 754 tablet_service.cc:1505] Processing CreateTablet for tablet 668f7073de1c41c6882cf15e65d90e46 (DEFAULT_TABLE table=default.Table_e29bb32f37634fbc8b3a87440b608151 [id=027c96a19c37427d94df4f30880757d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.348816 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 503d0732dd094099889fb56a747bbd33. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.349161 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 668f7073de1c41c6882cf15e65d90e46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.350616 753 tablet_service.cc:1505] Processing CreateTablet for tablet 52a8fd3c834a4137956a86ad4f198f0f (DEFAULT_TABLE table=default.Table_02b9c6449126452b9694503a3d6ad253 [id=68d633fa085b416ab906780b71edf145]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.351100 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52a8fd3c834a4137956a86ad4f198f0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.352221 525 master_service.cc:954] ReplaceTablet: received request to replace tablet ef3820c3d739476fad3d83d3aa658742 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:24.352655 755 tablet_service.cc:1505] Processing CreateTablet for tablet 503d0732dd094099889fb56a747bbd33 (DEFAULT_TABLE table=default.Table_7c051fe4fcce4b96a48330196dbf2ae1 [id=a5c73cb1e5a349ca8675f4ba4cb8a82d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.353207 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 503d0732dd094099889fb56a747bbd33. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.355515 752 tablet_service.cc:1505] Processing CreateTablet for tablet aac844f429fc4328a261a326181f6c35 (DEFAULT_TABLE table=default.Table_485ce9a08de848209ce36bf5eb89ac0f [id=5e424c1c1c7c470f8640c7a7a571a11d]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:24.356478 512 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 52a8fd3c834a4137956a86ad4f198f0f (table default.Table_02b9c6449126452b9694503a3d6ad253 [id=68d633fa085b416ab906780b71edf145], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 52a8fd3c834a4137956a86ad4f198f0f consensus state has no leader
I20251025 07:52:24.356526 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aac844f429fc4328a261a326181f6c35. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.356935 518 master_service.cc:954] ReplaceTablet: received request to replace tablet ef3820c3d739476fad3d83d3aa658742 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:24.359966 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 668f7073de1c41c6882cf15e65d90e46 (DEFAULT_TABLE table=default.Table_e29bb32f37634fbc8b3a87440b608151 [id=027c96a19c37427d94df4f30880757d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.360085 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 52a8fd3c834a4137956a86ad4f198f0f (DEFAULT_TABLE table=default.Table_02b9c6449126452b9694503a3d6ad253 [id=68d633fa085b416ab906780b71edf145]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.360524 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 668f7073de1c41c6882cf15e65d90e46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.362600 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52a8fd3c834a4137956a86ad4f198f0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.367651 1025 tablet_service.cc:1505] Processing CreateTablet for tablet aac844f429fc4328a261a326181f6c35 (DEFAULT_TABLE table=default.Table_485ce9a08de848209ce36bf5eb89ac0f [id=5e424c1c1c7c470f8640c7a7a571a11d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.368199 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aac844f429fc4328a261a326181f6c35. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.370455 892 tablet_service.cc:1505] Processing CreateTablet for tablet 668f7073de1c41c6882cf15e65d90e46 (DEFAULT_TABLE table=default.Table_e29bb32f37634fbc8b3a87440b608151 [id=027c96a19c37427d94df4f30880757d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.370908 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 668f7073de1c41c6882cf15e65d90e46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.371573 521 master_service.cc:954] ReplaceTablet: received request to replace tablet ef3820c3d739476fad3d83d3aa658742 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:24.371948 522 master_service.cc:954] ReplaceTablet: received request to replace tablet ef3820c3d739476fad3d83d3aa658742 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:24.374511 890 tablet_service.cc:1505] Processing CreateTablet for tablet aac844f429fc4328a261a326181f6c35 (DEFAULT_TABLE table=default.Table_485ce9a08de848209ce36bf5eb89ac0f [id=5e424c1c1c7c470f8640c7a7a571a11d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.342484 1029 tablet_service.cc:1505] Processing CreateTablet for tablet ef3820c3d739476fad3d83d3aa658742 (DEFAULT_TABLE table=default.Table_eb8bbd390e29466e875081ce562d1bf8 [id=768d2afbbcb24c26818cf1f64b330428]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.375000 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aac844f429fc4328a261a326181f6c35. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.375016 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef3820c3d739476fad3d83d3aa658742. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.375662 891 tablet_service.cc:1505] Processing CreateTablet for tablet 52a8fd3c834a4137956a86ad4f198f0f (DEFAULT_TABLE table=default.Table_02b9c6449126452b9694503a3d6ad253 [id=68d633fa085b416ab906780b71edf145]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.376199 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52a8fd3c834a4137956a86ad4f198f0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.390349 1166 tablet_bootstrap.cc:492] T ef3820c3d739476fad3d83d3aa658742 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:24.391448 1167 tablet_bootstrap.cc:492] T 668f7073de1c41c6882cf15e65d90e46 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:24.395098 1166 tablet_bootstrap.cc:654] T ef3820c3d739476fad3d83d3aa658742 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:24.396620 1166 log.cc:826] T ef3820c3d739476fad3d83d3aa658742 P 3aea2869ff814cc7b142012faa274382: Log is configured to *not* fsync() on all Append() calls
I20251025 07:52:24.400292 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet ef3820c3d739476fad3d83d3aa658742
I20251025 07:52:24.401016 524 catalog_manager.cc:6719] ReplaceTablet: tablet ef3820c3d739476fad3d83d3aa658742 deleted and replaced by tablet d709aa500f8b422499c30843acd1a5ae
I20251025 07:52:24.402741 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet ef3820c3d739476fad3d83d3aa658742 with delete_type TABLET_DATA_DELETED (replaced by tablet d709aa500f8b422499c30843acd1a5ae) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:24.407315 756 tablet_service.cc:1552] Processing DeleteTablet for tablet ef3820c3d739476fad3d83d3aa658742 with delete_type TABLET_DATA_DELETED (replaced by tablet d709aa500f8b422499c30843acd1a5ae) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:24.407763 890 tablet_service.cc:1552] Processing DeleteTablet for tablet ef3820c3d739476fad3d83d3aa658742 with delete_type TABLET_DATA_DELETED (replaced by tablet d709aa500f8b422499c30843acd1a5ae) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:24.413286 1167 tablet_bootstrap.cc:654] T 668f7073de1c41c6882cf15e65d90e46 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:24.414805 1167 log.cc:826] T 668f7073de1c41c6882cf15e65d90e46 P 23eeffd3de254ce18db5febfde971c6b: Log is configured to *not* fsync() on all Append() calls
I20251025 07:52:24.415345 892 tablet_service.cc:1505] Processing CreateTablet for tablet d709aa500f8b422499c30843acd1a5ae (DEFAULT_TABLE table=default.Table_33eaf5d540914e43aad9bc6e22eb698a [id=768d2afbbcb24c26818cf1f64b330428]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.415462 1169 tablet_bootstrap.cc:492] T 52a8fd3c834a4137956a86ad4f198f0f P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:24.415753 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d709aa500f8b422499c30843acd1a5ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.416437 1026 tablet_service.cc:1505] Processing CreateTablet for tablet d709aa500f8b422499c30843acd1a5ae (DEFAULT_TABLE table=default.Table_33eaf5d540914e43aad9bc6e22eb698a [id=768d2afbbcb24c26818cf1f64b330428]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.416810 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d709aa500f8b422499c30843acd1a5ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.419878 1169 tablet_bootstrap.cc:654] T 52a8fd3c834a4137956a86ad4f198f0f P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:24.415345 754 tablet_service.cc:1505] Processing CreateTablet for tablet d709aa500f8b422499c30843acd1a5ae (DEFAULT_TABLE table=default.Table_33eaf5d540914e43aad9bc6e22eb698a [id=768d2afbbcb24c26818cf1f64b330428]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.423867 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d709aa500f8b422499c30843acd1a5ae. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:24.423847 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet ef3820c3d739476fad3d83d3aa658742 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef3820c3d739476fad3d83d3aa658742 already in progress: creating tablet
I20251025 07:52:24.425241 1166 tablet_bootstrap.cc:492] T ef3820c3d739476fad3d83d3aa658742 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:24.425494 1166 ts_tablet_manager.cc:1403] T ef3820c3d739476fad3d83d3aa658742 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.010s
W20251025 07:52:24.430927 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet ef3820c3d739476fad3d83d3aa658742 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef3820c3d739476fad3d83d3aa658742 already in progress: creating tablet
I20251025 07:52:24.431931 1166 raft_consensus.cc:359] T ef3820c3d739476fad3d83d3aa658742 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.432466 1166 raft_consensus.cc:385] T ef3820c3d739476fad3d83d3aa658742 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:24.432540 1167 tablet_bootstrap.cc:492] T 668f7073de1c41c6882cf15e65d90e46 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:24.432624 1166 raft_consensus.cc:740] T ef3820c3d739476fad3d83d3aa658742 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
W20251025 07:52:24.432680 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet ef3820c3d739476fad3d83d3aa658742 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef3820c3d739476fad3d83d3aa658742 already in progress: creating tablet
I20251025 07:52:24.433449 1167 ts_tablet_manager.cc:1403] T 668f7073de1c41c6882cf15e65d90e46 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.042s user 0.001s sys 0.009s
I20251025 07:52:24.433455 1166 consensus_queue.cc:260] T ef3820c3d739476fad3d83d3aa658742 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.422122 1169 log.cc:826] T 52a8fd3c834a4137956a86ad4f198f0f P db42be7ad3c741a0aa427f27e245993b: Log is configured to *not* fsync() on all Append() calls
I20251025 07:52:24.437353 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_e4be2263552c4760b6f89c431054d3ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:24.437949 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e4be2263552c4760b6f89c431054d3ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:24.442677 1166 ts_tablet_manager.cc:1434] T ef3820c3d739476fad3d83d3aa658742 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.017s user 0.009s sys 0.007s
I20251025 07:52:24.445274 1166 tablet_bootstrap.cc:492] T 503d0732dd094099889fb56a747bbd33 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:24.447084 1166 tablet_bootstrap.cc:654] T 503d0732dd094099889fb56a747bbd33 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:24.448181 890 tablet_service.cc:1552] Processing DeleteTablet for tablet ef3820c3d739476fad3d83d3aa658742 with delete_type TABLET_DATA_DELETED (replaced by tablet d709aa500f8b422499c30843acd1a5ae) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:24.448452 1172 tablet_replica.cc:333] T ef3820c3d739476fad3d83d3aa658742 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:24.448853 1172 raft_consensus.cc:2243] T ef3820c3d739476fad3d83d3aa658742 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:24.449242 1172 raft_consensus.cc:2272] T ef3820c3d739476fad3d83d3aa658742 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:24.449695 1169 tablet_bootstrap.cc:492] T 52a8fd3c834a4137956a86ad4f198f0f P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:24.449981 1169 ts_tablet_manager.cc:1403] T 52a8fd3c834a4137956a86ad4f198f0f P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.007s
I20251025 07:52:24.452586 1172 ts_tablet_manager.cc:1916] T ef3820c3d739476fad3d83d3aa658742 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:24.455367 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_325d6151a4134532ad8b68c9c0aa6f64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:24.455958 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_325d6151a4134532ad8b68c9c0aa6f64 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:24.457407 1169 raft_consensus.cc:359] T 52a8fd3c834a4137956a86ad4f198f0f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.457808 1169 raft_consensus.cc:385] T 52a8fd3c834a4137956a86ad4f198f0f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:24.458016 1169 raft_consensus.cc:740] T 52a8fd3c834a4137956a86ad4f198f0f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:24.458760 756 tablet_service.cc:1505] Processing CreateTablet for tablet f05afed029d446ea9eb0073642c7312d (DEFAULT_TABLE table=default.Table_e4be2263552c4760b6f89c431054d3ba [id=e7a5b463914840c980a306b7d46ea871]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.459206 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f05afed029d446ea9eb0073642c7312d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.459196 1169 consensus_queue.cc:260] T 52a8fd3c834a4137956a86ad4f198f0f P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.460098 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_d0152965363148028bd560ecb3b594e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:24.460718 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0152965363148028bd560ecb3b594e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:24.463361 1026 tablet_service.cc:1505] Processing CreateTablet for tablet f05afed029d446ea9eb0073642c7312d (DEFAULT_TABLE table=default.Table_e4be2263552c4760b6f89c431054d3ba [id=e7a5b463914840c980a306b7d46ea871]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.463774 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f05afed029d446ea9eb0073642c7312d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.464197 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_6cd1984401d2468e9494afe02d87f74f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:24.464805 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6cd1984401d2468e9494afe02d87f74f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:24.466313 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_d6012ae31a9c4b2899e81f96ea4a244e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:24.466866 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6012ae31a9c4b2899e81f96ea4a244e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:24.468524 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_b307db62a04146b7bccc879f740e7274"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:24.469135 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b307db62a04146b7bccc879f740e7274 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:24.470593 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_43eec86aa9fc405fa761a8eaa89d2db0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:24.471076 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43eec86aa9fc405fa761a8eaa89d2db0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:24.472170 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_ae4075623a14448aaba9f22ae53d0755"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:24.472609 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae4075623a14448aaba9f22ae53d0755 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:24.474884 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_ad812bcf86a04b0b89a3f0bc961062a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:24.475401 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad812bcf86a04b0b89a3f0bc961062a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:24.476464 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_929d091973a047ad9123baa6c4b317aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:24.476929 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_929d091973a047ad9123baa6c4b317aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:24.479655 890 tablet_service.cc:1505] Processing CreateTablet for tablet f05afed029d446ea9eb0073642c7312d (DEFAULT_TABLE table=default.Table_e4be2263552c4760b6f89c431054d3ba [id=e7a5b463914840c980a306b7d46ea871]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.480191 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f05afed029d446ea9eb0073642c7312d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.490628 1166 tablet_bootstrap.cc:492] T 503d0732dd094099889fb56a747bbd33 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:24.490783 1166 ts_tablet_manager.cc:1403] T 503d0732dd094099889fb56a747bbd33 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.001s
I20251025 07:52:24.491709 1166 raft_consensus.cc:359] T 503d0732dd094099889fb56a747bbd33 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.491906 1166 raft_consensus.cc:385] T 503d0732dd094099889fb56a747bbd33 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:24.491995 1166 raft_consensus.cc:740] T 503d0732dd094099889fb56a747bbd33 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:24.491863 1167 raft_consensus.cc:359] T 668f7073de1c41c6882cf15e65d90e46 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.492283 1167 raft_consensus.cc:385] T 668f7073de1c41c6882cf15e65d90e46 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:24.492282 1166 consensus_queue.cc:260] T 503d0732dd094099889fb56a747bbd33 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.492386 1167 raft_consensus.cc:740] T 668f7073de1c41c6882cf15e65d90e46 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:24.492803 1166 ts_tablet_manager.cc:1434] T 503d0732dd094099889fb56a747bbd33 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:24.492750 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_b307db62a04146b7bccc879f740e7274" } new_table_name: "default.Table_a6d264fbdb6c49f383edfb0311e0af2f" modify_external_catalogs: true
I20251025 07:52:24.493084 1166 tablet_bootstrap.cc:492] T 52a8fd3c834a4137956a86ad4f198f0f P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:24.493152 1167 consensus_queue.cc:260] T 668f7073de1c41c6882cf15e65d90e46 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.495234 1166 tablet_bootstrap.cc:654] T 52a8fd3c834a4137956a86ad4f198f0f P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:24.504038 1166 tablet_bootstrap.cc:492] T 52a8fd3c834a4137956a86ad4f198f0f P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:24.504225 1166 ts_tablet_manager.cc:1403] T 52a8fd3c834a4137956a86ad4f198f0f P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20251025 07:52:24.505069 1166 raft_consensus.cc:359] T 52a8fd3c834a4137956a86ad4f198f0f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.505303 1166 raft_consensus.cc:385] T 52a8fd3c834a4137956a86ad4f198f0f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:24.505409 1166 raft_consensus.cc:740] T 52a8fd3c834a4137956a86ad4f198f0f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:24.505645 1166 consensus_queue.cc:260] T 52a8fd3c834a4137956a86ad4f198f0f P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.506109 1166 ts_tablet_manager.cc:1434] T 52a8fd3c834a4137956a86ad4f198f0f P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:24.507309 1166 tablet_bootstrap.cc:492] T 668f7073de1c41c6882cf15e65d90e46 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:24.509591 1166 tablet_bootstrap.cc:654] T 668f7073de1c41c6882cf15e65d90e46 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:24.512768 521 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 503d0732dd094099889fb56a747bbd33 (table default.Table_7c051fe4fcce4b96a48330196dbf2ae1 [id=a5c73cb1e5a349ca8675f4ba4cb8a82d]). Expected version 1 got 0
I20251025 07:52:24.515045 756 tablet_service.cc:1505] Processing CreateTablet for tablet 243825c6dd2d4461a01edec88f33f6ee (DEFAULT_TABLE table=default.Table_325d6151a4134532ad8b68c9c0aa6f64 [id=68668d9bd5f34db8ab8f64ff1dd6d32f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.515499 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 243825c6dd2d4461a01edec88f33f6ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.515832 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 243825c6dd2d4461a01edec88f33f6ee (DEFAULT_TABLE table=default.Table_325d6151a4134532ad8b68c9c0aa6f64 [id=68668d9bd5f34db8ab8f64ff1dd6d32f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.516237 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 243825c6dd2d4461a01edec88f33f6ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.516624 892 tablet_service.cc:1505] Processing CreateTablet for tablet 243825c6dd2d4461a01edec88f33f6ee (DEFAULT_TABLE table=default.Table_325d6151a4134532ad8b68c9c0aa6f64 [id=68668d9bd5f34db8ab8f64ff1dd6d32f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.517072 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 243825c6dd2d4461a01edec88f33f6ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.523264 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_325d6151a4134532ad8b68c9c0aa6f64" } modify_external_catalogs: true
I20251025 07:52:24.523506 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_325d6151a4134532ad8b68c9c0aa6f64" } modify_external_catalogs: true
I20251025 07:52:24.525795 521 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 52a8fd3c834a4137956a86ad4f198f0f (table default.Table_02b9c6449126452b9694503a3d6ad253 [id=68d633fa085b416ab906780b71edf145]). Expected version 1 got 0
I20251025 07:52:24.526428 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_e29bb32f37634fbc8b3a87440b608151" } modify_external_catalogs: true
I20251025 07:52:24.526621 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_e29bb32f37634fbc8b3a87440b608151" } modify_external_catalogs: true
I20251025 07:52:24.528191 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_d0152965363148028bd560ecb3b594e5" } new_table_name: "default.Table_8a06efe638604f20bda568cdb8382a48" modify_external_catalogs: true
I20251025 07:52:24.529242 1166 tablet_bootstrap.cc:492] T 668f7073de1c41c6882cf15e65d90e46 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:24.529181 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_e4be2263552c4760b6f89c431054d3ba" } new_table_name: "default.Table_c96e62972b3c4ee48b6c3ea90831a34f" modify_external_catalogs: true
I20251025 07:52:24.529413 1166 ts_tablet_manager.cc:1403] T 668f7073de1c41c6882cf15e65d90e46 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.006s
I20251025 07:52:24.530221 1166 raft_consensus.cc:359] T 668f7073de1c41c6882cf15e65d90e46 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.530956 1166 raft_consensus.cc:385] T 668f7073de1c41c6882cf15e65d90e46 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:24.531042 1166 raft_consensus.cc:740] T 668f7073de1c41c6882cf15e65d90e46 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:24.531087 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_485ce9a08de848209ce36bf5eb89ac0f" } new_table_name: "default.Table_42908396d76f4df8b63e3ce11dcc2dfb" modify_external_catalogs: true
I20251025 07:52:24.531149 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_33eaf5d540914e43aad9bc6e22eb698a" } modify_external_catalogs: true
I20251025 07:52:24.531260 1166 consensus_queue.cc:260] T 668f7073de1c41c6882cf15e65d90e46 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.531471 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_7c051fe4fcce4b96a48330196dbf2ae1" } new_table_name: "default.Table_533c8546736848ea9ec200f63386630c" modify_external_catalogs: true
I20251025 07:52:24.532706 1166 ts_tablet_manager.cc:1434] T 668f7073de1c41c6882cf15e65d90e46 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251025 07:52:24.532979 1166 tablet_bootstrap.cc:492] T aac844f429fc4328a261a326181f6c35 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:24.533703 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_d6012ae31a9c4b2899e81f96ea4a244e" } modify_external_catalogs: true
I20251025 07:52:24.534651 1166 tablet_bootstrap.cc:654] T aac844f429fc4328a261a326181f6c35 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:24.535344 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_33eaf5d540914e43aad9bc6e22eb698a" } modify_external_catalogs: true
I20251025 07:52:24.538760 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_d6012ae31a9c4b2899e81f96ea4a244e" } modify_external_catalogs: true
I20251025 07:52:24.542811 1166 tablet_bootstrap.cc:492] T aac844f429fc4328a261a326181f6c35 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:24.543414 1166 ts_tablet_manager.cc:1403] T aac844f429fc4328a261a326181f6c35 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.002s
I20251025 07:52:24.544420 1166 raft_consensus.cc:359] T aac844f429fc4328a261a326181f6c35 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.545557 1166 raft_consensus.cc:385] T aac844f429fc4328a261a326181f6c35 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:24.545816 1166 raft_consensus.cc:740] T aac844f429fc4328a261a326181f6c35 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:24.546322 1166 consensus_queue.cc:260] T aac844f429fc4328a261a326181f6c35 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.547834 1166 ts_tablet_manager.cc:1434] T aac844f429fc4328a261a326181f6c35 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251025 07:52:24.548559 1166 tablet_bootstrap.cc:492] T d709aa500f8b422499c30843acd1a5ae P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:24.551900 1166 tablet_bootstrap.cc:654] T d709aa500f8b422499c30843acd1a5ae P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:24.556545 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e84dbbd4a0bd4e4dbf1212fefbf00fd4 has no consensus state
I20251025 07:52:24.558338 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_02b9c6449126452b9694503a3d6ad253" } modify_external_catalogs: true
I20251025 07:52:24.558931 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_02b9c6449126452b9694503a3d6ad253" } modify_external_catalogs: true
I20251025 07:52:24.561153 1166 tablet_bootstrap.cc:492] T d709aa500f8b422499c30843acd1a5ae P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:24.561367 1166 ts_tablet_manager.cc:1403] T d709aa500f8b422499c30843acd1a5ae P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.003s
I20251025 07:52:24.562685 1174 raft_consensus.cc:493] T 503d0732dd094099889fb56a747bbd33 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:24.562953 1166 raft_consensus.cc:359] T d709aa500f8b422499c30843acd1a5ae P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.563153 1166 raft_consensus.cc:385] T d709aa500f8b422499c30843acd1a5ae P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:24.562983 1174 raft_consensus.cc:515] T 503d0732dd094099889fb56a747bbd33 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.563259 1166 raft_consensus.cc:740] T d709aa500f8b422499c30843acd1a5ae P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:24.564117 1166 consensus_queue.cc:260] T d709aa500f8b422499c30843acd1a5ae P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.565603 1174 leader_election.cc:290] T 503d0732dd094099889fb56a747bbd33 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:24.566324 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 243825c6dd2d4461a01edec88f33f6ee
I20251025 07:52:24.568380 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 243825c6dd2d4461a01edec88f33f6ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:24.569099 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 243825c6dd2d4461a01edec88f33f6ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:47408
W20251025 07:52:24.569126 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 243825c6dd2d4461a01edec88f33f6ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 243825c6dd2d4461a01edec88f33f6ee already in progress: creating tablet
W20251025 07:52:24.569890 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 243825c6dd2d4461a01edec88f33f6ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 243825c6dd2d4461a01edec88f33f6ee already in progress: creating tablet
I20251025 07:52:24.570930 1166 ts_tablet_manager.cc:1434] T d709aa500f8b422499c30843acd1a5ae P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.009s user 0.002s sys 0.002s
I20251025 07:52:24.571228 1166 tablet_bootstrap.cc:492] T f05afed029d446ea9eb0073642c7312d P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:24.571410 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 243825c6dd2d4461a01edec88f33f6ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:24.573031 1166 tablet_bootstrap.cc:654] T f05afed029d446ea9eb0073642c7312d P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:24.529222 1172 ts_tablet_manager.cc:1929] T ef3820c3d739476fad3d83d3aa658742 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:24.573208 1172 log.cc:1199] T ef3820c3d739476fad3d83d3aa658742 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/ef3820c3d739476fad3d83d3aa658742
I20251025 07:52:24.576324 1172 ts_tablet_manager.cc:1950] T ef3820c3d739476fad3d83d3aa658742 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:24.577404 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 668f7073de1c41c6882cf15e65d90e46
I20251025 07:52:24.578176 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet ef3820c3d739476fad3d83d3aa658742 (table default.Table_33eaf5d540914e43aad9bc6e22eb698a [id=768d2afbbcb24c26818cf1f64b330428]) successfully deleted
I20251025 07:52:24.578384 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 668f7073de1c41c6882cf15e65d90e46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:24.579433 1172 tablet_replica.cc:333] T 668f7073de1c41c6882cf15e65d90e46 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:24.579702 1172 raft_consensus.cc:2243] T 668f7073de1c41c6882cf15e65d90e46 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:24.580272 1172 raft_consensus.cc:2272] T 668f7073de1c41c6882cf15e65d90e46 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:24.580263 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRBHlYt/build/asan/bin/kudu with pid 563
W20251025 07:52:24.580742 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 243825c6dd2d4461a01edec88f33f6ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 243825c6dd2d4461a01edec88f33f6ee already in progress: creating tablet
I20251025 07:52:24.583814 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 668f7073de1c41c6882cf15e65d90e46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:24.586432 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 668f7073de1c41c6882cf15e65d90e46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:48830
W20251025 07:52:24.587144 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 668f7073de1c41c6882cf15e65d90e46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 668f7073de1c41c6882cf15e65d90e46 already in progress: creating tablet
W20251025 07:52:24.587854 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 668f7073de1c41c6882cf15e65d90e46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 668f7073de1c41c6882cf15e65d90e46 already in progress: creating tablet
I20251025 07:52:24.593418 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "503d0732dd094099889fb56a747bbd33" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
W20251025 07:52:24.594666 847 leader_election.cc:343] T 503d0732dd094099889fb56a747bbd33 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:24.595667 1169 ts_tablet_manager.cc:1434] T 52a8fd3c834a4137956a86ad4f198f0f P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.146s user 0.045s sys 0.037s
I20251025 07:52:24.596223 1167 ts_tablet_manager.cc:1434] T 668f7073de1c41c6882cf15e65d90e46 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.163s user 0.047s sys 0.022s
I20251025 07:52:24.597483 1169 tablet_bootstrap.cc:492] T 668f7073de1c41c6882cf15e65d90e46 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:24.598604 1167 tablet_bootstrap.cc:492] T aac844f429fc4328a261a326181f6c35 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:24.599370 1169 tablet_bootstrap.cc:654] T 668f7073de1c41c6882cf15e65d90e46 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:24.600356 1167 tablet_bootstrap.cc:654] T aac844f429fc4328a261a326181f6c35 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:24.601704 1096 heartbeater.cc:499] Master 127.0.104.61:42203 was elected leader, sending a full tablet report...
I20251025 07:52:24.602041 1169 tablet_bootstrap.cc:492] T 668f7073de1c41c6882cf15e65d90e46 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:24.602221 1169 ts_tablet_manager.cc:1403] T 668f7073de1c41c6882cf15e65d90e46 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251025 07:52:24.603229 1169 raft_consensus.cc:359] T 668f7073de1c41c6882cf15e65d90e46 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.603430 1169 raft_consensus.cc:385] T 668f7073de1c41c6882cf15e65d90e46 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:24.603534 1169 raft_consensus.cc:740] T 668f7073de1c41c6882cf15e65d90e46 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:24.603801 1169 consensus_queue.cc:260] T 668f7073de1c41c6882cf15e65d90e46 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.606783 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "503d0732dd094099889fb56a747bbd33" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:24.610167 1097 tablet.cc:2378] T 52a8fd3c834a4137956a86ad4f198f0f P db42be7ad3c741a0aa427f27e245993b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251025 07:52:24.615471 1172 ts_tablet_manager.cc:1916] T 668f7073de1c41c6882cf15e65d90e46 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:24.618790 1169 ts_tablet_manager.cc:1434] T 668f7073de1c41c6882cf15e65d90e46 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20251025 07:52:24.620813 1169 tablet_bootstrap.cc:492] T aac844f429fc4328a261a326181f6c35 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
W20251025 07:52:24.620805 850 leader_election.cc:343] T 503d0732dd094099889fb56a747bbd33 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:24.621081 850 leader_election.cc:304] T 503d0732dd094099889fb56a747bbd33 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:24.623919 517 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet aac844f429fc4328a261a326181f6c35 (table default.Table_485ce9a08de848209ce36bf5eb89ac0f [id=5e424c1c1c7c470f8640c7a7a571a11d]). Expected version 1 got 0
I20251025 07:52:24.624943 1169 tablet_bootstrap.cc:654] T aac844f429fc4328a261a326181f6c35 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:24.625391 520 master_service.cc:954] ReplaceTablet: received request to replace tablet 503d0732dd094099889fb56a747bbd33 from {username='slave'} at 127.0.0.1:45978
W20251025 07:52:24.625391 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d684da8134ce480693f470acdac176d4 has no consensus state
I20251025 07:52:24.626636 524 master_service.cc:954] ReplaceTablet: received request to replace tablet 503d0732dd094099889fb56a747bbd33 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:24.629171 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 668f7073de1c41c6882cf15e65d90e46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:24.629467 518 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet dd1c847722d54a178578604de9dab23e
I20251025 07:52:24.629711 1171 tablet_replica.cc:333] T 668f7073de1c41c6882cf15e65d90e46 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:24.630336 1171 raft_consensus.cc:2243] T 668f7073de1c41c6882cf15e65d90e46 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:24.630815 1171 raft_consensus.cc:2272] T 668f7073de1c41c6882cf15e65d90e46 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:24.632771 518 master_service.cc:954] ReplaceTablet: received request to replace tablet f05afed029d446ea9eb0073642c7312d from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:24.633088 1171 ts_tablet_manager.cc:1916] T 668f7073de1c41c6882cf15e65d90e46 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:24.644042 756 tablet_service.cc:1505] Processing CreateTablet for tablet 54cb4953ff264c13a0cc730b4c452ecc (DEFAULT_TABLE table=default.Table_ad812bcf86a04b0b89a3f0bc961062a9 [id=fe2800366dea4cdd85696d44df21a4c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.644490 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54cb4953ff264c13a0cc730b4c452ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.647979 1025 tablet_service.cc:1505] Processing CreateTablet for tablet e84dbbd4a0bd4e4dbf1212fefbf00fd4 (DEFAULT_TABLE table=default.Table_a6d264fbdb6c49f383edfb0311e0af2f [id=b6e173967bad47e6b306627221b80f94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.648379 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e84dbbd4a0bd4e4dbf1212fefbf00fd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.648871 754 tablet_service.cc:1505] Processing CreateTablet for tablet e84dbbd4a0bd4e4dbf1212fefbf00fd4 (DEFAULT_TABLE table=default.Table_a6d264fbdb6c49f383edfb0311e0af2f [id=b6e173967bad47e6b306627221b80f94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.649236 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e84dbbd4a0bd4e4dbf1212fefbf00fd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.650406 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 89bfb7e2f051454aae0af5755a79ca58 (DEFAULT_TABLE table=default.Table_ae4075623a14448aaba9f22ae53d0755 [id=d82b3e1317434cf08a0d51e8a9427224]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.650694 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89bfb7e2f051454aae0af5755a79ca58. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:24.642560 962 tablet.cc:2378] T 503d0732dd094099889fb56a747bbd33 P 3aea2869ff814cc7b142012faa274382: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251025 07:52:24.653039 1183 raft_consensus.cc:2749] T 503d0732dd094099889fb56a747bbd33 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:24.653774 753 tablet_service.cc:1505] Processing CreateTablet for tablet 89bfb7e2f051454aae0af5755a79ca58 (DEFAULT_TABLE table=default.Table_ae4075623a14448aaba9f22ae53d0755 [id=d82b3e1317434cf08a0d51e8a9427224]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.654127 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89bfb7e2f051454aae0af5755a79ca58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.654153 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 54cb4953ff264c13a0cc730b4c452ecc (DEFAULT_TABLE table=default.Table_ad812bcf86a04b0b89a3f0bc961062a9 [id=fe2800366dea4cdd85696d44df21a4c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.655310 755 tablet_service.cc:1505] Processing CreateTablet for tablet 5680e1adaf3f4b7f8c5e05523dd0950a (DEFAULT_TABLE table=default.Table_6cd1984401d2468e9494afe02d87f74f [id=cb037f61ee0b4b6cb8fb613263a278fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.655306 894 tablet_service.cc:1505] Processing CreateTablet for tablet 5680e1adaf3f4b7f8c5e05523dd0950a (DEFAULT_TABLE table=default.Table_6cd1984401d2468e9494afe02d87f74f [id=cb037f61ee0b4b6cb8fb613263a278fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.655639 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5680e1adaf3f4b7f8c5e05523dd0950a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.655740 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5680e1adaf3f4b7f8c5e05523dd0950a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.655689 889 tablet_service.cc:1505] Processing CreateTablet for tablet d684da8134ce480693f470acdac176d4 (DEFAULT_TABLE table=default.Table_8a06efe638604f20bda568cdb8382a48 [id=4443da834fc34a01989b1d8cce759866]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.656212 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d684da8134ce480693f470acdac176d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.656729 752 tablet_service.cc:1505] Processing CreateTablet for tablet 14fd790ba7054c1fb09cdfb19b001987 (DEFAULT_TABLE table=default.Table_43eec86aa9fc405fa761a8eaa89d2db0 [id=67beaeffaf2243ef86f6c45084781432]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.657037 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14fd790ba7054c1fb09cdfb19b001987. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.658488 893 tablet_service.cc:1505] Processing CreateTablet for tablet 14fd790ba7054c1fb09cdfb19b001987 (DEFAULT_TABLE table=default.Table_43eec86aa9fc405fa761a8eaa89d2db0 [id=67beaeffaf2243ef86f6c45084781432]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.658592 751 tablet_service.cc:1505] Processing CreateTablet for tablet d684da8134ce480693f470acdac176d4 (DEFAULT_TABLE table=default.Table_8a06efe638604f20bda568cdb8382a48 [id=4443da834fc34a01989b1d8cce759866]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.658820 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14fd790ba7054c1fb09cdfb19b001987. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.659147 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d684da8134ce480693f470acdac176d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.663158 1024 tablet_service.cc:1505] Processing CreateTablet for tablet d684da8134ce480693f470acdac176d4 (DEFAULT_TABLE table=default.Table_8a06efe638604f20bda568cdb8382a48 [id=4443da834fc34a01989b1d8cce759866]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.663455 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 14fd790ba7054c1fb09cdfb19b001987 (DEFAULT_TABLE table=default.Table_43eec86aa9fc405fa761a8eaa89d2db0 [id=67beaeffaf2243ef86f6c45084781432]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.663565 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d684da8134ce480693f470acdac176d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.663781 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14fd790ba7054c1fb09cdfb19b001987. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.665644 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet f05afed029d446ea9eb0073642c7312d
I20251025 07:52:24.666186 518 catalog_manager.cc:6719] ReplaceTablet: tablet f05afed029d446ea9eb0073642c7312d deleted and replaced by tablet 3be8063aee794d2abcfb1d6e24209c06
I20251025 07:52:24.666428 518 master_service.cc:954] ReplaceTablet: received request to replace tablet aac844f429fc4328a261a326181f6c35 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:24.669152 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet f05afed029d446ea9eb0073642c7312d with delete_type TABLET_DATA_DELETED (replaced by tablet 3be8063aee794d2abcfb1d6e24209c06) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:24.669386 888 tablet_service.cc:1552] Processing DeleteTablet for tablet f05afed029d446ea9eb0073642c7312d with delete_type TABLET_DATA_DELETED (replaced by tablet 3be8063aee794d2abcfb1d6e24209c06) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:24.669922 750 tablet_service.cc:1552] Processing DeleteTablet for tablet f05afed029d446ea9eb0073642c7312d with delete_type TABLET_DATA_DELETED (replaced by tablet 3be8063aee794d2abcfb1d6e24209c06) from {username='slave'} at 127.0.0.1:48830
W20251025 07:52:24.669973 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet f05afed029d446ea9eb0073642c7312d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f05afed029d446ea9eb0073642c7312d already in progress: creating tablet
I20251025 07:52:24.675143 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 503d0732dd094099889fb56a747bbd33
I20251025 07:52:24.675350 1171 ts_tablet_manager.cc:1929] T 668f7073de1c41c6882cf15e65d90e46 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:24.675505 1171 log.cc:1199] T 668f7073de1c41c6882cf15e65d90e46 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/668f7073de1c41c6882cf15e65d90e46
I20251025 07:52:24.675709 520 catalog_manager.cc:6719] ReplaceTablet: tablet 503d0732dd094099889fb56a747bbd33 deleted and replaced by tablet 20e4178c16bd4f18b1c6bc969f6a0c6c
I20251025 07:52:24.675935 1171 ts_tablet_manager.cc:1950] T 668f7073de1c41c6882cf15e65d90e46 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:24.675923 520 master_service.cc:954] ReplaceTablet: received request to replace tablet 668f7073de1c41c6882cf15e65d90e46 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:24.676554 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_1437bb6ce9154a15b1817a029151c71a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:24.677080 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1437bb6ce9154a15b1817a029151c71a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:24.651335 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 5680e1adaf3f4b7f8c5e05523dd0950a (DEFAULT_TABLE table=default.Table_6cd1984401d2468e9494afe02d87f74f [id=cb037f61ee0b4b6cb8fb613263a278fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.679128 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5680e1adaf3f4b7f8c5e05523dd0950a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.679349 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet 503d0732dd094099889fb56a747bbd33 with delete_type TABLET_DATA_DELETED (replaced by tablet 20e4178c16bd4f18b1c6bc969f6a0c6c) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:24.679549 1183 raft_consensus.cc:493] T d709aa500f8b422499c30843acd1a5ae P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:24.679739 1183 raft_consensus.cc:515] T d709aa500f8b422499c30843acd1a5ae P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.680459 1183 leader_election.cc:290] T d709aa500f8b422499c30843acd1a5ae P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:24.681313 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d709aa500f8b422499c30843acd1a5ae" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:24.681874 850 leader_election.cc:343] T d709aa500f8b422499c30843acd1a5ae P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:24.682370 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 503d0732dd094099889fb56a747bbd33 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 503d0732dd094099889fb56a747bbd33 already in progress: creating tablet
I20251025 07:52:24.682605 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 503d0732dd094099889fb56a747bbd33 with delete_type TABLET_DATA_DELETED (replaced by tablet 20e4178c16bd4f18b1c6bc969f6a0c6c) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:24.683943 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54cb4953ff264c13a0cc730b4c452ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.642892 1166 tablet_bootstrap.cc:492] T f05afed029d446ea9eb0073642c7312d P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:24.685992 1166 ts_tablet_manager.cc:1403] T f05afed029d446ea9eb0073642c7312d P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.115s user 0.005s sys 0.012s
I20251025 07:52:24.686874 1166 raft_consensus.cc:359] T f05afed029d446ea9eb0073642c7312d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:24.687067 1166 raft_consensus.cc:385] T f05afed029d446ea9eb0073642c7312d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:24.687132 1166 raft_consensus.cc:740] T f05afed029d446ea9eb0073642c7312d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:24.688210 1166 consensus_queue.cc:260] T f05afed029d446ea9eb0073642c7312d P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:24.688707 1166 ts_tablet_manager.cc:1434] T f05afed029d446ea9eb0073642c7312d P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251025 07:52:24.689000 1166 tablet_bootstrap.cc:492] T 243825c6dd2d4461a01edec88f33f6ee P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:24.645066 890 tablet_service.cc:1505] Processing CreateTablet for tablet 54cb4953ff264c13a0cc730b4c452ecc (DEFAULT_TABLE table=default.Table_ad812bcf86a04b0b89a3f0bc961062a9 [id=fe2800366dea4cdd85696d44df21a4c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.690603 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54cb4953ff264c13a0cc730b4c452ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.690776 1166 tablet_bootstrap.cc:654] T 243825c6dd2d4461a01edec88f33f6ee P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:24.692744 1167 tablet_bootstrap.cc:492] T aac844f429fc4328a261a326181f6c35 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:24.692888 1167 ts_tablet_manager.cc:1403] T aac844f429fc4328a261a326181f6c35 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.094s user 0.009s sys 0.026s
I20251025 07:52:24.693639 1167 raft_consensus.cc:359] T aac844f429fc4328a261a326181f6c35 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.693810 1167 raft_consensus.cc:385] T aac844f429fc4328a261a326181f6c35 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:24.693881 1167 raft_consensus.cc:740] T aac844f429fc4328a261a326181f6c35 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:24.694088 1167 consensus_queue.cc:260] T aac844f429fc4328a261a326181f6c35 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.694566 1167 ts_tablet_manager.cc:1434] T aac844f429fc4328a261a326181f6c35 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:24.695643 1167 tablet_bootstrap.cc:492] T 52a8fd3c834a4137956a86ad4f198f0f P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:24.647310 892 tablet_service.cc:1505] Processing CreateTablet for tablet e84dbbd4a0bd4e4dbf1212fefbf00fd4 (DEFAULT_TABLE table=default.Table_a6d264fbdb6c49f383edfb0311e0af2f [id=b6e173967bad47e6b306627221b80f94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.647607 891 tablet_service.cc:1505] Processing CreateTablet for tablet 89bfb7e2f051454aae0af5755a79ca58 (DEFAULT_TABLE table=default.Table_ae4075623a14448aaba9f22ae53d0755 [id=d82b3e1317434cf08a0d51e8a9427224]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.697045 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e84dbbd4a0bd4e4dbf1212fefbf00fd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.697316 1167 tablet_bootstrap.cc:654] T 52a8fd3c834a4137956a86ad4f198f0f P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:24.699761 1167 tablet_bootstrap.cc:492] T 52a8fd3c834a4137956a86ad4f198f0f P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:24.699954 1167 ts_tablet_manager.cc:1403] T 52a8fd3c834a4137956a86ad4f198f0f P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20251025 07:52:24.701093 1167 raft_consensus.cc:359] T 52a8fd3c834a4137956a86ad4f198f0f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.701359 1167 raft_consensus.cc:385] T 52a8fd3c834a4137956a86ad4f198f0f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:24.701443 1167 raft_consensus.cc:740] T 52a8fd3c834a4137956a86ad4f198f0f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:24.701857 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89bfb7e2f051454aae0af5755a79ca58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.701737 1167 consensus_queue.cc:260] T 52a8fd3c834a4137956a86ad4f198f0f P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.702378 1167 ts_tablet_manager.cc:1434] T 52a8fd3c834a4137956a86ad4f198f0f P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251025 07:52:24.702776 1167 tablet_bootstrap.cc:492] T 503d0732dd094099889fb56a747bbd33 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:24.704921 1167 tablet_bootstrap.cc:654] T 503d0732dd094099889fb56a747bbd33 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:24.710947 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 503d0732dd094099889fb56a747bbd33 with delete_type TABLET_DATA_DELETED (replaced by tablet 20e4178c16bd4f18b1c6bc969f6a0c6c) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:24.711519 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 668f7073de1c41c6882cf15e65d90e46 (table default.Table_e29bb32f37634fbc8b3a87440b608151 [id=027c96a19c37427d94df4f30880757d4]) successfully deleted
I20251025 07:52:24.711544 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d709aa500f8b422499c30843acd1a5ae" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
W20251025 07:52:24.711740 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet f05afed029d446ea9eb0073642c7312d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f05afed029d446ea9eb0073642c7312d already in progress: creating tablet
W20251025 07:52:24.712222 847 leader_election.cc:343] T d709aa500f8b422499c30843acd1a5ae P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:24.712260 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 503d0732dd094099889fb56a747bbd33 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 503d0732dd094099889fb56a747bbd33 already in progress: creating tablet
I20251025 07:52:24.712419 847 leader_election.cc:304] T d709aa500f8b422499c30843acd1a5ae P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:24.712985 1183 raft_consensus.cc:2749] T d709aa500f8b422499c30843acd1a5ae P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:24.719714 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 52a8fd3c834a4137956a86ad4f198f0f
W20251025 07:52:24.719910 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 70c3a36dda7b4919be20390f5797d09c has no consensus state
W20251025 07:52:24.720018 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 20e4178c16bd4f18b1c6bc969f6a0c6c has no consensus state
I20251025 07:52:24.720501 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_b1c43f4e0ede4adcba8b010f04bdb72a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:24.720618 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_349ac535104c4b4dbaaf963de2640242"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:24.721122 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1c43f4e0ede4adcba8b010f04bdb72a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:24.721794 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 52a8fd3c834a4137956a86ad4f198f0f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:56590
W20251025 07:52:24.721122 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_349ac535104c4b4dbaaf963de2640242 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:24.721985 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_f597fa2d07da4cb48dcddfa94ee440bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:24.722712 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f597fa2d07da4cb48dcddfa94ee440bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:24.724095 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet d709aa500f8b422499c30843acd1a5ae
I20251025 07:52:24.724328 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet 52a8fd3c834a4137956a86ad4f198f0f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:24.724632 1170 tablet_replica.cc:333] T 52a8fd3c834a4137956a86ad4f198f0f P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:24.724917 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_14cbf20f7ff449499612663b54f0acd9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:24.725123 1170 raft_consensus.cc:2243] T 52a8fd3c834a4137956a86ad4f198f0f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:24.725497 1170 raft_consensus.cc:2272] T 52a8fd3c834a4137956a86ad4f198f0f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:24.725539 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14cbf20f7ff449499612663b54f0acd9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:24.726977 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3be8063aee794d2abcfb1d6e24209c06 has no consensus state
I20251025 07:52:24.727237 1170 ts_tablet_manager.cc:1916] T 52a8fd3c834a4137956a86ad4f198f0f P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:24.727401 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_322b6ca99f7142a8a5ec18efe56d14b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:24.727806 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 52a8fd3c834a4137956a86ad4f198f0f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:48830
W20251025 07:52:24.727986 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_322b6ca99f7142a8a5ec18efe56d14b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:24.728713 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet aac844f429fc4328a261a326181f6c35
I20251025 07:52:24.729152 518 catalog_manager.cc:6719] ReplaceTablet: tablet aac844f429fc4328a261a326181f6c35 deleted and replaced by tablet 70c3a36dda7b4919be20390f5797d09c
I20251025 07:52:24.729506 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_1eb640b2196849a8b0136b390deea1c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:24.729902 1176 raft_consensus.cc:493] T 668f7073de1c41c6882cf15e65d90e46 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:24.730099 1176 raft_consensus.cc:515] T 668f7073de1c41c6882cf15e65d90e46 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.730404 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_f3ff22d1c2824deca3f8267fc3768f28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:24.730967 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3ff22d1c2824deca3f8267fc3768f28 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:24.731981 1176 leader_election.cc:290] T 668f7073de1c41c6882cf15e65d90e46 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:24.732345 756 tablet_service.cc:1552] Processing DeleteTablet for tablet aac844f429fc4328a261a326181f6c35 with delete_type TABLET_DATA_DELETED (replaced by tablet 70c3a36dda7b4919be20390f5797d09c) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:24.732628 755 tablet_service.cc:1552] Processing DeleteTablet for tablet d709aa500f8b422499c30843acd1a5ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:24.732900 1171 tablet_replica.cc:333] T 52a8fd3c834a4137956a86ad4f198f0f P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:24.733168 1171 raft_consensus.cc:2243] T 52a8fd3c834a4137956a86ad4f198f0f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:24.733381 1171 raft_consensus.cc:2272] T 52a8fd3c834a4137956a86ad4f198f0f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:24.733989 755 tablet_service.cc:1552] Processing DeleteTablet for tablet aac844f429fc4328a261a326181f6c35 with delete_type TABLET_DATA_DELETED (replaced by tablet 70c3a36dda7b4919be20390f5797d09c) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:24.737627 893 tablet_service.cc:1552] Processing DeleteTablet for tablet aac844f429fc4328a261a326181f6c35 with delete_type TABLET_DATA_DELETED (replaced by tablet 70c3a36dda7b4919be20390f5797d09c) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:24.737841 894 tablet_service.cc:1552] Processing DeleteTablet for tablet d709aa500f8b422499c30843acd1a5ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:24.738535 1171 ts_tablet_manager.cc:1916] T 52a8fd3c834a4137956a86ad4f198f0f P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:24.745493 1183 raft_consensus.cc:493] T f05afed029d446ea9eb0073642c7312d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:24.745682 1183 raft_consensus.cc:515] T f05afed029d446ea9eb0073642c7312d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:24.746495 1183 leader_election.cc:290] T f05afed029d446ea9eb0073642c7312d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:24.747258 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f05afed029d446ea9eb0073642c7312d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:24.747589 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f05afed029d446ea9eb0073642c7312d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
W20251025 07:52:24.747802 850 leader_election.cc:343] T f05afed029d446ea9eb0073642c7312d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:24.748073 847 leader_election.cc:343] T f05afed029d446ea9eb0073642c7312d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:24.748204 847 leader_election.cc:304] T f05afed029d446ea9eb0073642c7312d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:24.748580 1183 raft_consensus.cc:2749] T f05afed029d446ea9eb0073642c7312d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:24.751031 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1eb640b2196849a8b0136b390deea1c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:24.758591 1171 ts_tablet_manager.cc:1929] T 52a8fd3c834a4137956a86ad4f198f0f P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:24.758870 1171 log.cc:1199] T 52a8fd3c834a4137956a86ad4f198f0f P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/52a8fd3c834a4137956a86ad4f198f0f
I20251025 07:52:24.760073 1171 ts_tablet_manager.cc:1950] T 52a8fd3c834a4137956a86ad4f198f0f P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:24.761147 1170 ts_tablet_manager.cc:1929] T 52a8fd3c834a4137956a86ad4f198f0f P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:24.761337 1170 log.cc:1199] T 52a8fd3c834a4137956a86ad4f198f0f P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/52a8fd3c834a4137956a86ad4f198f0f
I20251025 07:52:24.761817 1170 ts_tablet_manager.cc:1950] T 52a8fd3c834a4137956a86ad4f198f0f P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:24.762800 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 52a8fd3c834a4137956a86ad4f198f0f (table default.Table_02b9c6449126452b9694503a3d6ad253 [id=68d633fa085b416ab906780b71edf145]) successfully deleted
I20251025 07:52:24.763649 1171 tablet_replica.cc:333] T aac844f429fc4328a261a326181f6c35 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:24.764245 1171 raft_consensus.cc:2243] T aac844f429fc4328a261a326181f6c35 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:24.764890 1171 raft_consensus.cc:2272] T aac844f429fc4328a261a326181f6c35 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:24.766009 1171 ts_tablet_manager.cc:1916] T aac844f429fc4328a261a326181f6c35 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:24.780170 1172 ts_tablet_manager.cc:1929] T 668f7073de1c41c6882cf15e65d90e46 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:24.780467 1172 log.cc:1199] T 668f7073de1c41c6882cf15e65d90e46 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/668f7073de1c41c6882cf15e65d90e46
I20251025 07:52:24.780957 1172 ts_tablet_manager.cc:1950] T 668f7073de1c41c6882cf15e65d90e46 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:24.781795 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_70998787ec5746b9bb56f6e5e642898e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:24.782464 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_929d091973a047ad9123baa6c4b317aa" } new_table_name: "default.Table_6ca844d1e78c4f19939985aec32a8afd" modify_external_catalogs: true
I20251025 07:52:24.783361 751 tablet_service.cc:1505] Processing CreateTablet for tablet b946855345bc473aadd00cc4db5b7cd8 (DEFAULT_TABLE table=default.Table_929d091973a047ad9123baa6c4b317aa [id=1e933863244e4792b4c1b4d6d5c674b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.783778 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b946855345bc473aadd00cc4db5b7cd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.785851 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_8a06efe638604f20bda568cdb8382a48" } modify_external_catalogs: true
I20251025 07:52:24.786018 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_8a06efe638604f20bda568cdb8382a48" } modify_external_catalogs: true
I20251025 07:52:24.786420 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_1437bb6ce9154a15b1817a029151c71a" } modify_external_catalogs: true
I20251025 07:52:24.786551 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_1437bb6ce9154a15b1817a029151c71a" } modify_external_catalogs: true
I20251025 07:52:24.786788 892 tablet_service.cc:1505] Processing CreateTablet for tablet b946855345bc473aadd00cc4db5b7cd8 (DEFAULT_TABLE table=default.Table_929d091973a047ad9123baa6c4b317aa [id=1e933863244e4792b4c1b4d6d5c674b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.787110 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_e0e89ced2ea94c139ed635849e78cb35"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:24.787263 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b946855345bc473aadd00cc4db5b7cd8. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:24.787709 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0e89ced2ea94c139ed635849e78cb35 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:24.788157 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 52a8fd3c834a4137956a86ad4f198f0f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:48830
W20251025 07:52:24.782473 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70998787ec5746b9bb56f6e5e642898e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:24.794157 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 52a8fd3c834a4137956a86ad4f198f0f (table default.Table_02b9c6449126452b9694503a3d6ad253 [id=68d633fa085b416ab906780b71edf145]) successfully deleted
I20251025 07:52:24.795729 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet d709aa500f8b422499c30843acd1a5ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:24.796084 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet f05afed029d446ea9eb0073642c7312d with delete_type TABLET_DATA_DELETED (replaced by tablet 3be8063aee794d2abcfb1d6e24209c06) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:24.796403 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet aac844f429fc4328a261a326181f6c35 with delete_type TABLET_DATA_DELETED (replaced by tablet 70c3a36dda7b4919be20390f5797d09c) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:24.796078 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 243825c6dd2d4461a01edec88f33f6ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:24.797763 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 668f7073de1c41c6882cf15e65d90e46 (table default.Table_e29bb32f37634fbc8b3a87440b608151 [id=027c96a19c37427d94df4f30880757d4]) successfully deleted
I20251025 07:52:24.799875 1172 tablet_replica.cc:333] T f05afed029d446ea9eb0073642c7312d P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:24.803063 1172 raft_consensus.cc:2243] T f05afed029d446ea9eb0073642c7312d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:24.803326 1172 raft_consensus.cc:2272] T f05afed029d446ea9eb0073642c7312d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:24.808199 1172 ts_tablet_manager.cc:1916] T f05afed029d446ea9eb0073642c7312d P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:24.808890 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b946855345bc473aadd00cc4db5b7cd8 consensus state has no leader
I20251025 07:52:24.811221 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet ef3820c3d739476fad3d83d3aa658742 with delete_type TABLET_DATA_DELETED (replaced by tablet d709aa500f8b422499c30843acd1a5ae) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:24.811916 755 tablet_service.cc:1505] Processing CreateTablet for tablet a02b627401a64e1583f9e6123ef1be6f (DEFAULT_TABLE table=default.Table_1437bb6ce9154a15b1817a029151c71a [id=9fd173d153294521a4f14c390b316594]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.812131 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 668f7073de1c41c6882cf15e65d90e46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:24.812371 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a02b627401a64e1583f9e6123ef1be6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.812507 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 503d0732dd094099889fb56a747bbd33 with delete_type TABLET_DATA_DELETED (replaced by tablet 20e4178c16bd4f18b1c6bc969f6a0c6c) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:24.812880 1170 tablet_replica.cc:333] T 668f7073de1c41c6882cf15e65d90e46 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:24.813063 1170 raft_consensus.cc:2243] T 668f7073de1c41c6882cf15e65d90e46 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:24.813216 1170 raft_consensus.cc:2272] T 668f7073de1c41c6882cf15e65d90e46 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:24.813648 892 tablet_service.cc:1505] Processing CreateTablet for tablet a02b627401a64e1583f9e6123ef1be6f (DEFAULT_TABLE table=default.Table_1437bb6ce9154a15b1817a029151c71a [id=9fd173d153294521a4f14c390b316594]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.813719 756 tablet_service.cc:1505] Processing CreateTablet for tablet 70c3a36dda7b4919be20390f5797d09c (DEFAULT_TABLE table=default.Table_42908396d76f4df8b63e3ce11dcc2dfb [id=5e424c1c1c7c470f8640c7a7a571a11d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.813967 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a02b627401a64e1583f9e6123ef1be6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.814095 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70c3a36dda7b4919be20390f5797d09c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.814599 1027 tablet_service.cc:1505] Processing CreateTablet for tablet b946855345bc473aadd00cc4db5b7cd8 (DEFAULT_TABLE table=default.Table_929d091973a047ad9123baa6c4b317aa [id=1e933863244e4792b4c1b4d6d5c674b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.814812 1170 ts_tablet_manager.cc:1916] T 668f7073de1c41c6882cf15e65d90e46 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:24.814958 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b946855345bc473aadd00cc4db5b7cd8. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:24.815896 959 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:37611 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.0.104.60:37611 (error 108)
W20251025 07:52:24.817005 710 connection.cc:537] client connection to 127.0.104.60:37611 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 07:52:24.817622 821 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:37611 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251025 07:52:24.820459 1167 tablet_bootstrap.cc:492] T 503d0732dd094099889fb56a747bbd33 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:24.820668 1167 ts_tablet_manager.cc:1403] T 503d0732dd094099889fb56a747bbd33 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.118s user 0.030s sys 0.065s
I20251025 07:52:24.821643 1167 raft_consensus.cc:359] T 503d0732dd094099889fb56a747bbd33 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.821924 1167 raft_consensus.cc:385] T 503d0732dd094099889fb56a747bbd33 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:24.821882 889 tablet_service.cc:1505] Processing CreateTablet for tablet 70c3a36dda7b4919be20390f5797d09c (DEFAULT_TABLE table=default.Table_42908396d76f4df8b63e3ce11dcc2dfb [id=5e424c1c1c7c470f8640c7a7a571a11d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.822073 1167 raft_consensus.cc:740] T 503d0732dd094099889fb56a747bbd33 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:24.822292 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70c3a36dda7b4919be20390f5797d09c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.822314 1167 consensus_queue.cc:260] T 503d0732dd094099889fb56a747bbd33 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.822435 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 70c3a36dda7b4919be20390f5797d09c (DEFAULT_TABLE table=default.Table_42908396d76f4df8b63e3ce11dcc2dfb [id=5e424c1c1c7c470f8640c7a7a571a11d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.822770 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70c3a36dda7b4919be20390f5797d09c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.822781 1167 ts_tablet_manager.cc:1434] T 503d0732dd094099889fb56a747bbd33 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251025 07:52:24.823071 1167 tablet_bootstrap.cc:492] T ef3820c3d739476fad3d83d3aa658742 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:24.823675 891 tablet_service.cc:1505] Processing CreateTablet for tablet 3be8063aee794d2abcfb1d6e24209c06 (DEFAULT_TABLE table=default.Table_c96e62972b3c4ee48b6c3ea90831a34f [id=e7a5b463914840c980a306b7d46ea871]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.824074 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3be8063aee794d2abcfb1d6e24209c06. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.824710 1167 tablet_bootstrap.cc:654] T ef3820c3d739476fad3d83d3aa658742 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:24.822321 756 tablet_service.cc:1505] Processing CreateTablet for tablet 3be8063aee794d2abcfb1d6e24209c06 (DEFAULT_TABLE table=default.Table_c96e62972b3c4ee48b6c3ea90831a34f [id=e7a5b463914840c980a306b7d46ea871]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.825853 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3be8063aee794d2abcfb1d6e24209c06. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.828156 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_ae4075623a14448aaba9f22ae53d0755" } new_table_name: "default.Table_5f94a1aa9fac44ae8a90d3554215d598" modify_external_catalogs: true
W20251025 07:52:24.832616 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet d709aa500f8b422499c30843acd1a5ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d709aa500f8b422499c30843acd1a5ae already in progress: creating tablet
I20251025 07:52:24.832950 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_ad812bcf86a04b0b89a3f0bc961062a9" } modify_external_catalogs: true
I20251025 07:52:24.833279 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_ad812bcf86a04b0b89a3f0bc961062a9" } modify_external_catalogs: true
W20251025 07:52:24.833559 983 connection.cc:537] client connection to 127.0.104.60:37611 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251025 07:52:24.833751 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 503d0732dd094099889fb56a747bbd33 with delete_type TABLET_DATA_DELETED (replaced by tablet 20e4178c16bd4f18b1c6bc969f6a0c6c) from {username='slave'} at 127.0.0.1:48830
W20251025 07:52:24.833979 1094 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:37611 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 07:52:24.833076 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet f05afed029d446ea9eb0073642c7312d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f05afed029d446ea9eb0073642c7312d already in progress: creating tablet
W20251025 07:52:24.834906 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet aac844f429fc4328a261a326181f6c35 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aac844f429fc4328a261a326181f6c35 already in progress: creating tablet
W20251025 07:52:24.835130 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 243825c6dd2d4461a01edec88f33f6ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 243825c6dd2d4461a01edec88f33f6ee already in progress: creating tablet
I20251025 07:52:24.835110 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 3be8063aee794d2abcfb1d6e24209c06 (DEFAULT_TABLE table=default.Table_c96e62972b3c4ee48b6c3ea90831a34f [id=e7a5b463914840c980a306b7d46ea871]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:24.835359 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet ef3820c3d739476fad3d83d3aa658742 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef3820c3d739476fad3d83d3aa658742 already in progress: creating tablet
I20251025 07:52:24.835377 755 tablet_service.cc:1505] Processing CreateTablet for tablet 20e4178c16bd4f18b1c6bc969f6a0c6c (DEFAULT_TABLE table=default.Table_533c8546736848ea9ec200f63386630c [id=a5c73cb1e5a349ca8675f4ba4cb8a82d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.835678 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3be8063aee794d2abcfb1d6e24209c06. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.835963 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20e4178c16bd4f18b1c6bc969f6a0c6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.838430 1027 tablet_service.cc:1505] Processing CreateTablet for tablet a02b627401a64e1583f9e6123ef1be6f (DEFAULT_TABLE table=default.Table_1437bb6ce9154a15b1817a029151c71a [id=9fd173d153294521a4f14c390b316594]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.839787 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a02b627401a64e1583f9e6123ef1be6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.839453 1169 tablet_bootstrap.cc:492] T aac844f429fc4328a261a326181f6c35 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:24.842429 1169 ts_tablet_manager.cc:1403] T aac844f429fc4328a261a326181f6c35 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.222s user 0.008s sys 0.018s
I20251025 07:52:24.843417 1169 raft_consensus.cc:359] T aac844f429fc4328a261a326181f6c35 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.843719 1169 raft_consensus.cc:385] T aac844f429fc4328a261a326181f6c35 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:24.843848 1169 raft_consensus.cc:740] T aac844f429fc4328a261a326181f6c35 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:24.844151 1169 consensus_queue.cc:260] T aac844f429fc4328a261a326181f6c35 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.848222 1169 ts_tablet_manager.cc:1434] T aac844f429fc4328a261a326181f6c35 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251025 07:52:24.853753 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet d684da8134ce480693f470acdac176d4
I20251025 07:52:24.854982 1166 tablet_bootstrap.cc:492] T 243825c6dd2d4461a01edec88f33f6ee P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:24.855142 1166 ts_tablet_manager.cc:1403] T 243825c6dd2d4461a01edec88f33f6ee P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.166s user 0.019s sys 0.010s
I20251025 07:52:24.855788 756 tablet_service.cc:1552] Processing DeleteTablet for tablet d684da8134ce480693f470acdac176d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:24.856021 1166 raft_consensus.cc:359] T 243825c6dd2d4461a01edec88f33f6ee P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:24.856205 1166 raft_consensus.cc:385] T 243825c6dd2d4461a01edec88f33f6ee P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:24.856302 1166 raft_consensus.cc:740] T 243825c6dd2d4461a01edec88f33f6ee P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:24.856564 1166 consensus_queue.cc:260] T 243825c6dd2d4461a01edec88f33f6ee P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:24.858819 1166 ts_tablet_manager.cc:1434] T 243825c6dd2d4461a01edec88f33f6ee P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
W20251025 07:52:24.858987 982 leader_election.cc:343] T 668f7073de1c41c6882cf15e65d90e46 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: 668f7073de1c41c6882cf15e65d90e46
I20251025 07:52:24.859503 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet a02b627401a64e1583f9e6123ef1be6f
W20251025 07:52:24.860401 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 89bfb7e2f051454aae0af5755a79ca58 consensus state has no leader
W20251025 07:52:24.861297 1185 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 07:52:24.816382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:37611 (local address 127.0.104.2:42265)
1025 07:52:24.816588 (+ 206us) negotiation.cc:107] Waiting for socket to connect
1025 07:52:24.816607 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1025 07:52:24.816723 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1025 07:52:24.860292 (+ 43569us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:37611: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":110}
I20251025 07:52:24.863231 1169 tablet_bootstrap.cc:492] T 503d0732dd094099889fb56a747bbd33 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:24.863619 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 54cb4953ff264c13a0cc730b4c452ecc
I20251025 07:52:24.864109 756 tablet_service.cc:1552] Processing DeleteTablet for tablet a02b627401a64e1583f9e6123ef1be6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:24.865127 1169 tablet_bootstrap.cc:654] T 503d0732dd094099889fb56a747bbd33 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:24.866868 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 20e4178c16bd4f18b1c6bc969f6a0c6c (DEFAULT_TABLE table=default.Table_533c8546736848ea9ec200f63386630c [id=a5c73cb1e5a349ca8675f4ba4cb8a82d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.867292 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20e4178c16bd4f18b1c6bc969f6a0c6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.868952 1166 tablet_bootstrap.cc:492] T 5680e1adaf3f4b7f8c5e05523dd0950a P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:24.870540 1171 ts_tablet_manager.cc:1929] T aac844f429fc4328a261a326181f6c35 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:24.870718 1171 log.cc:1199] T aac844f429fc4328a261a326181f6c35 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/aac844f429fc4328a261a326181f6c35
I20251025 07:52:24.871150 1171 ts_tablet_manager.cc:1950] T aac844f429fc4328a261a326181f6c35 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:24.871217 1167 tablet_bootstrap.cc:492] T ef3820c3d739476fad3d83d3aa658742 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:24.871423 1167 ts_tablet_manager.cc:1403] T ef3820c3d739476fad3d83d3aa658742 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.032s
I20251025 07:52:24.871850 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 54cb4953ff264c13a0cc730b4c452ecc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:24.872308 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet aac844f429fc4328a261a326181f6c35 (table default.Table_42908396d76f4df8b63e3ce11dcc2dfb [id=5e424c1c1c7c470f8640c7a7a571a11d]) successfully deleted
W20251025 07:52:24.872725 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet d709aa500f8b422499c30843acd1a5ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d709aa500f8b422499c30843acd1a5ae already in progress: creating tablet
I20251025 07:52:24.872238 1167 raft_consensus.cc:359] T ef3820c3d739476fad3d83d3aa658742 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.872936 1167 raft_consensus.cc:385] T ef3820c3d739476fad3d83d3aa658742 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:24.873018 1167 raft_consensus.cc:740] T ef3820c3d739476fad3d83d3aa658742 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
W20251025 07:52:24.872964 510 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet aac844f429fc4328a261a326181f6c35 because the tablet was not found. No further retry: Not found: Tablet not found: aac844f429fc4328a261a326181f6c35
W20251025 07:52:24.873230 510 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 52a8fd3c834a4137956a86ad4f198f0f because the tablet was not found. No further retry: Not found: Tablet not found: 52a8fd3c834a4137956a86ad4f198f0f
I20251025 07:52:24.873230 1167 consensus_queue.cc:260] T ef3820c3d739476fad3d83d3aa658742 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.872674 1171 tablet_replica.cc:333] T 503d0732dd094099889fb56a747bbd33 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:24.873569 1171 raft_consensus.cc:2243] T 503d0732dd094099889fb56a747bbd33 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:24.875463 1171 raft_consensus.cc:2272] T 503d0732dd094099889fb56a747bbd33 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:24.876026 1167 ts_tablet_manager.cc:1434] T ef3820c3d739476fad3d83d3aa658742 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251025 07:52:24.876341 1167 tablet_bootstrap.cc:492] T d709aa500f8b422499c30843acd1a5ae P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:24.877892 1167 tablet_bootstrap.cc:654] T d709aa500f8b422499c30843acd1a5ae P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:24.878727 525 master_service.cc:954] ReplaceTablet: received request to replace tablet 3be8063aee794d2abcfb1d6e24209c06 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:24.881821 1171 ts_tablet_manager.cc:1916] T 503d0732dd094099889fb56a747bbd33 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:24.881870 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet a02b627401a64e1583f9e6123ef1be6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:24.882161 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet aac844f429fc4328a261a326181f6c35 with delete_type TABLET_DATA_DELETED (replaced by tablet 70c3a36dda7b4919be20390f5797d09c) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:24.882289 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 54cb4953ff264c13a0cc730b4c452ecc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:24.882836 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet d684da8134ce480693f470acdac176d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:24.883057 1166 tablet_bootstrap.cc:654] T 5680e1adaf3f4b7f8c5e05523dd0950a P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:24.888373 1170 ts_tablet_manager.cc:1929] T 668f7073de1c41c6882cf15e65d90e46 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:24.888514 1170 log.cc:1199] T 668f7073de1c41c6882cf15e65d90e46 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/668f7073de1c41c6882cf15e65d90e46
W20251025 07:52:24.889060 824 tablet.cc:2378] T ef3820c3d739476fad3d83d3aa658742 P 23eeffd3de254ce18db5febfde971c6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251025 07:52:24.890013 511 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:37611, user_credentials={real_user=slave}} blocked reactor thread for 49846.3us
I20251025 07:52:24.891355 524 master_service.cc:954] ReplaceTablet: received request to replace tablet d709aa500f8b422499c30843acd1a5ae from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:24.892871 520 master_service.cc:954] ReplaceTablet: received request to replace tablet 503d0732dd094099889fb56a747bbd33 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:24.892935 1172 ts_tablet_manager.cc:1929] T f05afed029d446ea9eb0073642c7312d P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:24.893050 1172 log.cc:1199] T f05afed029d446ea9eb0073642c7312d P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/f05afed029d446ea9eb0073642c7312d
I20251025 07:52:24.893448 1172 ts_tablet_manager.cc:1950] T f05afed029d446ea9eb0073642c7312d P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:24.893903 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 3be8063aee794d2abcfb1d6e24209c06
I20251025 07:52:24.894076 891 tablet_service.cc:1552] Processing DeleteTablet for tablet d684da8134ce480693f470acdac176d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:24.894361 890 tablet_service.cc:1552] Processing DeleteTablet for tablet a02b627401a64e1583f9e6123ef1be6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:24.894541 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 243825c6dd2d4461a01edec88f33f6ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:24.894564 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 3be8063aee794d2abcfb1d6e24209c06 with delete_type TABLET_DATA_DELETED (replaced by tablet 2947502815d742e89cacb474b7fbb249) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:24.894706 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 54cb4953ff264c13a0cc730b4c452ecc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:24.895259 889 tablet_service.cc:1505] Processing CreateTablet for tablet 20e4178c16bd4f18b1c6bc969f6a0c6c (DEFAULT_TABLE table=default.Table_533c8546736848ea9ec200f63386630c [id=a5c73cb1e5a349ca8675f4ba4cb8a82d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.895767 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20e4178c16bd4f18b1c6bc969f6a0c6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.896746 525 catalog_manager.cc:6719] ReplaceTablet: tablet 3be8063aee794d2abcfb1d6e24209c06 deleted and replaced by tablet 2947502815d742e89cacb474b7fbb249
I20251025 07:52:24.899168 751 tablet_service.cc:1505] Processing CreateTablet for tablet 785d0ce85b234826ba8021fbd182bf3c (DEFAULT_TABLE table=default.Table_e0e89ced2ea94c139ed635849e78cb35 [id=807c428924b74fde847c2becad78d4ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.899566 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 785d0ce85b234826ba8021fbd182bf3c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:24.900019 1195 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 07:52:24.818109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:37611 (local address 127.0.104.1:41723)
1025 07:52:24.898760 (+ 80651us) negotiation.cc:107] Waiting for socket to connect
1025 07:52:24.898944 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:37611: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":80521,"thread_start_us":80416,"threads_started":1}
I20251025 07:52:24.900964 1169 tablet_bootstrap.cc:492] T 503d0732dd094099889fb56a747bbd33 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:24.901113 1169 ts_tablet_manager.cc:1403] T 503d0732dd094099889fb56a747bbd33 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.001s
I20251025 07:52:24.901888 1169 raft_consensus.cc:359] T 503d0732dd094099889fb56a747bbd33 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.902102 1169 raft_consensus.cc:385] T 503d0732dd094099889fb56a747bbd33 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:24.902236 1169 raft_consensus.cc:740] T 503d0732dd094099889fb56a747bbd33 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:24.902451 1169 consensus_queue.cc:260] T 503d0732dd094099889fb56a747bbd33 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.902906 1169 ts_tablet_manager.cc:1434] T 503d0732dd094099889fb56a747bbd33 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251025 07:52:24.903185 1169 tablet_bootstrap.cc:492] T ef3820c3d739476fad3d83d3aa658742 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:24.905169 1167 tablet_bootstrap.cc:492] T d709aa500f8b422499c30843acd1a5ae P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:24.905337 1167 ts_tablet_manager.cc:1403] T d709aa500f8b422499c30843acd1a5ae P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.004s
I20251025 07:52:24.905395 1172 tablet_replica.cc:333] T 503d0732dd094099889fb56a747bbd33 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:24.905594 1172 raft_consensus.cc:2243] T 503d0732dd094099889fb56a747bbd33 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:24.905743 1172 raft_consensus.cc:2272] T 503d0732dd094099889fb56a747bbd33 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:24.906363 1167 raft_consensus.cc:359] T d709aa500f8b422499c30843acd1a5ae P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.906553 1167 raft_consensus.cc:385] T d709aa500f8b422499c30843acd1a5ae P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:24.906641 1167 raft_consensus.cc:740] T d709aa500f8b422499c30843acd1a5ae P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:24.906832 1167 consensus_queue.cc:260] T d709aa500f8b422499c30843acd1a5ae P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.907464 1169 tablet_bootstrap.cc:654] T ef3820c3d739476fad3d83d3aa658742 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:24.907874 755 tablet_service.cc:1505] Processing CreateTablet for tablet 9cf88862e0934634b3dd172e6b40a0ba (DEFAULT_TABLE table=default.Table_1eb640b2196849a8b0136b390deea1c2 [id=ec85197b425f466ba7cd69e27caf6c93]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.908250 1167 ts_tablet_manager.cc:1434] T d709aa500f8b422499c30843acd1a5ae P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251025 07:52:24.908314 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cf88862e0934634b3dd172e6b40a0ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.908469 1170 ts_tablet_manager.cc:1950] T 668f7073de1c41c6882cf15e65d90e46 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:24.909008 1170 tablet_replica.cc:333] T 503d0732dd094099889fb56a747bbd33 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:24.909164 1170 raft_consensus.cc:2243] T 503d0732dd094099889fb56a747bbd33 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:24.909348 1170 raft_consensus.cc:2272] T 503d0732dd094099889fb56a747bbd33 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:24.909423 1167 tablet_bootstrap.cc:492] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:24.910637 1172 ts_tablet_manager.cc:1916] T 503d0732dd094099889fb56a747bbd33 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:24.911192 1167 tablet_bootstrap.cc:654] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:24.915621 1170 ts_tablet_manager.cc:1916] T 503d0732dd094099889fb56a747bbd33 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:24.916663 1167 tablet_bootstrap.cc:492] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:24.916818 1167 ts_tablet_manager.cc:1403] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20251025 07:52:24.916893 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 668f7073de1c41c6882cf15e65d90e46 (table default.Table_e29bb32f37634fbc8b3a87440b608151 [id=027c96a19c37427d94df4f30880757d4]) successfully deleted
I20251025 07:52:24.917616 1167 raft_consensus.cc:359] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:24.917827 1167 raft_consensus.cc:385] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:24.917912 1167 raft_consensus.cc:740] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:24.918040 751 tablet_service.cc:1505] Processing CreateTablet for tablet 2fac7b819e2642378e0d3f9230d0259c (DEFAULT_TABLE table=default.Table_f3ff22d1c2824deca3f8267fc3768f28 [id=6605e11e11804d408a6a4a9d403dde95]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.918253 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 785d0ce85b234826ba8021fbd182bf3c (DEFAULT_TABLE table=default.Table_e0e89ced2ea94c139ed635849e78cb35 [id=807c428924b74fde847c2becad78d4ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.918520 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fac7b819e2642378e0d3f9230d0259c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.918610 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 785d0ce85b234826ba8021fbd182bf3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.919862 1167 consensus_queue.cc:260] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:24.922601 1167 ts_tablet_manager.cc:1434] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251025 07:52:24.922787 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 9cf88862e0934634b3dd172e6b40a0ba (DEFAULT_TABLE table=default.Table_1eb640b2196849a8b0136b390deea1c2 [id=ec85197b425f466ba7cd69e27caf6c93]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.923211 1166 tablet_bootstrap.cc:492] T 5680e1adaf3f4b7f8c5e05523dd0950a P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:24.923278 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cf88862e0934634b3dd172e6b40a0ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.924293 1167 tablet_bootstrap.cc:492] T 243825c6dd2d4461a01edec88f33f6ee P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:24.924544 1166 ts_tablet_manager.cc:1403] T 5680e1adaf3f4b7f8c5e05523dd0950a P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.056s user 0.005s sys 0.001s
I20251025 07:52:24.925365 1166 raft_consensus.cc:359] T 5680e1adaf3f4b7f8c5e05523dd0950a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:24.925998 1167 tablet_bootstrap.cc:654] T 243825c6dd2d4461a01edec88f33f6ee P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:24.926252 1166 raft_consensus.cc:385] T 5680e1adaf3f4b7f8c5e05523dd0950a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:24.926463 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 3be8063aee794d2abcfb1d6e24209c06 with delete_type TABLET_DATA_DELETED (replaced by tablet 2947502815d742e89cacb474b7fbb249) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:24.926673 1166 raft_consensus.cc:740] T 5680e1adaf3f4b7f8c5e05523dd0950a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:24.926877 755 tablet_service.cc:1505] Processing CreateTablet for tablet 576c51d9a22d45f6a3642a7296b5bf23 (DEFAULT_TABLE table=default.Table_14cbf20f7ff449499612663b54f0acd9 [id=e9343178deb142339649b0cc6d26a9af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.927210 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 576c51d9a22d45f6a3642a7296b5bf23. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.927181 1166 consensus_queue.cc:260] T 5680e1adaf3f4b7f8c5e05523dd0950a P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:24.932919 1166 ts_tablet_manager.cc:1434] T 5680e1adaf3f4b7f8c5e05523dd0950a P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
W20251025 07:52:24.933878 983 leader_election.cc:343] T 668f7073de1c41c6882cf15e65d90e46 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 668f7073de1c41c6882cf15e65d90e46
I20251025 07:52:24.934473 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 2fac7b819e2642378e0d3f9230d0259c (DEFAULT_TABLE table=default.Table_f3ff22d1c2824deca3f8267fc3768f28 [id=6605e11e11804d408a6a4a9d403dde95]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.935740 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fac7b819e2642378e0d3f9230d0259c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.936443 756 tablet_service.cc:1505] Processing CreateTablet for tablet 4829ec41391145cc84327d60048d9b0c (DEFAULT_TABLE table=default.Table_f597fa2d07da4cb48dcddfa94ee440bc [id=94a9a4e752b547afb3d4d0b6806bdb54]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.936582 1166 tablet_bootstrap.cc:492] T 14fd790ba7054c1fb09cdfb19b001987 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:24.936822 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4829ec41391145cc84327d60048d9b0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.938225 1166 tablet_bootstrap.cc:654] T 14fd790ba7054c1fb09cdfb19b001987 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:24.938552 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 576c51d9a22d45f6a3642a7296b5bf23 (DEFAULT_TABLE table=default.Table_14cbf20f7ff449499612663b54f0acd9 [id=e9343178deb142339649b0cc6d26a9af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.939020 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 576c51d9a22d45f6a3642a7296b5bf23. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.942224 983 leader_election.cc:304] T 668f7073de1c41c6882cf15e65d90e46 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
W20251025 07:52:24.942752 983 raft_consensus.cc:2686] T 668f7073de1c41c6882cf15e65d90e46 P db42be7ad3c741a0aa427f27e245993b: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251025 07:52:24.945154 756 tablet_service.cc:1505] Processing CreateTablet for tablet 299572b2dbe244918dcebc5ea5f3c45a (DEFAULT_TABLE table=default.Table_349ac535104c4b4dbaaf963de2640242 [id=840b3ffcb44144548843457c48f46380]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.945427 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 4829ec41391145cc84327d60048d9b0c (DEFAULT_TABLE table=default.Table_f597fa2d07da4cb48dcddfa94ee440bc [id=94a9a4e752b547afb3d4d0b6806bdb54]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.945627 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 299572b2dbe244918dcebc5ea5f3c45a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.945573 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 299572b2dbe244918dcebc5ea5f3c45a (DEFAULT_TABLE table=default.Table_349ac535104c4b4dbaaf963de2640242 [id=840b3ffcb44144548843457c48f46380]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.945933 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4829ec41391145cc84327d60048d9b0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.946265 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 299572b2dbe244918dcebc5ea5f3c45a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.953732 754 tablet_service.cc:1505] Processing CreateTablet for tablet 713ae1dd444544bdabdd4a771e109d90 (DEFAULT_TABLE table=default.Table_70998787ec5746b9bb56f6e5e642898e [id=034108fa919f469bb53041fd03a1b5d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.953701 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_6cd1984401d2468e9494afe02d87f74f" } new_table_name: "default.Table_31d7f6f73d6e4123b1ff1df86ff62205" modify_external_catalogs: true
I20251025 07:52:24.954088 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 713ae1dd444544bdabdd4a771e109d90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.955045 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_70998787ec5746b9bb56f6e5e642898e" } modify_external_catalogs: true
I20251025 07:52:24.955210 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_70998787ec5746b9bb56f6e5e642898e" } modify_external_catalogs: true
I20251025 07:52:24.956283 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_42908396d76f4df8b63e3ce11dcc2dfb" } modify_external_catalogs: true
I20251025 07:52:24.956450 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_42908396d76f4df8b63e3ce11dcc2dfb" } modify_external_catalogs: true
I20251025 07:52:24.959636 518 master_service.cc:954] ReplaceTablet: received request to replace tablet 20e4178c16bd4f18b1c6bc969f6a0c6c from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:24.961236 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_f597fa2d07da4cb48dcddfa94ee440bc" } new_table_name: "default.Table_974bcc124afb435285e042c78588d367" modify_external_catalogs: true
I20251025 07:52:24.963284 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 3be8063aee794d2abcfb1d6e24209c06 with delete_type TABLET_DATA_DELETED (replaced by tablet 2947502815d742e89cacb474b7fbb249) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:24.963714 752 tablet_service.cc:1552] Processing DeleteTablet for tablet ef3820c3d739476fad3d83d3aa658742 with delete_type TABLET_DATA_DELETED (replaced by tablet d709aa500f8b422499c30843acd1a5ae) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:24.963721 889 tablet_service.cc:1505] Processing CreateTablet for tablet 785d0ce85b234826ba8021fbd182bf3c (DEFAULT_TABLE table=default.Table_e0e89ced2ea94c139ed635849e78cb35 [id=807c428924b74fde847c2becad78d4ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.964082 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_322b6ca99f7142a8a5ec18efe56d14b0" } new_table_name: "default.Table_028b24c9775b49ea8adf1a62fe9df71d" modify_external_catalogs: true
I20251025 07:52:24.964154 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 785d0ce85b234826ba8021fbd182bf3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.964497 1169 tablet_bootstrap.cc:492] T ef3820c3d739476fad3d83d3aa658742 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:24.964665 1169 ts_tablet_manager.cc:1403] T ef3820c3d739476fad3d83d3aa658742 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.062s user 0.000s sys 0.003s
I20251025 07:52:24.965512 1169 raft_consensus.cc:359] T ef3820c3d739476fad3d83d3aa658742 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.965705 1169 raft_consensus.cc:385] T ef3820c3d739476fad3d83d3aa658742 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:24.965787 1169 raft_consensus.cc:740] T ef3820c3d739476fad3d83d3aa658742 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:24.966013 1169 consensus_queue.cc:260] T ef3820c3d739476fad3d83d3aa658742 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:24.970721 1169 ts_tablet_manager.cc:1434] T ef3820c3d739476fad3d83d3aa658742 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251025 07:52:24.970755 892 tablet_service.cc:1505] Processing CreateTablet for tablet 9cf88862e0934634b3dd172e6b40a0ba (DEFAULT_TABLE table=default.Table_1eb640b2196849a8b0136b390deea1c2 [id=ec85197b425f466ba7cd69e27caf6c93]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.971132 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cf88862e0934634b3dd172e6b40a0ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.972494 1171 ts_tablet_manager.cc:1929] T 503d0732dd094099889fb56a747bbd33 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:24.972641 1171 log.cc:1199] T 503d0732dd094099889fb56a747bbd33 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/503d0732dd094099889fb56a747bbd33
I20251025 07:52:24.972936 1171 ts_tablet_manager.cc:1950] T 503d0732dd094099889fb56a747bbd33 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
W20251025 07:52:24.973152 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5680e1adaf3f4b7f8c5e05523dd0950a consensus state has no leader
I20251025 07:52:24.973764 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 70c3a36dda7b4919be20390f5797d09c
I20251025 07:52:24.974413 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 713ae1dd444544bdabdd4a771e109d90 (DEFAULT_TABLE table=default.Table_70998787ec5746b9bb56f6e5e642898e [id=034108fa919f469bb53041fd03a1b5d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.974807 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 713ae1dd444544bdabdd4a771e109d90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.975031 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 503d0732dd094099889fb56a747bbd33 (table default.Table_533c8546736848ea9ec200f63386630c [id=a5c73cb1e5a349ca8675f4ba4cb8a82d]) successfully deleted
W20251025 07:52:24.975279 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet d684da8134ce480693f470acdac176d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d684da8134ce480693f470acdac176d4 already in progress: creating tablet
I20251025 07:52:24.975833 1171 tablet_replica.cc:333] T ef3820c3d739476fad3d83d3aa658742 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:24.976020 1171 raft_consensus.cc:2243] T ef3820c3d739476fad3d83d3aa658742 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:24.976182 1171 raft_consensus.cc:2272] T ef3820c3d739476fad3d83d3aa658742 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:24.977216 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 70c3a36dda7b4919be20390f5797d09c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:24.977681 893 tablet_service.cc:1505] Processing CreateTablet for tablet 2fac7b819e2642378e0d3f9230d0259c (DEFAULT_TABLE table=default.Table_f3ff22d1c2824deca3f8267fc3768f28 [id=6605e11e11804d408a6a4a9d403dde95]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:24.978047 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet a02b627401a64e1583f9e6123ef1be6f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a02b627401a64e1583f9e6123ef1be6f already in progress: creating tablet
I20251025 07:52:24.978096 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fac7b819e2642378e0d3f9230d0259c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:24.978302 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 54cb4953ff264c13a0cc730b4c452ecc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54cb4953ff264c13a0cc730b4c452ecc already in progress: creating tablet
W20251025 07:52:24.978457 511 consensus_peers.cc:597] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 -> Peer b4a999d754c941ec850c143e4e24c407 (127.0.104.60:37611): Couldn't send request to peer b4a999d754c941ec850c143e4e24c407. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:37611: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20251025 07:52:24.978552 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 3be8063aee794d2abcfb1d6e24209c06 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3be8063aee794d2abcfb1d6e24209c06 already in progress: creating tablet
I20251025 07:52:24.979034 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 713ae1dd444544bdabdd4a771e109d90
I20251025 07:52:24.979761 1169 tablet_bootstrap.cc:492] T d709aa500f8b422499c30843acd1a5ae P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:24.980067 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet f05afed029d446ea9eb0073642c7312d (table default.Table_c96e62972b3c4ee48b6c3ea90831a34f [id=e7a5b463914840c980a306b7d46ea871]) successfully deleted
I20251025 07:52:24.979996 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_1c458711d56d4ddeba1d0bc09392a734"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:24.980649 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c458711d56d4ddeba1d0bc09392a734 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:24.981595 1169 tablet_bootstrap.cc:654] T d709aa500f8b422499c30843acd1a5ae P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:24.982436 894 tablet_service.cc:1505] Processing CreateTablet for tablet 576c51d9a22d45f6a3642a7296b5bf23 (DEFAULT_TABLE table=default.Table_14cbf20f7ff449499612663b54f0acd9 [id=e9343178deb142339649b0cc6d26a9af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.982769 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 70c3a36dda7b4919be20390f5797d09c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:24.983088 890 tablet_service.cc:1505] Processing CreateTablet for tablet 4829ec41391145cc84327d60048d9b0c (DEFAULT_TABLE table=default.Table_f597fa2d07da4cb48dcddfa94ee440bc [id=94a9a4e752b547afb3d4d0b6806bdb54]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:24.983177 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4829ec41391145cc84327d60048d9b0c consensus state has no leader
I20251025 07:52:24.983457 892 tablet_service.cc:1505] Processing CreateTablet for tablet 713ae1dd444544bdabdd4a771e109d90 (DEFAULT_TABLE table=default.Table_70998787ec5746b9bb56f6e5e642898e [id=034108fa919f469bb53041fd03a1b5d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.983619 891 tablet_service.cc:1505] Processing CreateTablet for tablet 299572b2dbe244918dcebc5ea5f3c45a (DEFAULT_TABLE table=default.Table_349ac535104c4b4dbaaf963de2640242 [id=840b3ffcb44144548843457c48f46380]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:24.983827 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 20e4178c16bd4f18b1c6bc969f6a0c6c
I20251025 07:52:24.982821 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 576c51d9a22d45f6a3642a7296b5bf23. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.984261 518 catalog_manager.cc:6719] ReplaceTablet: tablet 20e4178c16bd4f18b1c6bc969f6a0c6c deleted and replaced by tablet 50433881c0eb4aa5a389b6f5b4a4337a
I20251025 07:52:24.985369 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_819aedd5270f4d6cbaaaee7c2ff5f676"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:24.985908 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_819aedd5270f4d6cbaaaee7c2ff5f676 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:24.984185 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 713ae1dd444544bdabdd4a771e109d90 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:24.986812 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 20e4178c16bd4f18b1c6bc969f6a0c6c with delete_type TABLET_DATA_DELETED (replaced by tablet 50433881c0eb4aa5a389b6f5b4a4337a) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:24.984072 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 713ae1dd444544bdabdd4a771e109d90. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:24.988324 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3d5e0dd14e474fe09993a03851f8b1f0 consensus state has no leader
I20251025 07:52:24.988659 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 299572b2dbe244918dcebc5ea5f3c45a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.984380 1166 tablet_bootstrap.cc:492] T 14fd790ba7054c1fb09cdfb19b001987 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:24.989778 1166 ts_tablet_manager.cc:1403] T 14fd790ba7054c1fb09cdfb19b001987 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.000s
I20251025 07:52:24.990149 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4829ec41391145cc84327d60048d9b0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:24.991485 524 master_service.cc:954] ReplaceTablet: received request to replace tablet 54cb4953ff264c13a0cc730b4c452ecc from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:24.975194 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 70c3a36dda7b4919be20390f5797d09c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:24.992039 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_5b6c9357453f4ece8c4e824377ab91ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:24.992303 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 20e4178c16bd4f18b1c6bc969f6a0c6c with delete_type TABLET_DATA_DELETED (replaced by tablet 50433881c0eb4aa5a389b6f5b4a4337a) from {username='slave'} at 127.0.0.1:47408
W20251025 07:52:24.992583 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b6c9357453f4ece8c4e824377ab91ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:24.993402 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_26ee7c4758284bc0bf685f326ebfd478"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:24.993856 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26ee7c4758284bc0bf685f326ebfd478 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:24.995803 1166 raft_consensus.cc:359] T 14fd790ba7054c1fb09cdfb19b001987 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:24.996177 1166 raft_consensus.cc:385] T 14fd790ba7054c1fb09cdfb19b001987 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:24.976410 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet ef3820c3d739476fad3d83d3aa658742 with delete_type TABLET_DATA_DELETED (replaced by tablet d709aa500f8b422499c30843acd1a5ae) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:24.996517 1166 raft_consensus.cc:740] T 14fd790ba7054c1fb09cdfb19b001987 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:24.997262 1166 consensus_queue.cc:260] T 14fd790ba7054c1fb09cdfb19b001987 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:24.997761 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_ebc8b70bac6a4dc3930476eab4c441b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:24.998274 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ebc8b70bac6a4dc3930476eab4c441b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:24.998247 1171 ts_tablet_manager.cc:1916] T ef3820c3d739476fad3d83d3aa658742 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:25.002297 1166 ts_tablet_manager.cc:1434] T 14fd790ba7054c1fb09cdfb19b001987 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.012s user 0.002s sys 0.001s
I20251025 07:52:25.004166 752 tablet_service.cc:1552] Processing DeleteTablet for tablet f05afed029d446ea9eb0073642c7312d with delete_type TABLET_DATA_DELETED (replaced by tablet 3be8063aee794d2abcfb1d6e24209c06) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:25.004434 753 tablet_service.cc:1552] Processing DeleteTablet for tablet d709aa500f8b422499c30843acd1a5ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:25.008797 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_e3ffa05bcb8040c79de137fb6bdfc4ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:25.009356 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3ffa05bcb8040c79de137fb6bdfc4ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:25.010404 1166 tablet_bootstrap.cc:492] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:25.013553 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_533c8546736848ea9ec200f63386630c" } new_table_name: "default.Table_95222dffdec148e39f6429c366be2b4a" modify_external_catalogs: true
I20251025 07:52:25.013607 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_7a19acf479d64317bd910c04b08a8b4f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:25.014093 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a19acf479d64317bd910c04b08a8b4f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:25.014528 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_79969e4ace6f4227b2748b59136027d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:25.014999 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79969e4ace6f4227b2748b59136027d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:25.016013 1166 tablet_bootstrap.cc:654] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:25.016026 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_f461c120f4bc4fefbaf8c486f4f910f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:25.016495 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f461c120f4bc4fefbaf8c486f4f910f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:25.017323 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 20e4178c16bd4f18b1c6bc969f6a0c6c with delete_type TABLET_DATA_DELETED (replaced by tablet 50433881c0eb4aa5a389b6f5b4a4337a) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:25.017663 1178 raft_consensus.cc:493] T d709aa500f8b422499c30843acd1a5ae P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:25.018005 1178 raft_consensus.cc:515] T d709aa500f8b422499c30843acd1a5ae P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:25.018807 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 713ae1dd444544bdabdd4a771e109d90 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:25.019122 1172 ts_tablet_manager.cc:1929] T 503d0732dd094099889fb56a747bbd33 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:25.021509 1172 log.cc:1199] T 503d0732dd094099889fb56a747bbd33 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/503d0732dd094099889fb56a747bbd33
I20251025 07:52:25.021831 1178 leader_election.cc:290] T d709aa500f8b422499c30843acd1a5ae P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:25.022490 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 713ae1dd444544bdabdd4a771e109d90 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:25.022729 1170 ts_tablet_manager.cc:1929] T 503d0732dd094099889fb56a747bbd33 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:25.022837 1170 log.cc:1199] T 503d0732dd094099889fb56a747bbd33 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/503d0732dd094099889fb56a747bbd33
I20251025 07:52:25.022042 1172 ts_tablet_manager.cc:1950] T 503d0732dd094099889fb56a747bbd33 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:25.023218 1170 ts_tablet_manager.cc:1950] T 503d0732dd094099889fb56a747bbd33 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:25.024231 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 503d0732dd094099889fb56a747bbd33 (table default.Table_533c8546736848ea9ec200f63386630c [id=a5c73cb1e5a349ca8675f4ba4cb8a82d]) successfully deleted
W20251025 07:52:25.024856 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet a02b627401a64e1583f9e6123ef1be6f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a02b627401a64e1583f9e6123ef1be6f already in progress: creating tablet
I20251025 07:52:25.025636 1170 tablet_replica.cc:333] T aac844f429fc4328a261a326181f6c35 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:25.025852 1170 raft_consensus.cc:2243] T aac844f429fc4328a261a326181f6c35 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:25.026072 1170 raft_consensus.cc:2272] T aac844f429fc4328a261a326181f6c35 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:25.027207 1169 tablet_bootstrap.cc:492] T d709aa500f8b422499c30843acd1a5ae P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:25.028280 1169 ts_tablet_manager.cc:1403] T d709aa500f8b422499c30843acd1a5ae P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.049s user 0.000s sys 0.003s
I20251025 07:52:25.028434 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 503d0732dd094099889fb56a747bbd33 (table default.Table_533c8546736848ea9ec200f63386630c [id=a5c73cb1e5a349ca8675f4ba4cb8a82d]) successfully deleted
I20251025 07:52:25.029248 1172 tablet_replica.cc:333] T 52a8fd3c834a4137956a86ad4f198f0f P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:25.029227 1169 raft_consensus.cc:359] T d709aa500f8b422499c30843acd1a5ae P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:25.029451 1169 raft_consensus.cc:385] T d709aa500f8b422499c30843acd1a5ae P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:25.029554 1169 raft_consensus.cc:740] T d709aa500f8b422499c30843acd1a5ae P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:25.029778 1169 consensus_queue.cc:260] T d709aa500f8b422499c30843acd1a5ae P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
W20251025 07:52:25.030781 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 50433881c0eb4aa5a389b6f5b4a4337a has no consensus state
I20251025 07:52:25.031018 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_78f73ce8fabd4c58b0bbcd72b23c5aad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:25.031582 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78f73ce8fabd4c58b0bbcd72b23c5aad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:25.031723 1167 tablet_bootstrap.cc:492] T 243825c6dd2d4461a01edec88f33f6ee P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:25.034787 1167 ts_tablet_manager.cc:1403] T 243825c6dd2d4461a01edec88f33f6ee P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.111s user 0.018s sys 0.020s
I20251025 07:52:25.035069 1172 raft_consensus.cc:2243] T 52a8fd3c834a4137956a86ad4f198f0f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:25.035319 1172 raft_consensus.cc:2272] T 52a8fd3c834a4137956a86ad4f198f0f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:25.038249 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 3d5e0dd14e474fe09993a03851f8b1f0 (DEFAULT_TABLE table=default.Table_028b24c9775b49ea8adf1a62fe9df71d [id=1b5db2f5053c4cb3a3b40eb109d5d5d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.038189 894 tablet_service.cc:1505] Processing CreateTablet for tablet 3d5e0dd14e474fe09993a03851f8b1f0 (DEFAULT_TABLE table=default.Table_028b24c9775b49ea8adf1a62fe9df71d [id=1b5db2f5053c4cb3a3b40eb109d5d5d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.038693 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d5e0dd14e474fe09993a03851f8b1f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.039064 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d5e0dd14e474fe09993a03851f8b1f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.039178 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 19918a3c1f1d4ed6a78d5b4f0f4e38b3 (DEFAULT_TABLE table=default.Table_b1c43f4e0ede4adcba8b010f04bdb72a [id=ee3cc33c49494767b860c9f1b7fa0d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.039537 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19918a3c1f1d4ed6a78d5b4f0f4e38b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.046782 890 tablet_service.cc:1505] Processing CreateTablet for tablet 19918a3c1f1d4ed6a78d5b4f0f4e38b3 (DEFAULT_TABLE table=default.Table_b1c43f4e0ede4adcba8b010f04bdb72a [id=ee3cc33c49494767b860c9f1b7fa0d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.047443 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19918a3c1f1d4ed6a78d5b4f0f4e38b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.052532 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet d709aa500f8b422499c30843acd1a5ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:25.053066 1169 ts_tablet_manager.cc:1434] T d709aa500f8b422499c30843acd1a5ae P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.025s user 0.001s sys 0.001s
I20251025 07:52:25.055620 1172 ts_tablet_manager.cc:1916] T 52a8fd3c834a4137956a86ad4f198f0f P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:25.055935 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_1c458711d56d4ddeba1d0bc09392a734" } modify_external_catalogs: true
I20251025 07:52:25.056159 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_1c458711d56d4ddeba1d0bc09392a734" } modify_external_catalogs: true
I20251025 07:52:25.058104 1169 tablet_bootstrap.cc:492] T f05afed029d446ea9eb0073642c7312d P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:25.061455 1169 tablet_bootstrap.cc:654] T f05afed029d446ea9eb0073642c7312d P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:25.063236 1170 ts_tablet_manager.cc:1916] T aac844f429fc4328a261a326181f6c35 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:25.063589 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_14cbf20f7ff449499612663b54f0acd9" } modify_external_catalogs: true
I20251025 07:52:25.063776 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_14cbf20f7ff449499612663b54f0acd9" } modify_external_catalogs: true
I20251025 07:52:25.064443 1166 tablet_bootstrap.cc:492] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:25.063772 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 33c608beb4ef4a859a135b11e07c81a4 (DEFAULT_TABLE table=default.Table_7a19acf479d64317bd910c04b08a8b4f [id=8bbb3b9204634ee6a04f5a4753739476]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.065177 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33c608beb4ef4a859a135b11e07c81a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.066331 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_e3ffa05bcb8040c79de137fb6bdfc4ad" } new_table_name: "default.Table_9b8e648894e14f8595796e4698407495" modify_external_catalogs: true
I20251025 07:52:25.066850 1178 raft_consensus.cc:493] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:25.067839 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_1eb640b2196849a8b0136b390deea1c2" } modify_external_catalogs: true
I20251025 07:52:25.067998 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_1eb640b2196849a8b0136b390deea1c2" } modify_external_catalogs: true
I20251025 07:52:25.068212 1167 raft_consensus.cc:359] T 243825c6dd2d4461a01edec88f33f6ee P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.077390 1167 raft_consensus.cc:385] T 243825c6dd2d4461a01edec88f33f6ee P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:25.077477 1167 raft_consensus.cc:740] T 243825c6dd2d4461a01edec88f33f6ee P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:25.077706 1167 consensus_queue.cc:260] T 243825c6dd2d4461a01edec88f33f6ee P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.074489 1178 raft_consensus.cc:515] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.078887 1178 leader_election.cc:290] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:25.074963 756 tablet_service.cc:1505] Processing CreateTablet for tablet 3d5e0dd14e474fe09993a03851f8b1f0 (DEFAULT_TABLE table=default.Table_028b24c9775b49ea8adf1a62fe9df71d [id=1b5db2f5053c4cb3a3b40eb109d5d5d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.080358 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d5e0dd14e474fe09993a03851f8b1f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.081157 747 tablet_service.cc:1505] Processing CreateTablet for tablet 50ffbcdd4a9247bf99b8e2361bc02ef5 (DEFAULT_TABLE table=default.Table_26ee7c4758284bc0bf685f326ebfd478 [id=fa9b211cb1cb437992aef4ed8f2e7439]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.075261 754 tablet_service.cc:1505] Processing CreateTablet for tablet 50433881c0eb4aa5a389b6f5b4a4337a (DEFAULT_TABLE table=default.Table_95222dffdec148e39f6429c366be2b4a [id=a5c73cb1e5a349ca8675f4ba4cb8a82d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.075892 751 tablet_service.cc:1505] Processing CreateTablet for tablet 19918a3c1f1d4ed6a78d5b4f0f4e38b3 (DEFAULT_TABLE table=default.Table_b1c43f4e0ede4adcba8b010f04bdb72a [id=ee3cc33c49494767b860c9f1b7fa0d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.076138 753 tablet_service.cc:1505] Processing CreateTablet for tablet 2947502815d742e89cacb474b7fbb249 (DEFAULT_TABLE table=default.Table_c96e62972b3c4ee48b6c3ea90831a34f [id=e7a5b463914840c980a306b7d46ea871]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.076351 752 tablet_service.cc:1505] Processing CreateTablet for tablet a231f3d31b204f7b82f018892ccc8435 (DEFAULT_TABLE table=default.Table_1c458711d56d4ddeba1d0bc09392a734 [id=7cebd72e604e497db80a8bc823471e97]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.083443 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50433881c0eb4aa5a389b6f5b4a4337a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.082218 1169 tablet_bootstrap.cc:492] T f05afed029d446ea9eb0073642c7312d P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:25.084606 1028 tablet_service.cc:1505] Processing CreateTablet for tablet a231f3d31b204f7b82f018892ccc8435 (DEFAULT_TABLE table=default.Table_1c458711d56d4ddeba1d0bc09392a734 [id=7cebd72e604e497db80a8bc823471e97]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.084673 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 3c3eb43209f24592820638fc25486b78 (DEFAULT_TABLE table=default.Table_e3ffa05bcb8040c79de137fb6bdfc4ad [id=89d0b4e609a74dde8540eb740446be5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.088352 1169 ts_tablet_manager.cc:1403] T f05afed029d446ea9eb0073642c7312d P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.002s
I20251025 07:52:25.085544 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 94408a3a90224645ba3cacccf5de33f1 (DEFAULT_TABLE table=default.Table_819aedd5270f4d6cbaaaee7c2ff5f676 [id=1e4728f474814b36992fd99de8451b95]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.085065 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 50433881c0eb4aa5a389b6f5b4a4337a (DEFAULT_TABLE table=default.Table_95222dffdec148e39f6429c366be2b4a [id=a5c73cb1e5a349ca8675f4ba4cb8a82d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.086014 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 50ffbcdd4a9247bf99b8e2361bc02ef5 (DEFAULT_TABLE table=default.Table_26ee7c4758284bc0bf685f326ebfd478 [id=fa9b211cb1cb437992aef4ed8f2e7439]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.085063 1023 tablet_service.cc:1505] Processing CreateTablet for tablet 83084be89b2646b0bda208b3f6f60d77 (DEFAULT_TABLE table=default.Table_5b6c9357453f4ece8c4e824377ab91ca [id=31c55cf000ed4e4dbefe4a407fa1828c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.088465 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a231f3d31b204f7b82f018892ccc8435. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.086577 1021 tablet_service.cc:1505] Processing CreateTablet for tablet 796413da731245459cc5dffacc3deb2f (DEFAULT_TABLE table=default.Table_ebc8b70bac6a4dc3930476eab4c441b3 [id=d1b9d89b22bc4431a7d59ceeb9d75050]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.089214 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94408a3a90224645ba3cacccf5de33f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.089430 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet a231f3d31b204f7b82f018892ccc8435
I20251025 07:52:25.091574 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 2947502815d742e89cacb474b7fbb249 (DEFAULT_TABLE table=default.Table_c96e62972b3c4ee48b6c3ea90831a34f [id=e7a5b463914840c980a306b7d46ea871]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.094273 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c3eb43209f24592820638fc25486b78. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.096191 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50ffbcdd4a9247bf99b8e2361bc02ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.104594 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19918a3c1f1d4ed6a78d5b4f0f4e38b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.105064 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 243825c6dd2d4461a01edec88f33f6ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:25.105229 755 tablet_service.cc:1505] Processing CreateTablet for tablet 33c608beb4ef4a859a135b11e07c81a4 (DEFAULT_TABLE table=default.Table_7a19acf479d64317bd910c04b08a8b4f [id=8bbb3b9204634ee6a04f5a4753739476]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.106294 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a231f3d31b204f7b82f018892ccc8435. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.076606 750 tablet_service.cc:1505] Processing CreateTablet for tablet 3c3eb43209f24592820638fc25486b78 (DEFAULT_TABLE table=default.Table_e3ffa05bcb8040c79de137fb6bdfc4ad [id=89d0b4e609a74dde8540eb740446be5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.077157 748 tablet_service.cc:1505] Processing CreateTablet for tablet 83084be89b2646b0bda208b3f6f60d77 (DEFAULT_TABLE table=default.Table_5b6c9357453f4ece8c4e824377ab91ca [id=31c55cf000ed4e4dbefe4a407fa1828c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.076893 749 tablet_service.cc:1505] Processing CreateTablet for tablet 94408a3a90224645ba3cacccf5de33f1 (DEFAULT_TABLE table=default.Table_819aedd5270f4d6cbaaaee7c2ff5f676 [id=1e4728f474814b36992fd99de8451b95]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.109458 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 9cf88862e0934634b3dd172e6b40a0ba
I20251025 07:52:25.108171 746 tablet_service.cc:1505] Processing CreateTablet for tablet 796413da731245459cc5dffacc3deb2f (DEFAULT_TABLE table=default.Table_ebc8b70bac6a4dc3930476eab4c441b3 [id=d1b9d89b22bc4431a7d59ceeb9d75050]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.112754 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2947502815d742e89cacb474b7fbb249. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.113082 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50ffbcdd4a9247bf99b8e2361bc02ef5. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:25.113319 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3c3eb43209f24592820638fc25486b78 consensus state has no leader
I20251025 07:52:25.114120 745 tablet_service.cc:1552] Processing DeleteTablet for tablet a231f3d31b204f7b82f018892ccc8435 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:25.114848 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 576c51d9a22d45f6a3642a7296b5bf23
I20251025 07:52:25.115717 744 tablet_service.cc:1552] Processing DeleteTablet for tablet 9cf88862e0934634b3dd172e6b40a0ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:25.116983 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet a231f3d31b204f7b82f018892ccc8435 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:25.117113 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33c608beb4ef4a859a135b11e07c81a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.117471 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet 576c51d9a22d45f6a3642a7296b5bf23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:25.117604 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 9cf88862e0934634b3dd172e6b40a0ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:25.122196 744 tablet_service.cc:1552] Processing DeleteTablet for tablet 576c51d9a22d45f6a3642a7296b5bf23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:25.123486 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 796413da731245459cc5dffacc3deb2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.124531 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83084be89b2646b0bda208b3f6f60d77. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.125341 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94408a3a90224645ba3cacccf5de33f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.130232 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c3eb43209f24592820638fc25486b78. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.131898 1169 raft_consensus.cc:359] T f05afed029d446ea9eb0073642c7312d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.132450 1169 raft_consensus.cc:385] T f05afed029d446ea9eb0073642c7312d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:25.132558 1169 raft_consensus.cc:740] T f05afed029d446ea9eb0073642c7312d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:25.133072 744 tablet_service.cc:1505] Processing CreateTablet for tablet b00153c97104470883f6089d71d23708 (DEFAULT_TABLE table=default.Table_78f73ce8fabd4c58b0bbcd72b23c5aad [id=7ec5f704dc254c11b865c660ffabc4fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.133806 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b00153c97104470883f6089d71d23708. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.134254 745 tablet_service.cc:1505] Processing CreateTablet for tablet d0fd00baa02d48278afaa5b8c19182f0 (DEFAULT_TABLE table=default.Table_f461c120f4bc4fefbaf8c486f4f910f9 [id=ff352eacb0614dbbbb6ab6369b9192e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.134702 1028 tablet_service.cc:1505] Processing CreateTablet for tablet b00153c97104470883f6089d71d23708 (DEFAULT_TABLE table=default.Table_78f73ce8fabd4c58b0bbcd72b23c5aad [id=7ec5f704dc254c11b865c660ffabc4fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.134907 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0fd00baa02d48278afaa5b8c19182f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.136947 1020 tablet_service.cc:1505] Processing CreateTablet for tablet d0fd00baa02d48278afaa5b8c19182f0 (DEFAULT_TABLE table=default.Table_f461c120f4bc4fefbaf8c486f4f910f9 [id=ff352eacb0614dbbbb6ab6369b9192e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.147373 1169 consensus_queue.cc:260] T f05afed029d446ea9eb0073642c7312d P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.147840 743 tablet_service.cc:1505] Processing CreateTablet for tablet 198db9a45257435f86df2605992ccb34 (DEFAULT_TABLE table=default.Table_79969e4ace6f4227b2748b59136027d6 [id=a1135611169142ba9ef987e3e9f95ce8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.153728 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 198db9a45257435f86df2605992ccb34. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.155079 1019 tablet_service.cc:1505] Processing CreateTablet for tablet 198db9a45257435f86df2605992ccb34 (DEFAULT_TABLE table=default.Table_79969e4ace6f4227b2748b59136027d6 [id=a1135611169142ba9ef987e3e9f95ce8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.155974 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2947502815d742e89cacb474b7fbb249. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.156275 1176 raft_consensus.cc:493] T d709aa500f8b422499c30843acd1a5ae P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:25.156477 1176 raft_consensus.cc:515] T d709aa500f8b422499c30843acd1a5ae P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:25.156646 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 198db9a45257435f86df2605992ccb34. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.157172 1176 leader_election.cc:290] T d709aa500f8b422499c30843acd1a5ae P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:25.158047 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d709aa500f8b422499c30843acd1a5ae" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:25.159140 776 raft_consensus.cc:2468] T d709aa500f8b422499c30843acd1a5ae P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:25.159650 982 leader_election.cc:304] T d709aa500f8b422499c30843acd1a5ae P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:25.159971 1176 raft_consensus.cc:2804] T d709aa500f8b422499c30843acd1a5ae P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:25.160121 1176 raft_consensus.cc:493] T d709aa500f8b422499c30843acd1a5ae P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:25.160243 1176 raft_consensus.cc:3060] T d709aa500f8b422499c30843acd1a5ae P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:25.162722 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50433881c0eb4aa5a389b6f5b4a4337a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.163511 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83084be89b2646b0bda208b3f6f60d77. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.165212 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b00153c97104470883f6089d71d23708. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.165989 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 796413da731245459cc5dffacc3deb2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.167471 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0fd00baa02d48278afaa5b8c19182f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.171720 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d709aa500f8b422499c30843acd1a5ae" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:25.171931 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f05afed029d446ea9eb0073642c7312d" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:25.172602 1048 raft_consensus.cc:2468] T f05afed029d446ea9eb0073642c7312d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:25.173424 712 leader_election.cc:304] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:25.173918 1178 raft_consensus.cc:2804] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:25.174045 1178 raft_consensus.cc:493] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:25.174168 1178 raft_consensus.cc:3060] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:25.175772 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_f461c120f4bc4fefbaf8c486f4f910f9" } new_table_name: "default.Table_7c78bc17ef6343f29cc55facd6893789" modify_external_catalogs: true
I20251025 07:52:25.176021 1176 raft_consensus.cc:515] T d709aa500f8b422499c30843acd1a5ae P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:25.176621 1049 raft_consensus.cc:2393] T d709aa500f8b422499c30843acd1a5ae P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 23eeffd3de254ce18db5febfde971c6b in current term 1: Already voted for candidate db42be7ad3c741a0aa427f27e245993b in this term.
I20251025 07:52:25.176762 1176 leader_election.cc:290] T d709aa500f8b422499c30843acd1a5ae P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:25.182003 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d709aa500f8b422499c30843acd1a5ae" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:25.182291 776 raft_consensus.cc:3060] T d709aa500f8b422499c30843acd1a5ae P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:25.182590 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_e0e89ced2ea94c139ed635849e78cb35" } modify_external_catalogs: true
I20251025 07:52:25.182775 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_e0e89ced2ea94c139ed635849e78cb35" } modify_external_catalogs: true
I20251025 07:52:25.185923 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_a6d264fbdb6c49f383edfb0311e0af2f" } new_table_name: "default.Table_74a57a4ecba54609893bd090bf411e82" modify_external_catalogs: true
W20251025 07:52:25.193835 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d0fd00baa02d48278afaa5b8c19182f0 consensus state has no leader
I20251025 07:52:25.195547 1178 raft_consensus.cc:515] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.196439 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_5b6c9357453f4ece8c4e824377ab91ca" } new_table_name: "default.Table_7c2f0e6e0edf4ae6920557e0d5130468" modify_external_catalogs: true
I20251025 07:52:25.197712 1178 leader_election.cc:290] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:25.197727 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_c96e62972b3c4ee48b6c3ea90831a34f" } modify_external_catalogs: true
I20251025 07:52:25.197906 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_c96e62972b3c4ee48b6c3ea90831a34f" } modify_external_catalogs: true
I20251025 07:52:25.200476 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f05afed029d446ea9eb0073642c7312d" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:25.200842 1049 raft_consensus.cc:3060] T f05afed029d446ea9eb0073642c7312d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:25.201213 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e84dbbd4a0bd4e4dbf1212fefbf00fd4 consensus state has no leader
I20251025 07:52:25.202035 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 785d0ce85b234826ba8021fbd182bf3c
I20251025 07:52:25.205994 1049 raft_consensus.cc:2468] T f05afed029d446ea9eb0073642c7312d P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:25.206476 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_4c12bad86d7946b9ac019c8e3856dcba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:25.206672 712 leader_election.cc:304] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
W20251025 07:52:25.207033 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c12bad86d7946b9ac019c8e3856dcba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:25.209838 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_0b05061ca2634b54a45d609f64a24d2d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:25.210455 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b05061ca2634b54a45d609f64a24d2d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:25.214071 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet f05afed029d446ea9eb0073642c7312d with delete_type TABLET_DATA_DELETED (replaced by tablet 3be8063aee794d2abcfb1d6e24209c06) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:25.217857 1171 ts_tablet_manager.cc:1929] T ef3820c3d739476fad3d83d3aa658742 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:25.218164 1171 log.cc:1199] T ef3820c3d739476fad3d83d3aa658742 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/ef3820c3d739476fad3d83d3aa658742
W20251025 07:52:25.218451 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 83084be89b2646b0bda208b3f6f60d77 consensus state has no leader
I20251025 07:52:25.218876 1171 ts_tablet_manager.cc:1950] T ef3820c3d739476fad3d83d3aa658742 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:25.222791 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet ef3820c3d739476fad3d83d3aa658742 (table default.Table_33eaf5d540914e43aad9bc6e22eb698a [id=768d2afbbcb24c26818cf1f64b330428]) successfully deleted
W20251025 07:52:25.223737 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 70c3a36dda7b4919be20390f5797d09c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70c3a36dda7b4919be20390f5797d09c already in progress: creating tablet
I20251025 07:52:25.227319 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_a94956d51e1a4c50b5becc17e435070d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:25.227875 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a94956d51e1a4c50b5becc17e435070d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:25.229575 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_672fb5e1742840ae8235d645567de473"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:25.230055 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_672fb5e1742840ae8235d645567de473 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:25.231443 1166 ts_tablet_manager.cc:1403] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.055s user 0.004s sys 0.001s
I20251025 07:52:25.231388 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_56f5f8c6d70645309e7ea14dcf847a43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:25.231845 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56f5f8c6d70645309e7ea14dcf847a43 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:25.233314 1183 raft_consensus.cc:493] T 14fd790ba7054c1fb09cdfb19b001987 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:25.233436 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 2947502815d742e89cacb474b7fbb249
I20251025 07:52:25.235773 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_bc8ea7b87c1a4b3baa64c5f5618eff7a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:25.236215 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc8ea7b87c1a4b3baa64c5f5618eff7a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:25.237269 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_95222dffdec148e39f6429c366be2b4a" } new_table_name: "default.Table_60f1e862b7bc4bdba20dd152856f55e3" modify_external_catalogs: true
I20251025 07:52:25.237704 1170 ts_tablet_manager.cc:1929] T aac844f429fc4328a261a326181f6c35 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:25.237898 1170 log.cc:1199] T aac844f429fc4328a261a326181f6c35 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/aac844f429fc4328a261a326181f6c35
I20251025 07:52:25.238055 1178 raft_consensus.cc:2804] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:25.238368 1178 raft_consensus.cc:697] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:25.238488 1169 ts_tablet_manager.cc:1434] T f05afed029d446ea9eb0073642c7312d P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.149s user 0.003s sys 0.000s
I20251025 07:52:25.238876 1169 tablet_bootstrap.cc:492] T 243825c6dd2d4461a01edec88f33f6ee P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:25.239087 1178 consensus_queue.cc:237] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.240669 1169 tablet_bootstrap.cc:654] T 243825c6dd2d4461a01edec88f33f6ee P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:25.242465 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_66a70e784e9b49cdb516f657733f2a38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:25.242969 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66a70e784e9b49cdb516f657733f2a38 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:25.244479 1170 ts_tablet_manager.cc:1950] T aac844f429fc4328a261a326181f6c35 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:25.251144 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_197e51547cd245348f196e61fd525dae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:25.251394 1170 tablet_replica.cc:333] T ef3820c3d739476fad3d83d3aa658742 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:25.251556 1170 raft_consensus.cc:2243] T ef3820c3d739476fad3d83d3aa658742 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
W20251025 07:52:25.251688 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_197e51547cd245348f196e61fd525dae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:25.251735 1170 raft_consensus.cc:2272] T ef3820c3d739476fad3d83d3aa658742 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:25.252903 776 raft_consensus.cc:2468] T d709aa500f8b422499c30843acd1a5ae P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:25.252986 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_cc49d933a4b343aeb2220c3939495834"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:25.253439 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc49d933a4b343aeb2220c3939495834 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:25.253634 982 leader_election.cc:304] T d709aa500f8b422499c30843acd1a5ae P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:25.254060 1176 raft_consensus.cc:2804] T d709aa500f8b422499c30843acd1a5ae P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:25.254297 1176 raft_consensus.cc:697] T d709aa500f8b422499c30843acd1a5ae P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:25.254922 1176 consensus_queue.cc:237] T d709aa500f8b422499c30843acd1a5ae P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:25.255671 1169 tablet_bootstrap.cc:492] T 243825c6dd2d4461a01edec88f33f6ee P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:25.255808 1169 ts_tablet_manager.cc:1403] T 243825c6dd2d4461a01edec88f33f6ee P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.001s
I20251025 07:52:25.257285 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_e0bf55e24b084fdbb62f09cce80b50c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:25.257377 1169 raft_consensus.cc:359] T 243825c6dd2d4461a01edec88f33f6ee P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.257520 1169 raft_consensus.cc:385] T 243825c6dd2d4461a01edec88f33f6ee P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:25.257609 1169 raft_consensus.cc:740] T 243825c6dd2d4461a01edec88f33f6ee P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
W20251025 07:52:25.257848 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0bf55e24b084fdbb62f09cce80b50c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:25.257794 1169 consensus_queue.cc:260] T 243825c6dd2d4461a01edec88f33f6ee P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.258442 1169 ts_tablet_manager.cc:1434] T 243825c6dd2d4461a01edec88f33f6ee P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251025 07:52:25.258708 1169 tablet_bootstrap.cc:492] T d684da8134ce480693f470acdac176d4 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:25.260306 1169 tablet_bootstrap.cc:654] T d684da8134ce480693f470acdac176d4 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:25.254931 1170 ts_tablet_manager.cc:1916] T ef3820c3d739476fad3d83d3aa658742 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:25.273873 1170 ts_tablet_manager.cc:1929] T ef3820c3d739476fad3d83d3aa658742 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:25.274017 1170 log.cc:1199] T ef3820c3d739476fad3d83d3aa658742 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/ef3820c3d739476fad3d83d3aa658742
I20251025 07:52:25.274938 1169 tablet_bootstrap.cc:492] T d684da8134ce480693f470acdac176d4 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:25.275120 1169 ts_tablet_manager.cc:1403] T d684da8134ce480693f470acdac176d4 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.001s
I20251025 07:52:25.275869 1169 raft_consensus.cc:359] T d684da8134ce480693f470acdac176d4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.276129 1169 raft_consensus.cc:385] T d684da8134ce480693f470acdac176d4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:25.276201 1169 raft_consensus.cc:740] T d684da8134ce480693f470acdac176d4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:25.276470 1169 consensus_queue.cc:260] T d684da8134ce480693f470acdac176d4 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.277042 1169 ts_tablet_manager.cc:1434] T d684da8134ce480693f470acdac176d4 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:25.277375 1169 tablet_bootstrap.cc:492] T 5680e1adaf3f4b7f8c5e05523dd0950a P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:25.277771 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 785d0ce85b234826ba8021fbd182bf3c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:25.278121 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 2947502815d742e89cacb474b7fbb249 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47408
W20251025 07:52:25.279529 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 54cb4953ff264c13a0cc730b4c452ecc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54cb4953ff264c13a0cc730b4c452ecc already in progress: creating tablet
W20251025 07:52:25.279799 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet d684da8134ce480693f470acdac176d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d684da8134ce480693f470acdac176d4 already in progress: creating tablet
I20251025 07:52:25.280197 1169 tablet_bootstrap.cc:654] T 5680e1adaf3f4b7f8c5e05523dd0950a P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:25.280156 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 70c3a36dda7b4919be20390f5797d09c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70c3a36dda7b4919be20390f5797d09c already in progress: creating tablet
W20251025 07:52:25.280504 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 20e4178c16bd4f18b1c6bc969f6a0c6c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20e4178c16bd4f18b1c6bc969f6a0c6c already in progress: creating tablet
I20251025 07:52:25.281462 1170 ts_tablet_manager.cc:1950] T ef3820c3d739476fad3d83d3aa658742 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:25.283125 1170 tablet_replica.cc:333] T d709aa500f8b422499c30843acd1a5ae P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:25.286435 891 tablet_service.cc:1505] Processing CreateTablet for tablet 50433881c0eb4aa5a389b6f5b4a4337a (DEFAULT_TABLE table=default.Table_95222dffdec148e39f6429c366be2b4a [id=a5c73cb1e5a349ca8675f4ba4cb8a82d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.286890 892 tablet_service.cc:1505] Processing CreateTablet for tablet 33c608beb4ef4a859a135b11e07c81a4 (DEFAULT_TABLE table=default.Table_7a19acf479d64317bd910c04b08a8b4f [id=8bbb3b9204634ee6a04f5a4753739476]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.287397 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33c608beb4ef4a859a135b11e07c81a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.291380 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_7a19acf479d64317bd910c04b08a8b4f" } modify_external_catalogs: true
I20251025 07:52:25.291429 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d709aa500f8b422499c30843acd1a5ae" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:25.291560 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_7a19acf479d64317bd910c04b08a8b4f" } modify_external_catalogs: true
I20251025 07:52:25.291810 913 raft_consensus.cc:3060] T d709aa500f8b422499c30843acd1a5ae P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:25.295324 1183 raft_consensus.cc:515] T 14fd790ba7054c1fb09cdfb19b001987 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.292793 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d709aa500f8b422499c30843acd1a5ae" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:25.296559 889 tablet_service.cc:1505] Processing CreateTablet for tablet a231f3d31b204f7b82f018892ccc8435 (DEFAULT_TABLE table=default.Table_1c458711d56d4ddeba1d0bc09392a734 [id=7cebd72e604e497db80a8bc823471e97]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.297813 893 tablet_service.cc:1505] Processing CreateTablet for tablet 2947502815d742e89cacb474b7fbb249 (DEFAULT_TABLE table=default.Table_c96e62972b3c4ee48b6c3ea90831a34f [id=e7a5b463914840c980a306b7d46ea871]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.298660 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50433881c0eb4aa5a389b6f5b4a4337a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.295158 1166 raft_consensus.cc:359] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.300309 1166 raft_consensus.cc:385] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:25.300408 1166 raft_consensus.cc:740] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:25.300987 1166 consensus_queue.cc:260] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.302762 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet ef3820c3d739476fad3d83d3aa658742 (table default.Table_33eaf5d540914e43aad9bc6e22eb698a [id=768d2afbbcb24c26818cf1f64b330428]) successfully deleted
W20251025 07:52:25.303004 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 713ae1dd444544bdabdd4a771e109d90 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 713ae1dd444544bdabdd4a771e109d90 already in progress: creating tablet
I20251025 07:52:25.303026 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14fd790ba7054c1fb09cdfb19b001987" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:25.304466 850 leader_election.cc:343] T 14fd790ba7054c1fb09cdfb19b001987 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:25.305104 1183 leader_election.cc:290] T 14fd790ba7054c1fb09cdfb19b001987 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
W20251025 07:52:25.305619 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 50433881c0eb4aa5a389b6f5b4a4337a consensus state has no leader
I20251025 07:52:25.306718 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2947502815d742e89cacb474b7fbb249. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.312402 894 tablet_service.cc:1505] Processing CreateTablet for tablet 94408a3a90224645ba3cacccf5de33f1 (DEFAULT_TABLE table=default.Table_819aedd5270f4d6cbaaaee7c2ff5f676 [id=1e4728f474814b36992fd99de8451b95]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.315768 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_74a57a4ecba54609893bd090bf411e82" } new_table_name: "default.Table_826ad7d517d34a33a3759e733302ce66" modify_external_catalogs: true
I20251025 07:52:25.317718 884 tablet_service.cc:1552] Processing DeleteTablet for tablet a231f3d31b204f7b82f018892ccc8435 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:25.317797 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_bc8ea7b87c1a4b3baa64c5f5618eff7a" } modify_external_catalogs: true
I20251025 07:52:25.318064 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 9cf88862e0934634b3dd172e6b40a0ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:25.318043 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_bc8ea7b87c1a4b3baa64c5f5618eff7a" } modify_external_catalogs: true
I20251025 07:52:25.318322 882 tablet_service.cc:1552] Processing DeleteTablet for tablet 576c51d9a22d45f6a3642a7296b5bf23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:25.319671 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_43eec86aa9fc405fa761a8eaa89d2db0" } modify_external_catalogs: true
I20251025 07:52:25.319852 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_43eec86aa9fc405fa761a8eaa89d2db0" } modify_external_catalogs: true
I20251025 07:52:25.320469 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a231f3d31b204f7b82f018892ccc8435. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.313395 888 tablet_service.cc:1505] Processing CreateTablet for tablet 3c3eb43209f24592820638fc25486b78 (DEFAULT_TABLE table=default.Table_e3ffa05bcb8040c79de137fb6bdfc4ad [id=89d0b4e609a74dde8540eb740446be5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.328512 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94408a3a90224645ba3cacccf5de33f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.330658 886 tablet_service.cc:1505] Processing CreateTablet for tablet 50ffbcdd4a9247bf99b8e2361bc02ef5 (DEFAULT_TABLE table=default.Table_26ee7c4758284bc0bf685f326ebfd478 [id=fa9b211cb1cb437992aef4ed8f2e7439]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.332787 892 tablet_service.cc:1505] Processing CreateTablet for tablet d0fd00baa02d48278afaa5b8c19182f0 (DEFAULT_TABLE table=default.Table_f461c120f4bc4fefbaf8c486f4f910f9 [id=ff352eacb0614dbbbb6ab6369b9192e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.333930 1166 ts_tablet_manager.cc:1434] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.097s user 0.001s sys 0.001s
I20251025 07:52:25.336702 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 785d0ce85b234826ba8021fbd182bf3c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:25.338483 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 2947502815d742e89cacb474b7fbb249 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:25.344357 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0fd00baa02d48278afaa5b8c19182f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.340546 887 tablet_service.cc:1505] Processing CreateTablet for tablet 83084be89b2646b0bda208b3f6f60d77 (DEFAULT_TABLE table=default.Table_5b6c9357453f4ece8c4e824377ab91ca [id=31c55cf000ed4e4dbefe4a407fa1828c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.333803 882 tablet_service.cc:1505] Processing CreateTablet for tablet b00153c97104470883f6089d71d23708 (DEFAULT_TABLE table=default.Table_78f73ce8fabd4c58b0bbcd72b23c5aad [id=7ec5f704dc254c11b865c660ffabc4fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.349079 885 tablet_service.cc:1505] Processing CreateTablet for tablet 796413da731245459cc5dffacc3deb2f (DEFAULT_TABLE table=default.Table_ebc8b70bac6a4dc3930476eab4c441b3 [id=d1b9d89b22bc4431a7d59ceeb9d75050]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.352032 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c3eb43209f24592820638fc25486b78. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.353193 1166 tablet_bootstrap.cc:492] T d684da8134ce480693f470acdac176d4 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:25.355460 913 raft_consensus.cc:2468] T d709aa500f8b422499c30843acd1a5ae P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:25.350540 883 tablet_service.cc:1505] Processing CreateTablet for tablet 198db9a45257435f86df2605992ccb34 (DEFAULT_TABLE table=default.Table_79969e4ace6f4227b2748b59136027d6 [id=a1135611169142ba9ef987e3e9f95ce8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.357597 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50ffbcdd4a9247bf99b8e2361bc02ef5. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:25.358307 511 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 50433881c0eb4aa5a389b6f5b4a4337a (table default.Table_60f1e862b7bc4bdba20dd152856f55e3 [id=a5c73cb1e5a349ca8675f4ba4cb8a82d], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet 50433881c0eb4aa5a389b6f5b4a4337a consensus state has no leader
I20251025 07:52:25.358650 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83084be89b2646b0bda208b3f6f60d77. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.361217 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 796413da731245459cc5dffacc3deb2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.366240 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 9bf486d82b984d2c89ee32e84a85894f (DEFAULT_TABLE table=default.Table_0b05061ca2634b54a45d609f64a24d2d [id=5fcd7d82bf9443edbf595c28fcab99ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.366575 517 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet e84dbbd4a0bd4e4dbf1212fefbf00fd4 (table default.Table_74a57a4ecba54609893bd090bf411e82 [id=b6e173967bad47e6b306627221b80f94]). Expected version 2 got 0
I20251025 07:52:25.366714 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bf486d82b984d2c89ee32e84a85894f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.370565 1025 tablet_service.cc:1505] Processing CreateTablet for tablet cf2cab1552bd4e7394797c93b149f450 (DEFAULT_TABLE table=default.Table_4c12bad86d7946b9ac019c8e3856dcba [id=7de1e93439f94b2ebd4db3ed27f95872]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.371053 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf2cab1552bd4e7394797c93b149f450. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.374547 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b00153c97104470883f6089d71d23708. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:25.375217 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e84dbbd4a0bd4e4dbf1212fefbf00fd4 consensus state has no leader
I20251025 07:52:25.375694 526 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet ce5d1b1dafa946f2a29fdc2ebec26ec1
I20251025 07:52:25.376241 884 tablet_service.cc:1505] Processing CreateTablet for tablet cf2cab1552bd4e7394797c93b149f450 (DEFAULT_TABLE table=default.Table_4c12bad86d7946b9ac019c8e3856dcba [id=7de1e93439f94b2ebd4db3ed27f95872]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.377269 1212 raft_consensus.cc:493] T d684da8134ce480693f470acdac176d4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:25.377424 1212 raft_consensus.cc:515] T d684da8134ce480693f470acdac176d4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.377624 890 tablet_service.cc:1505] Processing CreateTablet for tablet 9bf486d82b984d2c89ee32e84a85894f (DEFAULT_TABLE table=default.Table_0b05061ca2634b54a45d609f64a24d2d [id=5fcd7d82bf9443edbf595c28fcab99ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.378365 1212 leader_election.cc:290] T d684da8134ce480693f470acdac176d4 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:25.390261 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 198db9a45257435f86df2605992ccb34. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.392331 517 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet e84dbbd4a0bd4e4dbf1212fefbf00fd4 (table default.Table_826ad7d517d34a33a3759e733302ce66 [id=b6e173967bad47e6b306627221b80f94]). Expected version 3 got 0
I20251025 07:52:25.397377 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_78f73ce8fabd4c58b0bbcd72b23c5aad" } new_table_name: "default.Table_b330f1a41da2427995e699545d5fc49b" modify_external_catalogs: true
I20251025 07:52:25.397935 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bf486d82b984d2c89ee32e84a85894f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.399101 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d684da8134ce480693f470acdac176d4" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:25.399503 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf2cab1552bd4e7394797c93b149f450. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:25.400179 983 leader_election.cc:343] T d684da8134ce480693f470acdac176d4 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:25.403105 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_0b05061ca2634b54a45d609f64a24d2d" } new_table_name: "default.Table_e43f4b13c0ba4d72b1c7292170cb7931" modify_external_catalogs: true
I20251025 07:52:25.407243 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_4c12bad86d7946b9ac019c8e3856dcba" } modify_external_catalogs: true
I20251025 07:52:25.407441 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_4c12bad86d7946b9ac019c8e3856dcba" } modify_external_catalogs: true
I20251025 07:52:25.420115 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_7c78bc17ef6343f29cc55facd6893789" } new_table_name: "default.Table_1d4029fc57dd4ecb84b8621c33d27958" modify_external_catalogs: true
I20251025 07:52:25.421772 525 master_service.cc:954] ReplaceTablet: received request to replace tablet 52a8fd3c834a4137956a86ad4f198f0f from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:25.428259 525 master_service.cc:954] ReplaceTablet: received request to replace tablet 89bfb7e2f051454aae0af5755a79ca58 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:25.419688 1218 raft_consensus.cc:493] T 243825c6dd2d4461a01edec88f33f6ee P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:25.433854 524 master_service.cc:954] ReplaceTablet: received request to replace tablet e84dbbd4a0bd4e4dbf1212fefbf00fd4 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:25.436331 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_a94956d51e1a4c50b5becc17e435070d" } modify_external_catalogs: true
I20251025 07:52:25.436522 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_a94956d51e1a4c50b5becc17e435070d" } modify_external_catalogs: true
I20251025 07:52:25.438273 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_373f40b4ac044088a071a6d26d7a6da0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:25.438498 1140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8603bddd8a084ff7a73a25641768dfa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } } }
I20251025 07:52:25.438748 1140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [sys.catalog]: This master's current role is: LEADER
W20251025 07:52:25.438964 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_373f40b4ac044088a071a6d26d7a6da0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:25.440187 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 14fd790ba7054c1fb09cdfb19b001987
I20251025 07:52:25.443161 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 14fd790ba7054c1fb09cdfb19b001987 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:25.446185 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_ebcfacbbec3147a189ff08cec113fc55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:25.446947 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ebcfacbbec3147a189ff08cec113fc55 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:25.460773 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 2947502815d742e89cacb474b7fbb249 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:25.461711 1171 tablet_replica.cc:333] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:25.463160 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 785d0ce85b234826ba8021fbd182bf3c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:25.465142 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14fd790ba7054c1fb09cdfb19b001987" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:25.468276 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d684da8134ce480693f470acdac176d4" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
W20251025 07:52:25.470294 982 leader_election.cc:343] T d684da8134ce480693f470acdac176d4 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:25.470525 982 leader_election.cc:304] T d684da8134ce480693f470acdac176d4 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:25.471102 1212 raft_consensus.cc:2749] T d684da8134ce480693f470acdac176d4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:25.472241 756 tablet_service.cc:1505] Processing CreateTablet for tablet 9bf486d82b984d2c89ee32e84a85894f (DEFAULT_TABLE table=default.Table_0b05061ca2634b54a45d609f64a24d2d [id=5fcd7d82bf9443edbf595c28fcab99ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.477836 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRBHlYt/build/asan/bin/kudu
/tmp/dist-test-taskRBHlYt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:37611
--webserver_interface=127.0.104.60
--webserver_port=38531
--builtin_ntp_servers=127.0.104.20:40091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:37421,127.0.104.61:42203,127.0.104.60:37611
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251025 07:52:25.478996 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 33c608beb4ef4a859a135b11e07c81a4
W20251025 07:52:25.480034 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9bf486d82b984d2c89ee32e84a85894f consensus state has no leader
W20251025 07:52:25.482547 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b00153c97104470883f6089d71d23708 consensus state has no leader
I20251025 07:52:25.483091 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet cf2cab1552bd4e7394797c93b149f450
I20251025 07:52:25.484018 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 33c608beb4ef4a859a135b11e07c81a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:25.484089 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet cf2cab1552bd4e7394797c93b149f450 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:25.487859 1172 ts_tablet_manager.cc:1929] T 52a8fd3c834a4137956a86ad4f198f0f P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:25.488008 1172 log.cc:1199] T 52a8fd3c834a4137956a86ad4f198f0f P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/52a8fd3c834a4137956a86ad4f198f0f
I20251025 07:52:25.488394 1172 ts_tablet_manager.cc:1950] T 52a8fd3c834a4137956a86ad4f198f0f P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:25.488962 1172 tablet_replica.cc:333] T aac844f429fc4328a261a326181f6c35 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:25.489225 1172 raft_consensus.cc:2243] T aac844f429fc4328a261a326181f6c35 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:25.489449 1172 raft_consensus.cc:2272] T aac844f429fc4328a261a326181f6c35 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:25.490774 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bf486d82b984d2c89ee32e84a85894f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.493320 1167 ts_tablet_manager.cc:1434] T 243825c6dd2d4461a01edec88f33f6ee P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.434s user 0.076s sys 0.077s
I20251025 07:52:25.493692 1167 tablet_bootstrap.cc:492] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:25.496194 1172 ts_tablet_manager.cc:1916] T aac844f429fc4328a261a326181f6c35 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:25.497902 1210 raft_consensus.cc:493] T 243825c6dd2d4461a01edec88f33f6ee P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:25.498435 1167 tablet_bootstrap.cc:654] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:25.502398 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 14fd790ba7054c1fb09cdfb19b001987 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:25.500015 1210 raft_consensus.cc:515] T 243825c6dd2d4461a01edec88f33f6ee P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.502691 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 33c608beb4ef4a859a135b11e07c81a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:25.503043 882 tablet_service.cc:1552] Processing DeleteTablet for tablet 14fd790ba7054c1fb09cdfb19b001987 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:25.504119 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "243825c6dd2d4461a01edec88f33f6ee" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:25.504413 1049 raft_consensus.cc:2468] T 243825c6dd2d4461a01edec88f33f6ee P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:25.504899 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 52a8fd3c834a4137956a86ad4f198f0f (table default.Table_02b9c6449126452b9694503a3d6ad253 [id=68d633fa085b416ab906780b71edf145]) successfully deleted
I20251025 07:52:25.505049 712 leader_election.cc:304] T 243825c6dd2d4461a01edec88f33f6ee P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:25.478032 1218 raft_consensus.cc:515] T 243825c6dd2d4461a01edec88f33f6ee P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.506513 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_072e0092adb64ef0ac6834dc18688ccf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:25.506822 1218 leader_election.cc:290] T 243825c6dd2d4461a01edec88f33f6ee P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
W20251025 07:52:25.507165 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_072e0092adb64ef0ac6834dc18688ccf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:25.507720 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 89bfb7e2f051454aae0af5755a79ca58
W20251025 07:52:25.508203 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d0fd00baa02d48278afaa5b8c19182f0 consensus state has no leader
I20251025 07:52:25.508296 525 catalog_manager.cc:6719] ReplaceTablet: tablet 89bfb7e2f051454aae0af5755a79ca58 deleted and replaced by tablet cd23b8094d7e472eba6bd170248858a9
I20251025 07:52:25.508512 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_b6cf4ff8abe642088cedf32609854bdd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:25.508606 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_5237c35dbfa7429e803d5008a8b3487c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:25.508687 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_7ce70fea0fe04c52a62033afb8eef977"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:25.509065 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ce70fea0fe04c52a62033afb8eef977 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:25.509136 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5237c35dbfa7429e803d5008a8b3487c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:25.508508 519 master_service.cc:954] ReplaceTablet: received request to replace tablet a02b627401a64e1583f9e6123ef1be6f from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:25.510356 1166 tablet_bootstrap.cc:654] T d684da8134ce480693f470acdac176d4 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:25.511795 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 89bfb7e2f051454aae0af5755a79ca58 with delete_type TABLET_DATA_DELETED (replaced by tablet cd23b8094d7e472eba6bd170248858a9) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:25.512382 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_a58a5fd6549343d5bb4c540a9dde185a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:25.512840 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a58a5fd6549343d5bb4c540a9dde185a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:25.509038 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b6cf4ff8abe642088cedf32609854bdd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:25.514796 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "243825c6dd2d4461a01edec88f33f6ee" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:25.515038 1049 raft_consensus.cc:2468] T 243825c6dd2d4461a01edec88f33f6ee P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:25.515422 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_197e51547cd245348f196e61fd525dae" } new_table_name: "default.Table_0eca32ca64da44198140ae7ec9a77767" modify_external_catalogs: true
I20251025 07:52:25.515925 850 leader_election.cc:304] T 243825c6dd2d4461a01edec88f33f6ee P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:25.517326 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d709aa500f8b422499c30843acd1a5ae" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:25.517611 913 raft_consensus.cc:2393] T d709aa500f8b422499c30843acd1a5ae P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 23eeffd3de254ce18db5febfde971c6b in current term 1: Already voted for candidate db42be7ad3c741a0aa427f27e245993b in this term.
W20251025 07:52:25.522256 847 leader_election.cc:343] T 14fd790ba7054c1fb09cdfb19b001987 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:25.522596 847 leader_election.cc:304] T 14fd790ba7054c1fb09cdfb19b001987 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:25.523710 1183 raft_consensus.cc:2749] T 14fd790ba7054c1fb09cdfb19b001987 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:25.524085 886 tablet_service.cc:1552] Processing DeleteTablet for tablet cf2cab1552bd4e7394797c93b149f450 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:25.524461 882 tablet_service.cc:1552] Processing DeleteTablet for tablet 33c608beb4ef4a859a135b11e07c81a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:25.525386 1218 raft_consensus.cc:2804] T 243825c6dd2d4461a01edec88f33f6ee P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:25.525544 1218 raft_consensus.cc:493] T 243825c6dd2d4461a01edec88f33f6ee P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:25.525660 1218 raft_consensus.cc:3060] T 243825c6dd2d4461a01edec88f33f6ee P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:25.525889 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet e84dbbd4a0bd4e4dbf1212fefbf00fd4
I20251025 07:52:25.526427 518 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet ab7a6348105a43d1856e79f8a14506d1
I20251025 07:52:25.527565 882 tablet_service.cc:1552] Processing DeleteTablet for tablet 89bfb7e2f051454aae0af5755a79ca58 with delete_type TABLET_DATA_DELETED (replaced by tablet cd23b8094d7e472eba6bd170248858a9) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:25.530930 520 master_service.cc:954] ReplaceTablet: received request to replace tablet 576c51d9a22d45f6a3642a7296b5bf23 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:25.531939 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet e84dbbd4a0bd4e4dbf1212fefbf00fd4 with delete_type TABLET_DATA_DELETED (replaced by tablet 742b3ec3f6604647b9e397532f1673dd) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:25.534966 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_4bafafcc9e8641b687d438725186f639"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:25.535756 524 catalog_manager.cc:6719] ReplaceTablet: tablet e84dbbd4a0bd4e4dbf1212fefbf00fd4 deleted and replaced by tablet 742b3ec3f6604647b9e397532f1673dd
W20251025 07:52:25.536504 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4bafafcc9e8641b687d438725186f639 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:25.538892 887 tablet_service.cc:1552] Processing DeleteTablet for tablet e84dbbd4a0bd4e4dbf1212fefbf00fd4 with delete_type TABLET_DATA_DELETED (replaced by tablet 742b3ec3f6604647b9e397532f1673dd) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:25.541317 752 tablet_service.cc:1552] Processing DeleteTablet for tablet cf2cab1552bd4e7394797c93b149f450 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:25.541512 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "243825c6dd2d4461a01edec88f33f6ee" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:25.542213 1210 leader_election.cc:290] T 243825c6dd2d4461a01edec88f33f6ee P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:25.542692 710 leader_election.cc:343] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: f05afed029d446ea9eb0073642c7312d
W20251025 07:52:25.547689 710 leader_election.cc:343] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: f05afed029d446ea9eb0073642c7312d
I20251025 07:52:25.548813 710 leader_election.cc:304] T d709aa500f8b422499c30843acd1a5ae P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:25.558157 1222 raft_consensus.cc:2804] T 243825c6dd2d4461a01edec88f33f6ee P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:25.558157 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_9f6b2f95721f488c86fde259a7cf95ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:25.558286 1222 raft_consensus.cc:493] T 243825c6dd2d4461a01edec88f33f6ee P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:25.558425 1222 raft_consensus.cc:3060] T 243825c6dd2d4461a01edec88f33f6ee P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:25.558828 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f6b2f95721f488c86fde259a7cf95ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:25.558702 1218 raft_consensus.cc:515] T 243825c6dd2d4461a01edec88f33f6ee P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.560752 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "243825c6dd2d4461a01edec88f33f6ee" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:25.560962 1049 raft_consensus.cc:3060] T 243825c6dd2d4461a01edec88f33f6ee P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:25.560871 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_ca46a7a783e547998d1be8476801a9e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:25.561229 1222 raft_consensus.cc:515] T 243825c6dd2d4461a01edec88f33f6ee P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:25.561380 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca46a7a783e547998d1be8476801a9e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:25.563802 1049 raft_consensus.cc:2468] T 243825c6dd2d4461a01edec88f33f6ee P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:25.564503 850 leader_election.cc:304] T 243825c6dd2d4461a01edec88f33f6ee P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:25.564864 1183 raft_consensus.cc:2804] T 243825c6dd2d4461a01edec88f33f6ee P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:25.565353 1183 raft_consensus.cc:697] T 243825c6dd2d4461a01edec88f33f6ee P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:25.565717 1222 leader_election.cc:290] T 243825c6dd2d4461a01edec88f33f6ee P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:25.565781 1183 consensus_queue.cc:237] T 243825c6dd2d4461a01edec88f33f6ee P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.566015 746 tablet_service.cc:1505] Processing CreateTablet for tablet cf2cab1552bd4e7394797c93b149f450 (DEFAULT_TABLE table=default.Table_4c12bad86d7946b9ac019c8e3856dcba [id=7de1e93439f94b2ebd4db3ed27f95872]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.566313 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 89bfb7e2f051454aae0af5755a79ca58 with delete_type TABLET_DATA_DELETED (replaced by tablet cd23b8094d7e472eba6bd170248858a9) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:25.566558 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf2cab1552bd4e7394797c93b149f450. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.571772 747 tablet_service.cc:1552] Processing DeleteTablet for tablet e84dbbd4a0bd4e4dbf1212fefbf00fd4 with delete_type TABLET_DATA_DELETED (replaced by tablet 742b3ec3f6604647b9e397532f1673dd) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:25.572196 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "243825c6dd2d4461a01edec88f33f6ee" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:25.572314 775 raft_consensus.cc:2393] T 243825c6dd2d4461a01edec88f33f6ee P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3aea2869ff814cc7b142012faa274382 in current term 1: Already voted for candidate 23eeffd3de254ce18db5febfde971c6b in this term.
I20251025 07:52:25.572347 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "243825c6dd2d4461a01edec88f33f6ee" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:25.572897 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "243825c6dd2d4461a01edec88f33f6ee" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:25.573140 1049 raft_consensus.cc:2393] T 243825c6dd2d4461a01edec88f33f6ee P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 23eeffd3de254ce18db5febfde971c6b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:25.573546 712 leader_election.cc:304] T 243825c6dd2d4461a01edec88f33f6ee P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:25.573647 1218 leader_election.cc:290] T 243825c6dd2d4461a01edec88f33f6ee P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:25.576683 1210 raft_consensus.cc:2749] T d709aa500f8b422499c30843acd1a5ae P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:25.578310 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "243825c6dd2d4461a01edec88f33f6ee" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:25.580205 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_ebc8b70bac6a4dc3930476eab4c441b3" } new_table_name: "default.Table_234642f70fac40dd867675499ace87a3" modify_external_catalogs: true
I20251025 07:52:25.582954 1222 raft_consensus.cc:2749] T 243825c6dd2d4461a01edec88f33f6ee P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251025 07:52:25.583527 775 raft_consensus.cc:2393] T 243825c6dd2d4461a01edec88f33f6ee P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3aea2869ff814cc7b142012faa274382 in current term 1: Already voted for candidate 23eeffd3de254ce18db5febfde971c6b in this term.
I20251025 07:52:25.586930 1167 tablet_bootstrap.cc:492] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:25.587124 1167 ts_tablet_manager.cc:1403] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.093s user 0.013s sys 0.000s
I20251025 07:52:25.587265 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_1d4029fc57dd4ecb84b8621c33d27958" } modify_external_catalogs: true
I20251025 07:52:25.587476 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_1d4029fc57dd4ecb84b8621c33d27958" } modify_external_catalogs: true
I20251025 07:52:25.587900 1167 raft_consensus.cc:359] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.588110 1167 raft_consensus.cc:385] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:25.588200 1167 raft_consensus.cc:740] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
W20251025 07:52:25.589051 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 87a93778620f4c84b3b3f7a3d7d7e87f has no consensus state
I20251025 07:52:25.591797 1167 consensus_queue.cc:260] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.592442 1167 ts_tablet_manager.cc:1434] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251025 07:52:25.592742 1167 tablet_bootstrap.cc:492] T 89bfb7e2f051454aae0af5755a79ca58 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:25.594316 1167 tablet_bootstrap.cc:654] T 89bfb7e2f051454aae0af5755a79ca58 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:25.600286 1171 raft_consensus.cc:2243] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:25.600951 1171 pending_rounds.cc:70] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b: Trying to abort 1 pending ops.
I20251025 07:52:25.601145 1171 pending_rounds.cc:77] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214607340506279936 op_type: NO_OP noop_request { }
I20251025 07:52:25.601323 1171 raft_consensus.cc:2889] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 07:52:25.601456 1171 raft_consensus.cc:2272] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:25.601863 1170 raft_consensus.cc:2243] T d709aa500f8b422499c30843acd1a5ae P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:25.602690 1170 pending_rounds.cc:70] T d709aa500f8b422499c30843acd1a5ae P db42be7ad3c741a0aa427f27e245993b: Trying to abort 1 pending ops.
I20251025 07:52:25.602862 1170 pending_rounds.cc:77] T d709aa500f8b422499c30843acd1a5ae P db42be7ad3c741a0aa427f27e245993b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214607340570603520 op_type: NO_OP noop_request { }
I20251025 07:52:25.603039 1170 raft_consensus.cc:2889] T d709aa500f8b422499c30843acd1a5ae P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 07:52:25.603185 1170 raft_consensus.cc:2272] T d709aa500f8b422499c30843acd1a5ae P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:25.613533 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 243825c6dd2d4461a01edec88f33f6ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:25.613598 1218 raft_consensus.cc:493] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:25.613835 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet d684da8134ce480693f470acdac176d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:25.613799 1218 raft_consensus.cc:515] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.615166 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e84dbbd4a0bd4e4dbf1212fefbf00fd4" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:25.615403 775 raft_consensus.cc:2468] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:25.615824 847 leader_election.cc:304] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:25.616128 1228 raft_consensus.cc:2804] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:25.616219 1228 raft_consensus.cc:493] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:25.616233 1167 tablet_bootstrap.cc:492] T 89bfb7e2f051454aae0af5755a79ca58 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:25.616345 1228 raft_consensus.cc:3060] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:25.616407 1167 ts_tablet_manager.cc:1403] T 89bfb7e2f051454aae0af5755a79ca58 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20251025 07:52:25.616739 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e84dbbd4a0bd4e4dbf1212fefbf00fd4" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:25.617266 850 leader_election.cc:343] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:25.617448 1218 leader_election.cc:290] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:25.617372 1167 raft_consensus.cc:359] T 89bfb7e2f051454aae0af5755a79ca58 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.617573 1167 raft_consensus.cc:385] T 89bfb7e2f051454aae0af5755a79ca58 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:25.617672 1167 raft_consensus.cc:740] T 89bfb7e2f051454aae0af5755a79ca58 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:25.617969 1167 consensus_queue.cc:260] T 89bfb7e2f051454aae0af5755a79ca58 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.618536 1167 ts_tablet_manager.cc:1434] T 89bfb7e2f051454aae0af5755a79ca58 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
W20251025 07:52:25.619272 1214 consensus_queue.cc:1175] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "23eeffd3de254ce18db5febfde971c6b" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20251025 07:52:25.619971 1228 raft_consensus.cc:515] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.621101 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e84dbbd4a0bd4e4dbf1212fefbf00fd4" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:25.621287 775 raft_consensus.cc:3060] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:25.622871 1167 tablet_bootstrap.cc:492] T 14fd790ba7054c1fb09cdfb19b001987 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:25.624351 775 raft_consensus.cc:2468] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:25.624907 847 leader_election.cc:304] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:25.625285 1166 tablet_bootstrap.cc:492] T d684da8134ce480693f470acdac176d4 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:25.625434 1166 ts_tablet_manager.cc:1403] T d684da8134ce480693f470acdac176d4 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.291s user 0.010s sys 0.005s
I20251025 07:52:25.626276 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e84dbbd4a0bd4e4dbf1212fefbf00fd4" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:25.626269 1166 raft_consensus.cc:359] T d684da8134ce480693f470acdac176d4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.626443 1166 raft_consensus.cc:385] T d684da8134ce480693f470acdac176d4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:25.626535 1166 raft_consensus.cc:740] T d684da8134ce480693f470acdac176d4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
W20251025 07:52:25.626838 850 leader_election.cc:343] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:25.627009 1228 leader_election.cc:290] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:25.626833 1166 consensus_queue.cc:260] T d684da8134ce480693f470acdac176d4 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.627300 1228 raft_consensus.cc:2804] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:25.627697 1166 ts_tablet_manager.cc:1434] T d684da8134ce480693f470acdac176d4 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251025 07:52:25.628010 1166 tablet_bootstrap.cc:492] T 89bfb7e2f051454aae0af5755a79ca58 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:25.628860 1171 ts_tablet_manager.cc:1916] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:25.631469 1167 tablet_bootstrap.cc:654] T 14fd790ba7054c1fb09cdfb19b001987 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:25.638749 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 796413da731245459cc5dffacc3deb2f consensus state has no leader
I20251025 07:52:25.643803 1024 tablet_service.cc:1505] Processing CreateTablet for tablet dd898450f2604987b08639cf87d0e402 (DEFAULT_TABLE table=default.Table_56f5f8c6d70645309e7ea14dcf847a43 [id=2dd014b2a3a94681967660e0399d334a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.644043 1027 tablet_service.cc:1505] Processing CreateTablet for tablet f51ec427dd9349a48eb221db04cf242b (DEFAULT_TABLE table=default.Table_cc49d933a4b343aeb2220c3939495834 [id=e5e538e59868418e9805492aa80f3240]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.644273 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd898450f2604987b08639cf87d0e402. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.644477 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f51ec427dd9349a48eb221db04cf242b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.644812 1020 tablet_service.cc:1505] Processing CreateTablet for tablet 5caac9c943bf42a5b2ae522d87fdfe8e (DEFAULT_TABLE table=default.Table_672fb5e1742840ae8235d645567de473 [id=ed06dbbb96a24ab39619f4186ba93191]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.645179 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5caac9c943bf42a5b2ae522d87fdfe8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.648996 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_cc49d933a4b343aeb2220c3939495834" } modify_external_catalogs: true
I20251025 07:52:25.649312 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_cc49d933a4b343aeb2220c3939495834" } modify_external_catalogs: true
I20251025 07:52:25.653347 1021 tablet_service.cc:1505] Processing CreateTablet for tablet 1483c40eee834f20a92e6b4cd89775b6 (DEFAULT_TABLE table=default.Table_e0bf55e24b084fdbb62f09cce80b50c5 [id=084833e021c24b4a84dd1b1a3bb08dfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.653795 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1483c40eee834f20a92e6b4cd89775b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.654084 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 1b31177a062b467187d68ed2034f5690 (DEFAULT_TABLE table=default.Table_66a70e784e9b49cdb516f657733f2a38 [id=0827603890c8466082bc6c306c9c66af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.654453 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b31177a062b467187d68ed2034f5690. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.655956 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 87a93778620f4c84b3b3f7a3d7d7e87f (DEFAULT_TABLE table=default.Table_0eca32ca64da44198140ae7ec9a77767 [id=2da58c150b514203abed5e20ffa98919]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.658519 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87a93778620f4c84b3b3f7a3d7d7e87f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.663576 524 master_service.cc:954] ReplaceTablet: received request to replace tablet aac844f429fc4328a261a326181f6c35 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:25.666283 744 tablet_service.cc:1505] Processing CreateTablet for tablet 5caac9c943bf42a5b2ae522d87fdfe8e (DEFAULT_TABLE table=default.Table_672fb5e1742840ae8235d645567de473 [id=ed06dbbb96a24ab39619f4186ba93191]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.666608 1170 ts_tablet_manager.cc:1916] T d709aa500f8b422499c30843acd1a5ae P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:25.646234 1218 raft_consensus.cc:493] T 5680e1adaf3f4b7f8c5e05523dd0950a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:25.648414 885 tablet_service.cc:1505] Processing CreateTablet for tablet 87a93778620f4c84b3b3f7a3d7d7e87f (DEFAULT_TABLE table=default.Table_0eca32ca64da44198140ae7ec9a77767 [id=2da58c150b514203abed5e20ffa98919]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.668503 1218 raft_consensus.cc:515] T 5680e1adaf3f4b7f8c5e05523dd0950a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.668749 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87a93778620f4c84b3b3f7a3d7d7e87f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.669166 1218 leader_election.cc:290] T 5680e1adaf3f4b7f8c5e05523dd0950a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:25.648638 884 tablet_service.cc:1505] Processing CreateTablet for tablet 5caac9c943bf42a5b2ae522d87fdfe8e (DEFAULT_TABLE table=default.Table_672fb5e1742840ae8235d645567de473 [id=ed06dbbb96a24ab39619f4186ba93191]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.670503 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5caac9c943bf42a5b2ae522d87fdfe8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.671023 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet d0fd00baa02d48278afaa5b8c19182f0
I20251025 07:52:25.671608 1166 tablet_bootstrap.cc:654] T 89bfb7e2f051454aae0af5755a79ca58 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:25.671783 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet f51ec427dd9349a48eb221db04cf242b
I20251025 07:52:25.673732 894 tablet_service.cc:1552] Processing DeleteTablet for tablet d0fd00baa02d48278afaa5b8c19182f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:25.673967 887 tablet_service.cc:1552] Processing DeleteTablet for tablet f51ec427dd9349a48eb221db04cf242b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:25.648873 892 tablet_service.cc:1505] Processing CreateTablet for tablet dd898450f2604987b08639cf87d0e402 (DEFAULT_TABLE table=default.Table_56f5f8c6d70645309e7ea14dcf847a43 [id=2dd014b2a3a94681967660e0399d334a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.654439 889 tablet_service.cc:1505] Processing CreateTablet for tablet f51ec427dd9349a48eb221db04cf242b (DEFAULT_TABLE table=default.Table_cc49d933a4b343aeb2220c3939495834 [id=e5e538e59868418e9805492aa80f3240]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.656065 1172 ts_tablet_manager.cc:1929] T aac844f429fc4328a261a326181f6c35 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:25.657083 883 tablet_service.cc:1505] Processing CreateTablet for tablet 1483c40eee834f20a92e6b4cd89775b6 (DEFAULT_TABLE table=default.Table_e0bf55e24b084fdbb62f09cce80b50c5 [id=084833e021c24b4a84dd1b1a3bb08dfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.659870 890 tablet_service.cc:1505] Processing CreateTablet for tablet 1b31177a062b467187d68ed2034f5690 (DEFAULT_TABLE table=default.Table_66a70e784e9b49cdb516f657733f2a38 [id=0827603890c8466082bc6c306c9c66af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.674459 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd898450f2604987b08639cf87d0e402. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.674659 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b31177a062b467187d68ed2034f5690. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.674819 746 tablet_service.cc:1505] Processing CreateTablet for tablet f51ec427dd9349a48eb221db04cf242b (DEFAULT_TABLE table=default.Table_cc49d933a4b343aeb2220c3939495834 [id=e5e538e59868418e9805492aa80f3240]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.678654 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1483c40eee834f20a92e6b4cd89775b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.684505 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f51ec427dd9349a48eb221db04cf242b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.687096 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_b6cf4ff8abe642088cedf32609854bdd" } modify_external_catalogs: true
I20251025 07:52:25.687326 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_b6cf4ff8abe642088cedf32609854bdd" } modify_external_catalogs: true
I20251025 07:52:25.688441 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_60f1e862b7bc4bdba20dd152856f55e3" } new_table_name: "default.Table_69658b94854242fcbb2acaa8bfb49850" modify_external_catalogs: true
I20251025 07:52:25.689743 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_373f40b4ac044088a071a6d26d7a6da0" } modify_external_catalogs: true
I20251025 07:52:25.689893 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_373f40b4ac044088a071a6d26d7a6da0" } modify_external_catalogs: true
I20251025 07:52:25.692106 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_974bcc124afb435285e042c78588d367" } new_table_name: "default.Table_45b1ec3be7e9460296fe5b819beaa6ea" modify_external_catalogs: true
I20251025 07:52:25.694226 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5caac9c943bf42a5b2ae522d87fdfe8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.696594 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5680e1adaf3f4b7f8c5e05523dd0950a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:25.659953 1228 raft_consensus.cc:697] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:25.697363 756 tablet_service.cc:1505] Processing CreateTablet for tablet 1b31177a062b467187d68ed2034f5690 (DEFAULT_TABLE table=default.Table_66a70e784e9b49cdb516f657733f2a38 [id=0827603890c8466082bc6c306c9c66af]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:25.697552 850 leader_election.cc:343] T 5680e1adaf3f4b7f8c5e05523dd0950a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:25.697403 1228 consensus_queue.cc:237] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.699079 1172 log.cc:1199] T aac844f429fc4328a261a326181f6c35 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/aac844f429fc4328a261a326181f6c35
I20251025 07:52:25.699594 1172 ts_tablet_manager.cc:1950] T aac844f429fc4328a261a326181f6c35 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:25.699983 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet f51ec427dd9349a48eb221db04cf242b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:25.700182 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet d0fd00baa02d48278afaa5b8c19182f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:25.700870 1172 tablet_replica.cc:333] T d709aa500f8b422499c30843acd1a5ae P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:25.701100 1172 raft_consensus.cc:2243] T d709aa500f8b422499c30843acd1a5ae P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:25.701342 1172 raft_consensus.cc:2272] T d709aa500f8b422499c30843acd1a5ae P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:25.701402 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f51ec427dd9349a48eb221db04cf242b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.701958 1172 ts_tablet_manager.cc:1916] T d709aa500f8b422499c30843acd1a5ae P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:25.706974 1166 tablet_bootstrap.cc:492] T 89bfb7e2f051454aae0af5755a79ca58 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:25.709585 1169 tablet_bootstrap.cc:492] T 5680e1adaf3f4b7f8c5e05523dd0950a P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:25.709753 1169 ts_tablet_manager.cc:1403] T 5680e1adaf3f4b7f8c5e05523dd0950a P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.432s user 0.071s sys 0.103s
I20251025 07:52:25.710960 1169 raft_consensus.cc:359] T 5680e1adaf3f4b7f8c5e05523dd0950a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.711254 1169 raft_consensus.cc:385] T 5680e1adaf3f4b7f8c5e05523dd0950a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:25.711608 1166 ts_tablet_manager.cc:1403] T 89bfb7e2f051454aae0af5755a79ca58 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.084s user 0.003s sys 0.000s
W20251025 07:52:25.712169 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4829ec41391145cc84327d60048d9b0c consensus state has no leader
I20251025 07:52:25.712168 1169 raft_consensus.cc:740] T 5680e1adaf3f4b7f8c5e05523dd0950a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:25.712411 1169 consensus_queue.cc:260] T 5680e1adaf3f4b7f8c5e05523dd0950a P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.713047 1169 ts_tablet_manager.cc:1434] T 5680e1adaf3f4b7f8c5e05523dd0950a P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
W20251025 07:52:25.714494 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 50433881c0eb4aa5a389b6f5b4a4337a consensus state has no leader
I20251025 07:52:25.716710 753 tablet_service.cc:1552] Processing DeleteTablet for tablet d0fd00baa02d48278afaa5b8c19182f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:25.721167 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_69658b94854242fcbb2acaa8bfb49850" } new_table_name: "default.Table_c73137b8706c43b2865a93af89cd428c" modify_external_catalogs: true
I20251025 07:52:25.721311 1169 tablet_bootstrap.cc:492] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:25.721864 521 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 5680e1adaf3f4b7f8c5e05523dd0950a (table default.Table_31d7f6f73d6e4123b1ff1df86ff62205 [id=cb037f61ee0b4b6cb8fb613263a278fa]). Expected version 1 got 0
I20251025 07:52:25.723959 750 tablet_service.cc:1505] Processing CreateTablet for tablet 87a93778620f4c84b3b3f7a3d7d7e87f (DEFAULT_TABLE table=default.Table_0eca32ca64da44198140ae7ec9a77767 [id=2da58c150b514203abed5e20ffa98919]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.724023 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5680e1adaf3f4b7f8c5e05523dd0950a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:25.724427 743 tablet_service.cc:1505] Processing CreateTablet for tablet dd898450f2604987b08639cf87d0e402 (DEFAULT_TABLE table=default.Table_56f5f8c6d70645309e7ea14dcf847a43 [id=2dd014b2a3a94681967660e0399d334a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.724891 1171 ts_tablet_manager.cc:1929] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:25.725064 1171 log.cc:1199] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/f05afed029d446ea9eb0073642c7312d
I20251025 07:52:25.725210 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b31177a062b467187d68ed2034f5690. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.725456 883 tablet_service.cc:1505] Processing CreateTablet for tablet 627a10a0714f4be8bdbced503fd6388d (DEFAULT_TABLE table=default.Table_a58a5fd6549343d5bb4c540a9dde185a [id=7fc07b0c33f74604be526c4a0df949e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.725692 747 tablet_service.cc:1505] Processing CreateTablet for tablet 1483c40eee834f20a92e6b4cd89775b6 (DEFAULT_TABLE table=default.Table_e0bf55e24b084fdbb62f09cce80b50c5 [id=084833e021c24b4a84dd1b1a3bb08dfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.725862 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 627a10a0714f4be8bdbced503fd6388d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:25.727048 847 leader_election.cc:343] T 5680e1adaf3f4b7f8c5e05523dd0950a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:25.727208 847 leader_election.cc:304] T 5680e1adaf3f4b7f8c5e05523dd0950a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:25.727288 1171 ts_tablet_manager.cc:1950] T f05afed029d446ea9eb0073642c7312d P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:25.728331 1171 tablet_replica.cc:333] T d709aa500f8b422499c30843acd1a5ae P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:25.728384 885 tablet_service.cc:1505] Processing CreateTablet for tablet 90764275825c4ffb8712cb883d37a551 (DEFAULT_TABLE table=default.Table_072e0092adb64ef0ac6834dc18688ccf [id=1d4cbe4a4f38445297ceeebb8ffdaa8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.728756 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90764275825c4ffb8712cb883d37a551. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.729341 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_ca46a7a783e547998d1be8476801a9e0" } modify_external_catalogs: true
I20251025 07:52:25.729525 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_ca46a7a783e547998d1be8476801a9e0" } modify_external_catalogs: true
I20251025 07:52:25.730155 1023 tablet_service.cc:1505] Processing CreateTablet for tablet 1aeaad5c5d01478bb997254ac372d9a6 (DEFAULT_TABLE table=default.Table_ebcfacbbec3147a189ff08cec113fc55 [id=8383812e49d24d98bf0ef5f985376d98]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.730501 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aeaad5c5d01478bb997254ac372d9a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.731422 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_4145d1c9c33c4351a89976c5c4c18d75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:25.731973 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4145d1c9c33c4351a89976c5c4c18d75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:25.734268 1218 raft_consensus.cc:2749] T 5680e1adaf3f4b7f8c5e05523dd0950a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:25.734881 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_8f502389fe564ef3a4e7e23819a8a6e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:25.735311 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd898450f2604987b08639cf87d0e402. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:25.735447 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f502389fe564ef3a4e7e23819a8a6e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:25.736069 892 tablet_service.cc:1505] Processing CreateTablet for tablet 1aeaad5c5d01478bb997254ac372d9a6 (DEFAULT_TABLE table=default.Table_ebcfacbbec3147a189ff08cec113fc55 [id=8383812e49d24d98bf0ef5f985376d98]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.736451 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aeaad5c5d01478bb997254ac372d9a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.737651 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 85fba8b37f2c4540aa81701052bf222a (DEFAULT_TABLE table=default.Table_5237c35dbfa7429e803d5008a8b3487c [id=0d597ecab6f54d1ba6cb5bd3436a9db8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.738225 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85fba8b37f2c4540aa81701052bf222a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.738191 752 tablet_service.cc:1505] Processing CreateTablet for tablet 90764275825c4ffb8712cb883d37a551 (DEFAULT_TABLE table=default.Table_072e0092adb64ef0ac6834dc18688ccf [id=1d4cbe4a4f38445297ceeebb8ffdaa8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.738711 1020 tablet_service.cc:1505] Processing CreateTablet for tablet 90764275825c4ffb8712cb883d37a551 (DEFAULT_TABLE table=default.Table_072e0092adb64ef0ac6834dc18688ccf [id=1d4cbe4a4f38445297ceeebb8ffdaa8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.739432 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90764275825c4ffb8712cb883d37a551. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.740754 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 79bc2678df8042e3994b0471afa2c304 (DEFAULT_TABLE table=default.Table_ca46a7a783e547998d1be8476801a9e0 [id=525e8a793cf14cd492ca2ee986231311]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.741088 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79bc2678df8042e3994b0471afa2c304. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.741859 1021 tablet_service.cc:1505] Processing CreateTablet for tablet 627a10a0714f4be8bdbced503fd6388d (DEFAULT_TABLE table=default.Table_a58a5fd6549343d5bb4c540a9dde185a [id=7fc07b0c33f74604be526c4a0df949e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.743202 887 tablet_service.cc:1505] Processing CreateTablet for tablet 8c6ff4c73e624af3bf2de36c4493b462 (DEFAULT_TABLE table=default.Table_9f6b2f95721f488c86fde259a7cf95ec [id=277005e0d596465293065585a7555f4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.743414 884 tablet_service.cc:1505] Processing CreateTablet for tablet 79bc2678df8042e3994b0471afa2c304 (DEFAULT_TABLE table=default.Table_ca46a7a783e547998d1be8476801a9e0 [id=525e8a793cf14cd492ca2ee986231311]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.743602 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c6ff4c73e624af3bf2de36c4493b462. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.743782 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79bc2678df8042e3994b0471afa2c304. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.744086 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 47284f96928749e6a5201103a8ee5f9f (DEFAULT_TABLE table=default.Table_373f40b4ac044088a071a6d26d7a6da0 [id=05672e3da0704022b93cc96d61278433]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.748198 745 tablet_service.cc:1552] Processing DeleteTablet for tablet f51ec427dd9349a48eb221db04cf242b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:25.748520 753 tablet_service.cc:1505] Processing CreateTablet for tablet 627a10a0714f4be8bdbced503fd6388d (DEFAULT_TABLE table=default.Table_a58a5fd6549343d5bb4c540a9dde185a [id=7fc07b0c33f74604be526c4a0df949e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.748857 1171 raft_consensus.cc:2243] T d709aa500f8b422499c30843acd1a5ae P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:25.749058 1171 raft_consensus.cc:2272] T d709aa500f8b422499c30843acd1a5ae P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:25.749780 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1483c40eee834f20a92e6b4cd89775b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.750269 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 2aa522477e934c89834026dc480fe7c0 (DEFAULT_TABLE table=default.Table_7ce70fea0fe04c52a62033afb8eef977 [id=a8b2a16fd037435893c8f60e92fd1627]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.751097 894 tablet_service.cc:1505] Processing CreateTablet for tablet 85e27fc52e9a4d62a8622645594f59c6 (DEFAULT_TABLE table=default.Table_4bafafcc9e8641b687d438725186f639 [id=73f69c9574ee4019ada5af76bd902107]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.751504 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85e27fc52e9a4d62a8622645594f59c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.751977 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 8c6ff4c73e624af3bf2de36c4493b462 (DEFAULT_TABLE table=default.Table_9f6b2f95721f488c86fde259a7cf95ec [id=277005e0d596465293065585a7555f4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.752701 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 627a10a0714f4be8bdbced503fd6388d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.753032 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 85e27fc52e9a4d62a8622645594f59c6 (DEFAULT_TABLE table=default.Table_4bafafcc9e8641b687d438725186f639 [id=73f69c9574ee4019ada5af76bd902107]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.753556 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47284f96928749e6a5201103a8ee5f9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.754597 882 tablet_service.cc:1505] Processing CreateTablet for tablet 2aa522477e934c89834026dc480fe7c0 (DEFAULT_TABLE table=default.Table_7ce70fea0fe04c52a62033afb8eef977 [id=a8b2a16fd037435893c8f60e92fd1627]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.754705 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2aa522477e934c89834026dc480fe7c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.754922 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2aa522477e934c89834026dc480fe7c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.755885 1019 tablet_service.cc:1505] Processing CreateTablet for tablet cd23b8094d7e472eba6bd170248858a9 (DEFAULT_TABLE table=default.Table_5f94a1aa9fac44ae8a90d3554215d598 [id=d82b3e1317434cf08a0d51e8a9427224]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.756959 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 742b3ec3f6604647b9e397532f1673dd (DEFAULT_TABLE table=default.Table_826ad7d517d34a33a3759e733302ce66 [id=b6e173967bad47e6b306627221b80f94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.758775 886 tablet_service.cc:1505] Processing CreateTablet for tablet cd23b8094d7e472eba6bd170248858a9 (DEFAULT_TABLE table=default.Table_5f94a1aa9fac44ae8a90d3554215d598 [id=d82b3e1317434cf08a0d51e8a9427224]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.758913 749 tablet_service.cc:1505] Processing CreateTablet for tablet 1aeaad5c5d01478bb997254ac372d9a6 (DEFAULT_TABLE table=default.Table_ebcfacbbec3147a189ff08cec113fc55 [id=8383812e49d24d98bf0ef5f985376d98]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.759439 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85e27fc52e9a4d62a8622645594f59c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.759965 748 tablet_service.cc:1505] Processing CreateTablet for tablet 85fba8b37f2c4540aa81701052bf222a (DEFAULT_TABLE table=default.Table_5237c35dbfa7429e803d5008a8b3487c [id=0d597ecab6f54d1ba6cb5bd3436a9db8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.760213 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd23b8094d7e472eba6bd170248858a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.760381 745 tablet_service.cc:1505] Processing CreateTablet for tablet 2aa522477e934c89834026dc480fe7c0 (DEFAULT_TABLE table=default.Table_7ce70fea0fe04c52a62033afb8eef977 [id=a8b2a16fd037435893c8f60e92fd1627]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.761061 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd23b8094d7e472eba6bd170248858a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.759496 755 tablet_service.cc:1505] Processing CreateTablet for tablet 79bc2678df8042e3994b0471afa2c304 (DEFAULT_TABLE table=default.Table_ca46a7a783e547998d1be8476801a9e0 [id=525e8a793cf14cd492ca2ee986231311]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.761343 1171 ts_tablet_manager.cc:1916] T d709aa500f8b422499c30843acd1a5ae P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:25.761574 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 742b3ec3f6604647b9e397532f1673dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.761742 751 tablet_service.cc:1505] Processing CreateTablet for tablet 47284f96928749e6a5201103a8ee5f9f (DEFAULT_TABLE table=default.Table_373f40b4ac044088a071a6d26d7a6da0 [id=05672e3da0704022b93cc96d61278433]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.742965 754 tablet_service.cc:1505] Processing CreateTablet for tablet 8c6ff4c73e624af3bf2de36c4493b462 (DEFAULT_TABLE table=default.Table_9f6b2f95721f488c86fde259a7cf95ec [id=277005e0d596465293065585a7555f4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.762509 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c6ff4c73e624af3bf2de36c4493b462. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.763330 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90764275825c4ffb8712cb883d37a551. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.763509 1018 tablet_service.cc:1505] Processing CreateTablet for tablet 567a1f5c216a4a519249da66a2a03ea4 (DEFAULT_TABLE table=default.Table_b6cf4ff8abe642088cedf32609854bdd [id=e55f743deb3c45f9a90c2f3a0ab68a3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.763918 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 567a1f5c216a4a519249da66a2a03ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.766201 1169 tablet_bootstrap.cc:654] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:25.766381 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 627a10a0714f4be8bdbced503fd6388d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.765378 742 tablet_service.cc:1505] Processing CreateTablet for tablet 85e27fc52e9a4d62a8622645594f59c6 (DEFAULT_TABLE table=default.Table_4bafafcc9e8641b687d438725186f639 [id=73f69c9574ee4019ada5af76bd902107]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.768626 891 tablet_service.cc:1505] Processing CreateTablet for tablet 567a1f5c216a4a519249da66a2a03ea4 (DEFAULT_TABLE table=default.Table_b6cf4ff8abe642088cedf32609854bdd [id=e55f743deb3c45f9a90c2f3a0ab68a3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.769196 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 567a1f5c216a4a519249da66a2a03ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.767365 740 tablet_service.cc:1505] Processing CreateTablet for tablet 742b3ec3f6604647b9e397532f1673dd (DEFAULT_TABLE table=default.Table_826ad7d517d34a33a3759e733302ce66 [id=b6e173967bad47e6b306627221b80f94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.776584 739 tablet_service.cc:1505] Processing CreateTablet for tablet 567a1f5c216a4a519249da66a2a03ea4 (DEFAULT_TABLE table=default.Table_b6cf4ff8abe642088cedf32609854bdd [id=e55f743deb3c45f9a90c2f3a0ab68a3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.766084 741 tablet_service.cc:1505] Processing CreateTablet for tablet cd23b8094d7e472eba6bd170248858a9 (DEFAULT_TABLE table=default.Table_5f94a1aa9fac44ae8a90d3554215d598 [id=d82b3e1317434cf08a0d51e8a9427224]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.777647 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47284f96928749e6a5201103a8ee5f9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.779417 1218 raft_consensus.cc:493] T d684da8134ce480693f470acdac176d4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:25.779592 1218 raft_consensus.cc:515] T d684da8134ce480693f470acdac176d4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.780309 1218 leader_election.cc:290] T d684da8134ce480693f470acdac176d4 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:25.781952 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d684da8134ce480693f470acdac176d4" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:25.782462 1049 raft_consensus.cc:2468] T d684da8134ce480693f470acdac176d4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:25.782943 850 leader_election.cc:304] T d684da8134ce480693f470acdac176d4 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:25.783262 1218 raft_consensus.cc:2804] T d684da8134ce480693f470acdac176d4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:25.783349 1218 raft_consensus.cc:493] T d684da8134ce480693f470acdac176d4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:25.783452 1218 raft_consensus.cc:3060] T d684da8134ce480693f470acdac176d4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:25.785084 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d684da8134ce480693f470acdac176d4" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:25.785791 890 tablet_service.cc:1505] Processing CreateTablet for tablet 85fba8b37f2c4540aa81701052bf222a (DEFAULT_TABLE table=default.Table_5237c35dbfa7429e803d5008a8b3487c [id=0d597ecab6f54d1ba6cb5bd3436a9db8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.786512 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85fba8b37f2c4540aa81701052bf222a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:25.789636 847 leader_election.cc:343] T d684da8134ce480693f470acdac176d4 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:25.790413 886 tablet_service.cc:1505] Processing CreateTablet for tablet 742b3ec3f6604647b9e397532f1673dd (DEFAULT_TABLE table=default.Table_826ad7d517d34a33a3759e733302ce66 [id=b6e173967bad47e6b306627221b80f94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.790870 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 742b3ec3f6604647b9e397532f1673dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.791594 1218 raft_consensus.cc:515] T d684da8134ce480693f470acdac176d4 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.792475 1218 leader_election.cc:290] T d684da8134ce480693f470acdac176d4 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:25.793483 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d684da8134ce480693f470acdac176d4" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:25.793409 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_59de6322049942169d98f535889113e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:25.793743 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_3055bcb146aa43d883f4e7208f6c6638"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:25.793912 1049 raft_consensus.cc:3060] T d684da8134ce480693f470acdac176d4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:25.794066 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59de6322049942169d98f535889113e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:25.794286 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3055bcb146aa43d883f4e7208f6c6638 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:25.795578 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_a0015f25f2554a0caeac8b446759378b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:25.796046 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0015f25f2554a0caeac8b446759378b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:25.796049 887 tablet_service.cc:1505] Processing CreateTablet for tablet 47284f96928749e6a5201103a8ee5f9f (DEFAULT_TABLE table=default.Table_373f40b4ac044088a071a6d26d7a6da0 [id=05672e3da0704022b93cc96d61278433]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.797047 1049 raft_consensus.cc:2468] T d684da8134ce480693f470acdac176d4 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:25.797194 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_26ee7c4758284bc0bf685f326ebfd478" } new_table_name: "default.Table_0be1d71825644f1abbd428583fa6ddfa" modify_external_catalogs: true
I20251025 07:52:25.797490 850 leader_election.cc:304] T d684da8134ce480693f470acdac176d4 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:25.801532 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47284f96928749e6a5201103a8ee5f9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.806310 1218 raft_consensus.cc:2804] T d684da8134ce480693f470acdac176d4 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:25.806509 1218 raft_consensus.cc:697] T d684da8134ce480693f470acdac176d4 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:25.806792 1218 consensus_queue.cc:237] T d684da8134ce480693f470acdac176d4 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.807246 1170 ts_tablet_manager.cc:1929] T d709aa500f8b422499c30843acd1a5ae P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:25.807384 1170 log.cc:1199] T d709aa500f8b422499c30843acd1a5ae P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/d709aa500f8b422499c30843acd1a5ae
I20251025 07:52:25.807854 1170 ts_tablet_manager.cc:1950] T d709aa500f8b422499c30843acd1a5ae P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:25.808019 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d684da8134ce480693f470acdac176d4" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:25.808524 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87a93778620f4c84b3b3f7a3d7d7e87f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.809021 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet d709aa500f8b422499c30843acd1a5ae (table default.Table_33eaf5d540914e43aad9bc6e22eb698a [id=768d2afbbcb24c26818cf1f64b330428]) successfully deleted
I20251025 07:52:25.809437 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 742b3ec3f6604647b9e397532f1673dd. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:25.809689 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet a231f3d31b204f7b82f018892ccc8435 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a231f3d31b204f7b82f018892ccc8435 already in progress: creating tablet
I20251025 07:52:25.810791 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd23b8094d7e472eba6bd170248858a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.811496 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aeaad5c5d01478bb997254ac372d9a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.812139 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79bc2678df8042e3994b0471afa2c304. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.812659 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 567a1f5c216a4a519249da66a2a03ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.813321 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85fba8b37f2c4540aa81701052bf222a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.813459 1166 raft_consensus.cc:359] T 89bfb7e2f051454aae0af5755a79ca58 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.813932 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85e27fc52e9a4d62a8622645594f59c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.814739 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c6ff4c73e624af3bf2de36c4493b462. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.815336 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2aa522477e934c89834026dc480fe7c0. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:25.818368 847 leader_election.cc:343] T d684da8134ce480693f470acdac176d4 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:25.823064 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 576c51d9a22d45f6a3642a7296b5bf23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 576c51d9a22d45f6a3642a7296b5bf23 already in progress: creating tablet
I20251025 07:52:25.823294 1166 raft_consensus.cc:385] T 89bfb7e2f051454aae0af5755a79ca58 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:25.823551 1166 raft_consensus.cc:740] T 89bfb7e2f051454aae0af5755a79ca58 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
W20251025 07:52:25.823714 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 9cf88862e0934634b3dd172e6b40a0ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9cf88862e0934634b3dd172e6b40a0ba already in progress: creating tablet
I20251025 07:52:25.824182 1166 consensus_queue.cc:260] T 89bfb7e2f051454aae0af5755a79ca58 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.826844 1170 tablet_replica.cc:333] T f05afed029d446ea9eb0073642c7312d P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:25.829202 1166 ts_tablet_manager.cc:1434] T 89bfb7e2f051454aae0af5755a79ca58 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.113s user 0.001s sys 0.000s
I20251025 07:52:25.843180 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 47284f96928749e6a5201103a8ee5f9f
I20251025 07:52:25.843333 1166 tablet_bootstrap.cc:492] T 54cb4953ff264c13a0cc730b4c452ecc P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:25.844534 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 567a1f5c216a4a519249da66a2a03ea4
I20251025 07:52:25.845256 1166 tablet_bootstrap.cc:654] T 54cb4953ff264c13a0cc730b4c452ecc P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:25.848264 1170 raft_consensus.cc:2243] T f05afed029d446ea9eb0073642c7312d P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:25.851004 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet 47284f96928749e6a5201103a8ee5f9f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:25.851243 1170 raft_consensus.cc:2272] T f05afed029d446ea9eb0073642c7312d P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:25.851912 1170 ts_tablet_manager.cc:1916] T f05afed029d446ea9eb0073642c7312d P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:25.852238 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 50433881c0eb4aa5a389b6f5b4a4337a consensus state has no leader
I20251025 07:52:25.852674 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_e7d32e91b9c64ce1b3fcff3d806e959f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:25.853279 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e7d32e91b9c64ce1b3fcff3d806e959f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:25.854523 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 79bc2678df8042e3994b0471afa2c304
I20251025 07:52:25.855283 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_4228cd02e10a4b65ad0134b261130561"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:25.855765 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4228cd02e10a4b65ad0134b261130561 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:25.856036 1172 ts_tablet_manager.cc:1929] T d709aa500f8b422499c30843acd1a5ae P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:25.856168 1172 log.cc:1199] T d709aa500f8b422499c30843acd1a5ae P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/d709aa500f8b422499c30843acd1a5ae
I20251025 07:52:25.857247 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_c9a55d4584dd403ca33bc5fcf06235e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:25.857743 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9a55d4584dd403ca33bc5fcf06235e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:25.859318 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_672fb5e1742840ae8235d645567de473" } new_table_name: "default.Table_60bfcc345b8649e9bc73fe3d077f46c7" modify_external_catalogs: true
W20251025 07:52:25.860517 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 50ffbcdd4a9247bf99b8e2361bc02ef5 consensus state has no leader
I20251025 07:52:25.861826 1172 ts_tablet_manager.cc:1950] T d709aa500f8b422499c30843acd1a5ae P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:25.864400 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 79bc2678df8042e3994b0471afa2c304 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:25.867820 1169 tablet_bootstrap.cc:492] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:25.868072 1169 ts_tablet_manager.cc:1403] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.155s user 0.002s sys 0.005s
I20251025 07:52:25.862169 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 21) took 1124 ms. Trace:
I20251025 07:52:25.868454 1172 rpcz_store.cc:276] 1025 07:52:24.737290 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:24.737757 (+ 467us) service_pool.cc:225] Handling call
1025 07:52:25.700795 (+963038us) ts_tablet_manager.cc:1151] Deleting tablet d709aa500f8b422499c30843acd1a5ae
1025 07:52:25.700803 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:25.818456 (+117653us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:25.855890 (+ 37434us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:25.862119 (+ 6229us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":962838}
I20251025 07:52:25.868912 1169 raft_consensus.cc:359] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.869204 1169 raft_consensus.cc:385] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:25.869335 1169 raft_consensus.cc:740] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:25.869475 1172 tablet_replica.cc:333] T d684da8134ce480693f470acdac176d4 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:25.869652 1169 consensus_queue.cc:260] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.870257 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet d709aa500f8b422499c30843acd1a5ae (table default.Table_33eaf5d540914e43aad9bc6e22eb698a [id=768d2afbbcb24c26818cf1f64b330428]) successfully deleted
I20251025 07:52:25.872677 1166 tablet_bootstrap.cc:492] T 54cb4953ff264c13a0cc730b4c452ecc P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:25.872898 1166 ts_tablet_manager.cc:1403] T 54cb4953ff264c13a0cc730b4c452ecc P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.001s
I20251025 07:52:25.873155 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_5aff13b5fcbc4899800760718fb3cc90"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:25.873771 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5aff13b5fcbc4899800760718fb3cc90 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:25.873745 1166 raft_consensus.cc:359] T 54cb4953ff264c13a0cc730b4c452ecc P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.874054 1169 ts_tablet_manager.cc:1434] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251025 07:52:25.873940 1166 raft_consensus.cc:385] T 54cb4953ff264c13a0cc730b4c452ecc P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:25.874213 1166 raft_consensus.cc:740] T 54cb4953ff264c13a0cc730b4c452ecc P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:25.874614 1166 consensus_queue.cc:260] T 54cb4953ff264c13a0cc730b4c452ecc P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.876075 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_56f5f8c6d70645309e7ea14dcf847a43" } modify_external_catalogs: true
I20251025 07:52:25.876245 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_56f5f8c6d70645309e7ea14dcf847a43" } modify_external_catalogs: true
I20251025 07:52:25.876713 1166 ts_tablet_manager.cc:1434] T 54cb4953ff264c13a0cc730b4c452ecc P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251025 07:52:25.876971 1169 tablet_bootstrap.cc:492] T 89bfb7e2f051454aae0af5755a79ca58 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:25.877576 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_ac1708f4dd0545f4b3411b218a61b6a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:25.878067 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac1708f4dd0545f4b3411b218a61b6a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:25.878661 1169 tablet_bootstrap.cc:654] T 89bfb7e2f051454aae0af5755a79ca58 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:25.879284 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_3055bcb146aa43d883f4e7208f6c6638" } modify_external_catalogs: true
I20251025 07:52:25.879422 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_3055bcb146aa43d883f4e7208f6c6638" } modify_external_catalogs: true
I20251025 07:52:25.881330 1166 tablet_bootstrap.cc:492] T b946855345bc473aadd00cc4db5b7cd8 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:25.883531 1166 tablet_bootstrap.cc:654] T b946855345bc473aadd00cc4db5b7cd8 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:25.884727 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 567a1f5c216a4a519249da66a2a03ea4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:25.885087 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 79bc2678df8042e3994b0471afa2c304 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:25.886058 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 47284f96928749e6a5201103a8ee5f9f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:25.889323 1170 ts_tablet_manager.cc:1929] T f05afed029d446ea9eb0073642c7312d P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:25.893819 1170 log.cc:1199] T f05afed029d446ea9eb0073642c7312d P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/f05afed029d446ea9eb0073642c7312d
I20251025 07:52:25.894335 1170 ts_tablet_manager.cc:1950] T f05afed029d446ea9eb0073642c7312d P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:25.895303 525 master_service.cc:954] ReplaceTablet: received request to replace tablet 1b31177a062b467187d68ed2034f5690 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:25.893695 1169 tablet_bootstrap.cc:492] T 89bfb7e2f051454aae0af5755a79ca58 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:25.895618 1169 ts_tablet_manager.cc:1403] T 89bfb7e2f051454aae0af5755a79ca58 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20251025 07:52:25.896474 1169 raft_consensus.cc:359] T 89bfb7e2f051454aae0af5755a79ca58 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.896760 1169 raft_consensus.cc:385] T 89bfb7e2f051454aae0af5755a79ca58 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:25.896901 1169 raft_consensus.cc:740] T 89bfb7e2f051454aae0af5755a79ca58 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:25.897330 1169 consensus_queue.cc:260] T 89bfb7e2f051454aae0af5755a79ca58 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.897822 1172 raft_consensus.cc:2243] T d684da8134ce480693f470acdac176d4 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:25.901162 1169 ts_tablet_manager.cc:1434] T 89bfb7e2f051454aae0af5755a79ca58 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251025 07:52:25.902714 1172 pending_rounds.cc:70] T d684da8134ce480693f470acdac176d4 P 3aea2869ff814cc7b142012faa274382: Trying to abort 1 pending ops.
I20251025 07:52:25.902989 1169 tablet_bootstrap.cc:492] T 54cb4953ff264c13a0cc730b4c452ecc P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:25.902995 1172 pending_rounds.cc:77] T d684da8134ce480693f470acdac176d4 P 3aea2869ff814cc7b142012faa274382: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214607342860021760 op_type: NO_OP noop_request { }
I20251025 07:52:25.903247 1172 raft_consensus.cc:2889] T d684da8134ce480693f470acdac176d4 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 07:52:25.903821 1172 raft_consensus.cc:2272] T d684da8134ce480693f470acdac176d4 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:25.907034 519 master_service.cc:954] ReplaceTablet: received request to replace tablet 785d0ce85b234826ba8021fbd182bf3c from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:25.907476 519 master_service.cc:954] ReplaceTablet: received request to replace tablet 3be8063aee794d2abcfb1d6e24209c06 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:25.908047 1169 tablet_bootstrap.cc:654] T 54cb4953ff264c13a0cc730b4c452ecc P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:25.909642 521 master_service.cc:954] ReplaceTablet: received request to replace tablet 90764275825c4ffb8712cb883d37a551 from {username='slave'} at 127.0.0.1:45978
W20251025 07:52:25.910168 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 785d0ce85b234826ba8021fbd182bf3c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 785d0ce85b234826ba8021fbd182bf3c already in progress: creating tablet
W20251025 07:52:25.910404 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 2947502815d742e89cacb474b7fbb249 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2947502815d742e89cacb474b7fbb249 already in progress: creating tablet
W20251025 07:52:25.910580 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 14fd790ba7054c1fb09cdfb19b001987 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14fd790ba7054c1fb09cdfb19b001987 already in progress: creating tablet
I20251025 07:52:25.911520 519 master_service.cc:954] ReplaceTablet: received request to replace tablet b946855345bc473aadd00cc4db5b7cd8 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:25.912492 1170 tablet_replica.cc:333] T 89bfb7e2f051454aae0af5755a79ca58 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
W20251025 07:52:25.912683 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5caac9c943bf42a5b2ae522d87fdfe8e consensus state has no leader
I20251025 07:52:25.912674 1170 raft_consensus.cc:2243] T 89bfb7e2f051454aae0af5755a79ca58 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:25.912876 1170 raft_consensus.cc:2272] T 89bfb7e2f051454aae0af5755a79ca58 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:25.913312 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet dd898450f2604987b08639cf87d0e402
I20251025 07:52:25.913564 1170 ts_tablet_manager.cc:1916] T 89bfb7e2f051454aae0af5755a79ca58 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:25.913829 1172 ts_tablet_manager.cc:1916] T d684da8134ce480693f470acdac176d4 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:25.916774 887 tablet_service.cc:1552] Processing DeleteTablet for tablet dd898450f2604987b08639cf87d0e402 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:25.920073 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_c9a55d4584dd403ca33bc5fcf06235e7" } new_table_name: "default.Table_e11142c0939b457d8dc181f50a9b8310" modify_external_catalogs: true
I20251025 07:52:25.921600 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_0be1d71825644f1abbd428583fa6ddfa" } new_table_name: "default.Table_0042201efe904a84936c293cd01cebde" modify_external_catalogs: true
I20251025 07:52:25.922235 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 79bc2678df8042e3994b0471afa2c304 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:48830
W20251025 07:52:25.924757 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 33c608beb4ef4a859a135b11e07c81a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33c608beb4ef4a859a135b11e07c81a4 already in progress: creating tablet
I20251025 07:52:25.924926 1170 ts_tablet_manager.cc:1929] T 89bfb7e2f051454aae0af5755a79ca58 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:25.925040 1170 log.cc:1199] T 89bfb7e2f051454aae0af5755a79ca58 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/89bfb7e2f051454aae0af5755a79ca58
W20251025 07:52:25.925067 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet cf2cab1552bd4e7394797c93b149f450 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf2cab1552bd4e7394797c93b149f450 already in progress: creating tablet
I20251025 07:52:25.925562 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet dd898450f2604987b08639cf87d0e402 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:25.936164 1170 ts_tablet_manager.cc:1950] T 89bfb7e2f051454aae0af5755a79ca58 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:25.937633 1172 ts_tablet_manager.cc:1929] T d684da8134ce480693f470acdac176d4 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251025 07:52:25.938438 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a13bf4d7eb1440f7a28b85cb1bf99667 has no consensus state
I20251025 07:52:25.938670 1170 tablet_replica.cc:333] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
W20251025 07:52:25.938874 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 50ffbcdd4a9247bf99b8e2361bc02ef5 consensus state has no leader
I20251025 07:52:25.938848 1170 raft_consensus.cc:2243] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:25.939039 1170 raft_consensus.cc:2272] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:25.939045 1172 log.cc:1199] T d684da8134ce480693f470acdac176d4 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/d684da8134ce480693f470acdac176d4
I20251025 07:52:25.940279 890 tablet_service.cc:1505] Processing CreateTablet for tablet 08da9d2687704725aacd11cd92c6a9f9 (DEFAULT_TABLE table=default.Table_3055bcb146aa43d883f4e7208f6c6638 [id=78236f2583524e1fa95840afb7aef43f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.940474 1170 ts_tablet_manager.cc:1916] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:25.940655 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08da9d2687704725aacd11cd92c6a9f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.941327 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 89bfb7e2f051454aae0af5755a79ca58 (table default.Table_5f94a1aa9fac44ae8a90d3554215d598 [id=d82b3e1317434cf08a0d51e8a9427224]) successfully deleted
I20251025 07:52:25.942173 1021 tablet_service.cc:1505] Processing CreateTablet for tablet 08da9d2687704725aacd11cd92c6a9f9 (DEFAULT_TABLE table=default.Table_3055bcb146aa43d883f4e7208f6c6638 [id=78236f2583524e1fa95840afb7aef43f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.942379 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 5cba9c05dd424cc788febc614a34819a (DEFAULT_TABLE table=default.Table_a0015f25f2554a0caeac8b446759378b [id=51b12903e6024e1a9adc2f9fa7400937]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.942430 886 tablet_service.cc:1505] Processing CreateTablet for tablet 8e6d76b1a97945cfb6c3055e1b622f8a (DEFAULT_TABLE table=default.Table_59de6322049942169d98f535889113e3 [id=f5c1be203e2b4831a025871cf8a2ef17]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.942580 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08da9d2687704725aacd11cd92c6a9f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.942785 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e6d76b1a97945cfb6c3055e1b622f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.944011 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cba9c05dd424cc788febc614a34819a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.946416 1020 tablet_service.cc:1505] Processing CreateTablet for tablet 7e869c65771f4339a546dfeebb227c62 (DEFAULT_TABLE table=default.Table_8f502389fe564ef3a4e7e23819a8a6e8 [id=56f4e62bc2354befa7d7374e07b103de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.946893 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 8e6d76b1a97945cfb6c3055e1b622f8a (DEFAULT_TABLE table=default.Table_59de6322049942169d98f535889113e3 [id=f5c1be203e2b4831a025871cf8a2ef17]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.947278 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e869c65771f4339a546dfeebb227c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.947261 887 tablet_service.cc:1505] Processing CreateTablet for tablet 5cba9c05dd424cc788febc614a34819a (DEFAULT_TABLE table=default.Table_a0015f25f2554a0caeac8b446759378b [id=51b12903e6024e1a9adc2f9fa7400937]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.947588 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cba9c05dd424cc788febc614a34819a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.950364 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e6d76b1a97945cfb6c3055e1b622f8a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:25.951167 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:43563, user_credentials={real_user=slave}} blocked reactor thread for 44835.3us
I20251025 07:52:25.951853 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 567a1f5c216a4a519249da66a2a03ea4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:25.952626 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 47284f96928749e6a5201103a8ee5f9f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:25.954242 739 tablet_service.cc:1505] Processing CreateTablet for tablet 5cba9c05dd424cc788febc614a34819a (DEFAULT_TABLE table=default.Table_a0015f25f2554a0caeac8b446759378b [id=51b12903e6024e1a9adc2f9fa7400937]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.954686 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cba9c05dd424cc788febc614a34819a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.956508 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 567a1f5c216a4a519249da66a2a03ea4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:25.957139 752 tablet_service.cc:1505] Processing CreateTablet for tablet 8e6d76b1a97945cfb6c3055e1b622f8a (DEFAULT_TABLE table=default.Table_59de6322049942169d98f535889113e3 [id=f5c1be203e2b4831a025871cf8a2ef17]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.958319 1171 ts_tablet_manager.cc:1929] T d709aa500f8b422499c30843acd1a5ae P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:25.958720 743 tablet_service.cc:1552] Processing DeleteTablet for tablet dd898450f2604987b08639cf87d0e402 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:25.958983 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e6d76b1a97945cfb6c3055e1b622f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.962299 740 tablet_service.cc:1505] Processing CreateTablet for tablet 08da9d2687704725aacd11cd92c6a9f9 (DEFAULT_TABLE table=default.Table_3055bcb146aa43d883f4e7208f6c6638 [id=78236f2583524e1fa95840afb7aef43f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.962698 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08da9d2687704725aacd11cd92c6a9f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.965920 1171 log.cc:1199] T d709aa500f8b422499c30843acd1a5ae P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/d709aa500f8b422499c30843acd1a5ae
I20251025 07:52:25.966001 894 tablet_service.cc:1505] Processing CreateTablet for tablet 7e869c65771f4339a546dfeebb227c62 (DEFAULT_TABLE table=default.Table_8f502389fe564ef3a4e7e23819a8a6e8 [id=56f4e62bc2354befa7d7374e07b103de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.966429 1171 ts_tablet_manager.cc:1950] T d709aa500f8b422499c30843acd1a5ae P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:25.966446 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e869c65771f4339a546dfeebb227c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.967501 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet d709aa500f8b422499c30843acd1a5ae (table default.Table_33eaf5d540914e43aad9bc6e22eb698a [id=768d2afbbcb24c26818cf1f64b330428]) successfully deleted
W20251025 07:52:25.968182 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 20e4178c16bd4f18b1c6bc969f6a0c6c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20e4178c16bd4f18b1c6bc969f6a0c6c already in progress: creating tablet
W20251025 07:52:25.969303 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 713ae1dd444544bdabdd4a771e109d90 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 713ae1dd444544bdabdd4a771e109d90 already in progress: creating tablet
I20251025 07:52:25.969748 1171 tablet_replica.cc:333] T 243825c6dd2d4461a01edec88f33f6ee P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:25.970044 1171 raft_consensus.cc:2243] T 243825c6dd2d4461a01edec88f33f6ee P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:25.970366 1171 raft_consensus.cc:2272] T 243825c6dd2d4461a01edec88f33f6ee P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:25.971041 1171 ts_tablet_manager.cc:1916] T 243825c6dd2d4461a01edec88f33f6ee P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:25.971757 753 tablet_service.cc:1505] Processing CreateTablet for tablet 7e869c65771f4339a546dfeebb227c62 (DEFAULT_TABLE table=default.Table_8f502389fe564ef3a4e7e23819a8a6e8 [id=56f4e62bc2354befa7d7374e07b103de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:25.983785 1167 tablet_bootstrap.cc:492] T 14fd790ba7054c1fb09cdfb19b001987 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:25.985999 1167 ts_tablet_manager.cc:1403] T 14fd790ba7054c1fb09cdfb19b001987 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.363s user 0.014s sys 0.014s
I20251025 07:52:25.986830 1167 raft_consensus.cc:359] T 14fd790ba7054c1fb09cdfb19b001987 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.987154 1167 raft_consensus.cc:385] T 14fd790ba7054c1fb09cdfb19b001987 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:25.987270 1167 raft_consensus.cc:740] T 14fd790ba7054c1fb09cdfb19b001987 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:25.987571 1167 consensus_queue.cc:260] T 14fd790ba7054c1fb09cdfb19b001987 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:25.988399 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet b946855345bc473aadd00cc4db5b7cd8
I20251025 07:52:25.988934 519 catalog_manager.cc:6719] ReplaceTablet: tablet b946855345bc473aadd00cc4db5b7cd8 deleted and replaced by tablet fab67b56eaa44b54886c17181d8c52b2
I20251025 07:52:25.989362 1167 ts_tablet_manager.cc:1434] T 14fd790ba7054c1fb09cdfb19b001987 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251025 07:52:25.989672 1167 tablet_bootstrap.cc:492] T 5680e1adaf3f4b7f8c5e05523dd0950a P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:25.990989 750 tablet_service.cc:1552] Processing DeleteTablet for tablet b946855345bc473aadd00cc4db5b7cd8 with delete_type TABLET_DATA_DELETED (replaced by tablet fab67b56eaa44b54886c17181d8c52b2) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:25.991277 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet b946855345bc473aadd00cc4db5b7cd8 with delete_type TABLET_DATA_DELETED (replaced by tablet fab67b56eaa44b54886c17181d8c52b2) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:25.992354 887 tablet_service.cc:1552] Processing DeleteTablet for tablet b946855345bc473aadd00cc4db5b7cd8 with delete_type TABLET_DATA_DELETED (replaced by tablet fab67b56eaa44b54886c17181d8c52b2) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:25.994427 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e869c65771f4339a546dfeebb227c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:25.994531 1172 ts_tablet_manager.cc:1950] T d684da8134ce480693f470acdac176d4 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:26.004613 1166 tablet_bootstrap.cc:492] T b946855345bc473aadd00cc4db5b7cd8 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:26.005226 1166 ts_tablet_manager.cc:1403] T b946855345bc473aadd00cc4db5b7cd8 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.127s user 0.004s sys 0.003s
I20251025 07:52:26.006312 1166 raft_consensus.cc:359] T b946855345bc473aadd00cc4db5b7cd8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.006763 1166 raft_consensus.cc:385] T b946855345bc473aadd00cc4db5b7cd8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:26.006897 1166 raft_consensus.cc:740] T b946855345bc473aadd00cc4db5b7cd8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:26.007417 1166 consensus_queue.cc:260] T b946855345bc473aadd00cc4db5b7cd8 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.009418 1167 tablet_bootstrap.cc:654] T 5680e1adaf3f4b7f8c5e05523dd0950a P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:26.016317 889 tablet_service.cc:1505] Processing CreateTablet for tablet fd9b4be139d347f4a6a60767029c342a (DEFAULT_TABLE table=default.Table_4145d1c9c33c4351a89976c5c4c18d75 [id=58190b46780d49eb9f429626ddce531c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.017984 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd9b4be139d347f4a6a60767029c342a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.020144 1020 tablet_service.cc:1505] Processing CreateTablet for tablet f3c87a83082b421cb4be546ac4ce7ad8 (DEFAULT_TABLE table=default.Table_ac1708f4dd0545f4b3411b218a61b6a5 [id=f3794fa081364666b944c412415bdba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.020447 1021 tablet_service.cc:1505] Processing CreateTablet for tablet a13bf4d7eb1440f7a28b85cb1bf99667 (DEFAULT_TABLE table=default.Table_e11142c0939b457d8dc181f50a9b8310 [id=3a3af35e11944a6f97fc0144cb9d08d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.020597 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3c87a83082b421cb4be546ac4ce7ad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.021018 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 2784bbfb97d5466f9ec856add7d0b29a (DEFAULT_TABLE table=default.Table_5aff13b5fcbc4899800760718fb3cc90 [id=ab3900039f5c491bbc1b96925deb1f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.021366 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2784bbfb97d5466f9ec856add7d0b29a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.021828 886 tablet_service.cc:1505] Processing CreateTablet for tablet 2784bbfb97d5466f9ec856add7d0b29a (DEFAULT_TABLE table=default.Table_5aff13b5fcbc4899800760718fb3cc90 [id=ab3900039f5c491bbc1b96925deb1f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.022418 1026 tablet_service.cc:1505] Processing CreateTablet for tablet ac0ea298bc114c56a8093e12def21ce1 (DEFAULT_TABLE table=default.Table_4228cd02e10a4b65ad0134b261130561 [id=c68c2e10aae74b6ea22df9c3715eac49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.022732 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac0ea298bc114c56a8093e12def21ce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.025194 884 tablet_service.cc:1505] Processing CreateTablet for tablet 21e52928580c4762b0ae7152490ef955 (DEFAULT_TABLE table=default.Table_e7d32e91b9c64ce1b3fcff3d806e959f [id=bab869e5001c4d999daa22083fca128a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.026012 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 21e52928580c4762b0ae7152490ef955 (DEFAULT_TABLE table=default.Table_e7d32e91b9c64ce1b3fcff3d806e959f [id=bab869e5001c4d999daa22083fca128a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.026464 887 tablet_service.cc:1505] Processing CreateTablet for tablet f3c87a83082b421cb4be546ac4ce7ad8 (DEFAULT_TABLE table=default.Table_ac1708f4dd0545f4b3411b218a61b6a5 [id=f3794fa081364666b944c412415bdba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.026894 890 tablet_service.cc:1505] Processing CreateTablet for tablet a13bf4d7eb1440f7a28b85cb1bf99667 (DEFAULT_TABLE table=default.Table_e11142c0939b457d8dc181f50a9b8310 [id=3a3af35e11944a6f97fc0144cb9d08d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.027112 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a13bf4d7eb1440f7a28b85cb1bf99667. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.027245 883 tablet_service.cc:1505] Processing CreateTablet for tablet ac0ea298bc114c56a8093e12def21ce1 (DEFAULT_TABLE table=default.Table_4228cd02e10a4b65ad0134b261130561 [id=c68c2e10aae74b6ea22df9c3715eac49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.027704 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2784bbfb97d5466f9ec856add7d0b29a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.017943 743 tablet_service.cc:1505] Processing CreateTablet for tablet 2784bbfb97d5466f9ec856add7d0b29a (DEFAULT_TABLE table=default.Table_5aff13b5fcbc4899800760718fb3cc90 [id=ab3900039f5c491bbc1b96925deb1f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.028354 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2784bbfb97d5466f9ec856add7d0b29a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.030268 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a13bf4d7eb1440f7a28b85cb1bf99667. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.018359 741 tablet_service.cc:1505] Processing CreateTablet for tablet ac0ea298bc114c56a8093e12def21ce1 (DEFAULT_TABLE table=default.Table_4228cd02e10a4b65ad0134b261130561 [id=c68c2e10aae74b6ea22df9c3715eac49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.034209 1022 tablet_service.cc:1505] Processing CreateTablet for tablet fd9b4be139d347f4a6a60767029c342a (DEFAULT_TABLE table=default.Table_4145d1c9c33c4351a89976c5c4c18d75 [id=58190b46780d49eb9f429626ddce531c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.034508 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac0ea298bc114c56a8093e12def21ce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.034657 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3c87a83082b421cb4be546ac4ce7ad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.034684 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21e52928580c4762b0ae7152490ef955. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.037801 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21e52928580c4762b0ae7152490ef955. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.018704 755 tablet_service.cc:1505] Processing CreateTablet for tablet 21e52928580c4762b0ae7152490ef955 (DEFAULT_TABLE table=default.Table_e7d32e91b9c64ce1b3fcff3d806e959f [id=bab869e5001c4d999daa22083fca128a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.038591 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21e52928580c4762b0ae7152490ef955. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.039439 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd9b4be139d347f4a6a60767029c342a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.019023 754 tablet_service.cc:1505] Processing CreateTablet for tablet fd9b4be139d347f4a6a60767029c342a (DEFAULT_TABLE table=default.Table_4145d1c9c33c4351a89976c5c4c18d75 [id=58190b46780d49eb9f429626ddce531c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.042490 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd9b4be139d347f4a6a60767029c342a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.048010 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac0ea298bc114c56a8093e12def21ce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.048650 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_072e0092adb64ef0ac6834dc18688ccf" } modify_external_catalogs: true
I20251025 07:52:26.048858 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_072e0092adb64ef0ac6834dc18688ccf" } modify_external_catalogs: true
I20251025 07:52:26.020658 750 tablet_service.cc:1505] Processing CreateTablet for tablet a13bf4d7eb1440f7a28b85cb1bf99667 (DEFAULT_TABLE table=default.Table_e11142c0939b457d8dc181f50a9b8310 [id=3a3af35e11944a6f97fc0144cb9d08d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.049947 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a13bf4d7eb1440f7a28b85cb1bf99667. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.050127 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_4228cd02e10a4b65ad0134b261130561" } modify_external_catalogs: true
I20251025 07:52:26.050334 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_4228cd02e10a4b65ad0134b261130561" } modify_external_catalogs: true
I20251025 07:52:26.058395 1169 tablet_bootstrap.cc:492] T 54cb4953ff264c13a0cc730b4c452ecc P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:26.058627 1169 ts_tablet_manager.cc:1403] T 54cb4953ff264c13a0cc730b4c452ecc P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.156s user 0.011s sys 0.016s
I20251025 07:52:26.059510 1169 raft_consensus.cc:359] T 54cb4953ff264c13a0cc730b4c452ecc P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.059779 1169 raft_consensus.cc:385] T 54cb4953ff264c13a0cc730b4c452ecc P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:26.059913 1169 raft_consensus.cc:740] T 54cb4953ff264c13a0cc730b4c452ecc P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:26.060247 1169 consensus_queue.cc:260] T 54cb4953ff264c13a0cc730b4c452ecc P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.063143 1169 ts_tablet_manager.cc:1434] T 54cb4953ff264c13a0cc730b4c452ecc P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251025 07:52:26.063728 1169 tablet_bootstrap.cc:492] T 14fd790ba7054c1fb09cdfb19b001987 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:26.064747 740 tablet_service.cc:1505] Processing CreateTablet for tablet f3c87a83082b421cb4be546ac4ce7ad8 (DEFAULT_TABLE table=default.Table_ac1708f4dd0545f4b3411b218a61b6a5 [id=f3794fa081364666b944c412415bdba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.065411 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3c87a83082b421cb4be546ac4ce7ad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.066025 1169 tablet_bootstrap.cc:654] T 14fd790ba7054c1fb09cdfb19b001987 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:26.066588 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 54cb4953ff264c13a0cc730b4c452ecc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:26.068493 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_e11142c0939b457d8dc181f50a9b8310" } new_table_name: "default.Table_6b9cad0ecb4a4d74941d9904282d676d" modify_external_catalogs: true
I20251025 07:52:26.069057 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_cde98da280d3479697eb4a5fe5405a95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:26.069701 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cde98da280d3479697eb4a5fe5405a95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.070472 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_b330f1a41da2427995e699545d5fc49b" } modify_external_catalogs: true
I20251025 07:52:26.070715 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_b330f1a41da2427995e699545d5fc49b" } modify_external_catalogs: true
I20251025 07:52:26.071605 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_892da226f7054e61b5ffc882b1d5c35a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:26.072142 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_892da226f7054e61b5ffc882b1d5c35a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.073211 1170 ts_tablet_manager.cc:1929] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:26.074263 1171 ts_tablet_manager.cc:1929] T 243825c6dd2d4461a01edec88f33f6ee P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:26.074597 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_234642f70fac40dd867675499ace87a3" } new_table_name: "default.Table_293596c64cca4768bc5869fd9924f856" modify_external_catalogs: true
I20251025 07:52:26.074852 1170 log.cc:1199] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/e84dbbd4a0bd4e4dbf1212fefbf00fd4
I20251025 07:52:26.075253 1170 ts_tablet_manager.cc:1950] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:26.076311 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet e84dbbd4a0bd4e4dbf1212fefbf00fd4 (table default.Table_826ad7d517d34a33a3759e733302ce66 [id=b6e173967bad47e6b306627221b80f94]) successfully deleted
I20251025 07:52:26.076769 1171 log.cc:1199] T 243825c6dd2d4461a01edec88f33f6ee P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/243825c6dd2d4461a01edec88f33f6ee
I20251025 07:52:26.077571 1171 ts_tablet_manager.cc:1950] T 243825c6dd2d4461a01edec88f33f6ee P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:26.082015 1166 ts_tablet_manager.cc:1434] T b946855345bc473aadd00cc4db5b7cd8 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251025 07:52:26.083766 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 243825c6dd2d4461a01edec88f33f6ee (table default.Table_325d6151a4134532ad8b68c9c0aa6f64 [id=68668d9bd5f34db8ab8f64ff1dd6d32f]) successfully deleted
I20251025 07:52:26.084398 1170 tablet_replica.cc:333] T 243825c6dd2d4461a01edec88f33f6ee P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:26.084821 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 08da9d2687704725aacd11cd92c6a9f9
I20251025 07:52:26.085124 1170 raft_consensus.cc:2243] T 243825c6dd2d4461a01edec88f33f6ee P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:26.085330 1170 raft_consensus.cc:2272] T 243825c6dd2d4461a01edec88f33f6ee P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:26.085640 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_4cf617a8064c4be99402936ea4cf514b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:26.085868 1170 ts_tablet_manager.cc:1916] T 243825c6dd2d4461a01edec88f33f6ee P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:26.086596 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 1b31177a062b467187d68ed2034f5690
I20251025 07:52:26.086759 1166 tablet_bootstrap.cc:492] T 70c3a36dda7b4919be20390f5797d09c P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:26.086998 525 catalog_manager.cc:6719] ReplaceTablet: tablet 1b31177a062b467187d68ed2034f5690 deleted and replaced by tablet b772025852794a49b072f37cccbd3291
I20251025 07:52:26.087417 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_92cbcd8230f249a8b2821f69a81b41f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:26.087917 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_92cbcd8230f249a8b2821f69a81b41f1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.088939 1166 tablet_bootstrap.cc:654] T 70c3a36dda7b4919be20390f5797d09c P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:26.089306 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 90764275825c4ffb8712cb883d37a551
I20251025 07:52:26.089759 521 catalog_manager.cc:6719] ReplaceTablet: tablet 90764275825c4ffb8712cb883d37a551 deleted and replaced by tablet 35ba9d23262248cda6fdf03c51be11a4
I20251025 07:52:26.089960 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 28) took 1111 ms. Trace:
I20251025 07:52:26.090124 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_40e767d985de4910930e0114543720fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:26.090097 1172 rpcz_store.cc:276] 1025 07:52:24.893592 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:24.893966 (+ 374us) service_pool.cc:225] Handling call
1025 07:52:25.869381 (+975415us) ts_tablet_manager.cc:1151] Deleting tablet d684da8134ce480693f470acdac176d4
1025 07:52:25.869392 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:25.902348 (+ 32956us) spinlock_profiling.cc:243] Waited 28.1 ms on lock 0x616000120454. stack: 00007ffaa94bd9f6 00007ffaa94bd7b0 00007ffaa84993ad 0000000000983f04 0000000000983ecd 000000000098389a 00007ffab5383100 00007ffab66d0ae3 00007ffab928d287 00007ffab92c02d4 00007ffab92a0d70 00007ffab92a0afd 0000000000a8867c 00007ffaa9503d7f 00007ffaa950704b 00007ffaa9506ec8
1025 07:52:25.923627 (+ 21279us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:25.937459 (+ 13832us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:26.005157 (+ 67698us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":64587648,"tablet-delete.queue_time_us":975152}
W20251025 07:52:26.090664 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40e767d985de4910930e0114543720fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.090853 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 29) took 1197 ms. Trace:
I20251025 07:52:26.091477 1172 rpcz_store.cc:276] 1025 07:52:24.893691 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:24.894268 (+ 577us) service_pool.cc:225] Handling call
1025 07:52:26.090678 (+1196410us) ts_tablet_manager.cc:1151] Deleting tablet a02b627401a64e1583f9e6123ef1be6f
1025 07:52:26.090691 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:26.090817 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1196210}
I20251025 07:52:26.091991 1172 tablet_replica.cc:333] T 243825c6dd2d4461a01edec88f33f6ee P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:26.092231 1172 raft_consensus.cc:2243] T 243825c6dd2d4461a01edec88f33f6ee P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:26.092528 1172 pending_rounds.cc:70] T 243825c6dd2d4461a01edec88f33f6ee P 3aea2869ff814cc7b142012faa274382: Trying to abort 1 pending ops.
I20251025 07:52:26.092609 1172 pending_rounds.cc:77] T 243825c6dd2d4461a01edec88f33f6ee P 3aea2869ff814cc7b142012faa274382: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214607341841920000 op_type: NO_OP noop_request { }
I20251025 07:52:26.092733 1172 raft_consensus.cc:2889] T 243825c6dd2d4461a01edec88f33f6ee P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 07:52:26.092860 1172 raft_consensus.cc:2272] T 243825c6dd2d4461a01edec88f33f6ee P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:26.093299 1171 tablet_replica.cc:333] T 14fd790ba7054c1fb09cdfb19b001987 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:26.093521 1171 raft_consensus.cc:2243] T 14fd790ba7054c1fb09cdfb19b001987 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:26.093703 1171 raft_consensus.cc:2272] T 14fd790ba7054c1fb09cdfb19b001987 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:26.093864 1172 ts_tablet_manager.cc:1916] T 243825c6dd2d4461a01edec88f33f6ee P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:26.095001 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet 1b31177a062b467187d68ed2034f5690 with delete_type TABLET_DATA_DELETED (replaced by tablet b772025852794a49b072f37cccbd3291) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:26.095120 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 08da9d2687704725aacd11cd92c6a9f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:26.095265 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 90764275825c4ffb8712cb883d37a551 with delete_type TABLET_DATA_DELETED (replaced by tablet 35ba9d23262248cda6fdf03c51be11a4) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:26.095373 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 90764275825c4ffb8712cb883d37a551 with delete_type TABLET_DATA_DELETED (replaced by tablet 35ba9d23262248cda6fdf03c51be11a4) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:26.095530 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 1b31177a062b467187d68ed2034f5690 with delete_type TABLET_DATA_DELETED (replaced by tablet b772025852794a49b072f37cccbd3291) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:26.095649 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 08da9d2687704725aacd11cd92c6a9f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:26.097250 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet d684da8134ce480693f470acdac176d4 (table default.Table_8a06efe638604f20bda568cdb8382a48 [id=4443da834fc34a01989b1d8cce759866]) successfully deleted
W20251025 07:52:26.097512 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet a02b627401a64e1583f9e6123ef1be6f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a02b627401a64e1583f9e6123ef1be6f already in progress: creating tablet
I20251025 07:52:26.097791 1171 ts_tablet_manager.cc:1916] T 14fd790ba7054c1fb09cdfb19b001987 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:26.097898 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet a231f3d31b204f7b82f018892ccc8435 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a231f3d31b204f7b82f018892ccc8435 already in progress: creating tablet
W20251025 07:52:26.098282 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 9cf88862e0934634b3dd172e6b40a0ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9cf88862e0934634b3dd172e6b40a0ba already in progress: creating tablet
W20251025 07:52:26.098497 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 576c51d9a22d45f6a3642a7296b5bf23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 576c51d9a22d45f6a3642a7296b5bf23 already in progress: creating tablet
W20251025 07:52:26.098664 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 2947502815d742e89cacb474b7fbb249 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2947502815d742e89cacb474b7fbb249 already in progress: creating tablet
W20251025 07:52:26.098842 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 785d0ce85b234826ba8021fbd182bf3c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 785d0ce85b234826ba8021fbd182bf3c already in progress: creating tablet
W20251025 07:52:26.099102 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4cf617a8064c4be99402936ea4cf514b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.105561 1167 tablet_bootstrap.cc:492] T 5680e1adaf3f4b7f8c5e05523dd0950a P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:26.105728 1167 ts_tablet_manager.cc:1403] T 5680e1adaf3f4b7f8c5e05523dd0950a P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.116s user 0.006s sys 0.001s
I20251025 07:52:26.106578 1167 raft_consensus.cc:359] T 5680e1adaf3f4b7f8c5e05523dd0950a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.106770 1167 raft_consensus.cc:385] T 5680e1adaf3f4b7f8c5e05523dd0950a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:26.106855 1167 raft_consensus.cc:740] T 5680e1adaf3f4b7f8c5e05523dd0950a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:26.107127 1167 consensus_queue.cc:260] T 5680e1adaf3f4b7f8c5e05523dd0950a P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.108649 1167 ts_tablet_manager.cc:1434] T 5680e1adaf3f4b7f8c5e05523dd0950a P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251025 07:52:26.108927 1167 tablet_bootstrap.cc:492] T 54cb4953ff264c13a0cc730b4c452ecc P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:26.111264 1167 tablet_bootstrap.cc:654] T 54cb4953ff264c13a0cc730b4c452ecc P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:26.116377 1176 raft_consensus.cc:493] T 5680e1adaf3f4b7f8c5e05523dd0950a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:26.116549 1176 raft_consensus.cc:515] T 5680e1adaf3f4b7f8c5e05523dd0950a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.097958 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 90764275825c4ffb8712cb883d37a551 with delete_type TABLET_DATA_DELETED (replaced by tablet 35ba9d23262248cda6fdf03c51be11a4) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:26.117260 1176 leader_election.cc:290] T 5680e1adaf3f4b7f8c5e05523dd0950a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:26.118119 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5680e1adaf3f4b7f8c5e05523dd0950a" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:26.118364 775 raft_consensus.cc:2468] T 5680e1adaf3f4b7f8c5e05523dd0950a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:26.118574 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5680e1adaf3f4b7f8c5e05523dd0950a" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:26.118788 914 raft_consensus.cc:2468] T 5680e1adaf3f4b7f8c5e05523dd0950a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:26.118917 982 leader_election.cc:304] T 5680e1adaf3f4b7f8c5e05523dd0950a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:26.119197 1176 raft_consensus.cc:2804] T 5680e1adaf3f4b7f8c5e05523dd0950a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:26.119302 1176 raft_consensus.cc:493] T 5680e1adaf3f4b7f8c5e05523dd0950a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:26.119385 1176 raft_consensus.cc:3060] T 5680e1adaf3f4b7f8c5e05523dd0950a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:26.104106 886 tablet_service.cc:1552] Processing DeleteTablet for tablet 08da9d2687704725aacd11cd92c6a9f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:26.121552 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet ac0ea298bc114c56a8093e12def21ce1
I20251025 07:52:26.104337 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 1b31177a062b467187d68ed2034f5690 with delete_type TABLET_DATA_DELETED (replaced by tablet b772025852794a49b072f37cccbd3291) from {username='slave'} at 127.0.0.1:56590
W20251025 07:52:26.122640 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a13bf4d7eb1440f7a28b85cb1bf99667 consensus state has no leader
I20251025 07:52:26.122766 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet ac0ea298bc114c56a8093e12def21ce1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47408
W20251025 07:52:26.124707 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 796413da731245459cc5dffacc3deb2f consensus state has no leader
I20251025 07:52:26.124821 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_4b3d7b3a731942189cdc214938a45820"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:26.125115 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_6921ba3d8feb4bd49739be0ba51dab61"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:26.125419 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b3d7b3a731942189cdc214938a45820 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:26.125505 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6921ba3d8feb4bd49739be0ba51dab61 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.125524 1176 raft_consensus.cc:515] T 5680e1adaf3f4b7f8c5e05523dd0950a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.126564 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5680e1adaf3f4b7f8c5e05523dd0950a" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:26.126789 775 raft_consensus.cc:3060] T 5680e1adaf3f4b7f8c5e05523dd0950a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:26.126981 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_cf686e1101314426a3ab24fd46a8974f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:26.127465 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf686e1101314426a3ab24fd46a8974f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.128677 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet b00153c97104470883f6089d71d23708
I20251025 07:52:26.130338 775 raft_consensus.cc:2468] T 5680e1adaf3f4b7f8c5e05523dd0950a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:26.130821 982 leader_election.cc:304] T 5680e1adaf3f4b7f8c5e05523dd0950a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:26.131055 1166 tablet_bootstrap.cc:492] T 70c3a36dda7b4919be20390f5797d09c P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:26.131237 1166 ts_tablet_manager.cc:1403] T 70c3a36dda7b4919be20390f5797d09c P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.001s
I20251025 07:52:26.132236 1176 leader_election.cc:290] T 5680e1adaf3f4b7f8c5e05523dd0950a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:26.132148 1166 raft_consensus.cc:359] T 70c3a36dda7b4919be20390f5797d09c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.132324 1166 raft_consensus.cc:385] T 70c3a36dda7b4919be20390f5797d09c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:26.132403 1166 raft_consensus.cc:740] T 70c3a36dda7b4919be20390f5797d09c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:26.132421 1176 raft_consensus.cc:2804] T 5680e1adaf3f4b7f8c5e05523dd0950a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:26.132525 1176 raft_consensus.cc:697] T 5680e1adaf3f4b7f8c5e05523dd0950a P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:26.132627 1166 consensus_queue.cc:260] T 70c3a36dda7b4919be20390f5797d09c P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.132774 1176 consensus_queue.cc:237] T 5680e1adaf3f4b7f8c5e05523dd0950a P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.134557 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_2253f6700c2f4a3a9847023ba98b80ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:26.135018 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2253f6700c2f4a3a9847023ba98b80ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.135718 1166 ts_tablet_manager.cc:1434] T 70c3a36dda7b4919be20390f5797d09c P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251025 07:52:26.138265 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet b00153c97104470883f6089d71d23708 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:26.138329 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5680e1adaf3f4b7f8c5e05523dd0950a" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:26.138505 914 raft_consensus.cc:3060] T 5680e1adaf3f4b7f8c5e05523dd0950a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:26.139096 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_59de6322049942169d98f535889113e3" } new_table_name: "default.Table_e49cd337b7254c18ba468dddbb318d4c" modify_external_catalogs: true
I20251025 07:52:26.141459 914 raft_consensus.cc:2468] T 5680e1adaf3f4b7f8c5e05523dd0950a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:26.143776 750 tablet_service.cc:1552] Processing DeleteTablet for tablet b00153c97104470883f6089d71d23708 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:26.144429 1169 tablet_bootstrap.cc:492] T 14fd790ba7054c1fb09cdfb19b001987 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:26.145081 740 tablet_service.cc:1552] Processing DeleteTablet for tablet ac0ea298bc114c56a8093e12def21ce1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:26.145390 894 tablet_service.cc:1552] Processing DeleteTablet for tablet b00153c97104470883f6089d71d23708 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:26.145407 1169 ts_tablet_manager.cc:1403] T 14fd790ba7054c1fb09cdfb19b001987 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.082s user 0.018s sys 0.016s
I20251025 07:52:26.145623 883 tablet_service.cc:1552] Processing DeleteTablet for tablet ac0ea298bc114c56a8093e12def21ce1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:26.148214 1166 tablet_bootstrap.cc:492] T 3be8063aee794d2abcfb1d6e24209c06 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:26.149845 1169 raft_consensus.cc:359] T 14fd790ba7054c1fb09cdfb19b001987 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.152742 1166 tablet_bootstrap.cc:654] T 3be8063aee794d2abcfb1d6e24209c06 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:26.153832 1169 raft_consensus.cc:385] T 14fd790ba7054c1fb09cdfb19b001987 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:26.154042 1169 raft_consensus.cc:740] T 14fd790ba7054c1fb09cdfb19b001987 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:26.154424 1169 consensus_queue.cc:260] T 14fd790ba7054c1fb09cdfb19b001987 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.155525 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_c361b0312923449da58cd2d6a012be15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:26.156075 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c361b0312923449da58cd2d6a012be15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.157416 521 master_service.cc:954] ReplaceTablet: received request to replace tablet 5caac9c943bf42a5b2ae522d87fdfe8e from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:26.157778 1167 tablet_bootstrap.cc:492] T 54cb4953ff264c13a0cc730b4c452ecc P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:26.158032 1167 ts_tablet_manager.cc:1403] T 54cb4953ff264c13a0cc730b4c452ecc P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.003s
I20251025 07:52:26.158036 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_819aedd5270f4d6cbaaaee7c2ff5f676" } modify_external_catalogs: true
I20251025 07:52:26.158241 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_819aedd5270f4d6cbaaaee7c2ff5f676" } modify_external_catalogs: true
I20251025 07:52:26.159078 1167 raft_consensus.cc:359] T 54cb4953ff264c13a0cc730b4c452ecc P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.159297 1167 raft_consensus.cc:385] T 54cb4953ff264c13a0cc730b4c452ecc P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:26.159384 1167 raft_consensus.cc:740] T 54cb4953ff264c13a0cc730b4c452ecc P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:26.160233 1167 consensus_queue.cc:260] T 54cb4953ff264c13a0cc730b4c452ecc P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.163370 1167 ts_tablet_manager.cc:1434] T 54cb4953ff264c13a0cc730b4c452ecc P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251025 07:52:26.163733 1167 tablet_bootstrap.cc:492] T d684da8134ce480693f470acdac176d4 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:26.165575 1167 tablet_bootstrap.cc:654] T d684da8134ce480693f470acdac176d4 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:26.167805 1166 tablet_bootstrap.cc:492] T 3be8063aee794d2abcfb1d6e24209c06 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:26.168009 1166 ts_tablet_manager.cc:1403] T 3be8063aee794d2abcfb1d6e24209c06 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.001s
I20251025 07:52:26.169299 1166 raft_consensus.cc:359] T 3be8063aee794d2abcfb1d6e24209c06 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.169490 1166 raft_consensus.cc:385] T 3be8063aee794d2abcfb1d6e24209c06 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:26.169564 1166 raft_consensus.cc:740] T 3be8063aee794d2abcfb1d6e24209c06 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:26.170410 1166 consensus_queue.cc:260] T 3be8063aee794d2abcfb1d6e24209c06 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.171430 518 master_service.cc:954] ReplaceTablet: received request to replace tablet d0fd00baa02d48278afaa5b8c19182f0 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:26.171957 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_e43f4b13c0ba4d72b1c7292170cb7931" } new_table_name: "default.Table_3f53ffa013f94faf941d634d22a37ea4" modify_external_catalogs: true
I20251025 07:52:26.173451 1166 ts_tablet_manager.cc:1434] T 3be8063aee794d2abcfb1d6e24209c06 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20251025 07:52:26.174454 740 tablet_service.cc:1505] Processing CreateTablet for tablet fab67b56eaa44b54886c17181d8c52b2 (DEFAULT_TABLE table=default.Table_6ca844d1e78c4f19939985aec32a8afd [id=1e933863244e4792b4c1b4d6d5c674b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.174948 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fab67b56eaa44b54886c17181d8c52b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.178601 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_cde98da280d3479697eb4a5fe5405a95" } new_table_name: "default.Table_46d719baf05a4de09e6bd95dc90e4d0f" modify_external_catalogs: true
W20251025 07:52:26.180002 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8e6d76b1a97945cfb6c3055e1b622f8a consensus state has no leader
I20251025 07:52:26.182245 1172 ts_tablet_manager.cc:1929] T 243825c6dd2d4461a01edec88f33f6ee P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:26.182415 1172 log.cc:1199] T 243825c6dd2d4461a01edec88f33f6ee P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/243825c6dd2d4461a01edec88f33f6ee
I20251025 07:52:26.183136 1172 ts_tablet_manager.cc:1950] T 243825c6dd2d4461a01edec88f33f6ee P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:26.183132 1021 tablet_service.cc:1505] Processing CreateTablet for tablet fab67b56eaa44b54886c17181d8c52b2 (DEFAULT_TABLE table=default.Table_6ca844d1e78c4f19939985aec32a8afd [id=1e933863244e4792b4c1b4d6d5c674b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.183570 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fab67b56eaa44b54886c17181d8c52b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.183980 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 30) took 1290 ms. Trace:
I20251025 07:52:26.186028 883 tablet_service.cc:1505] Processing CreateTablet for tablet fab67b56eaa44b54886c17181d8c52b2 (DEFAULT_TABLE table=default.Table_6ca844d1e78c4f19939985aec32a8afd [id=1e933863244e4792b4c1b4d6d5c674b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.189646 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fab67b56eaa44b54886c17181d8c52b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.191257 1170 ts_tablet_manager.cc:1929] T 243825c6dd2d4461a01edec88f33f6ee P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:26.191399 1170 log.cc:1199] T 243825c6dd2d4461a01edec88f33f6ee P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/243825c6dd2d4461a01edec88f33f6ee
I20251025 07:52:26.191768 1170 ts_tablet_manager.cc:1950] T 243825c6dd2d4461a01edec88f33f6ee P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:26.192656 1170 tablet_replica.cc:333] T d684da8134ce480693f470acdac176d4 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:26.192768 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 243825c6dd2d4461a01edec88f33f6ee (table default.Table_325d6151a4134532ad8b68c9c0aa6f64 [id=68668d9bd5f34db8ab8f64ff1dd6d32f]) successfully deleted
I20251025 07:52:26.192845 1170 raft_consensus.cc:2243] T d684da8134ce480693f470acdac176d4 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:26.193017 1170 raft_consensus.cc:2272] T d684da8134ce480693f470acdac176d4 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:26.194497 1172 rpcz_store.cc:276] 1025 07:52:24.893798 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:24.894491 (+ 693us) service_pool.cc:225] Handling call
1025 07:52:26.091905 (+1197414us) ts_tablet_manager.cc:1151] Deleting tablet 243825c6dd2d4461a01edec88f33f6ee
1025 07:52:26.091929 (+ 24us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:26.123620 (+ 31691us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:26.182069 (+ 58449us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:26.183941 (+ 1872us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1197269}
I20251025 07:52:26.195487 1172 tablet_replica.cc:333] T 54cb4953ff264c13a0cc730b4c452ecc P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:26.195674 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 243825c6dd2d4461a01edec88f33f6ee (table default.Table_325d6151a4134532ad8b68c9c0aa6f64 [id=68668d9bd5f34db8ab8f64ff1dd6d32f]) successfully deleted
I20251025 07:52:26.195842 1172 raft_consensus.cc:2243] T 54cb4953ff264c13a0cc730b4c452ecc P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:26.196025 1172 raft_consensus.cc:2272] T 54cb4953ff264c13a0cc730b4c452ecc P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:26.196878 1172 ts_tablet_manager.cc:1916] T 54cb4953ff264c13a0cc730b4c452ecc P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:26.200563 1169 ts_tablet_manager.cc:1434] T 14fd790ba7054c1fb09cdfb19b001987 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.054s user 0.001s sys 0.005s
W20251025 07:52:26.203266 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9bf486d82b984d2c89ee32e84a85894f consensus state has no leader
I20251025 07:52:26.203846 1169 tablet_bootstrap.cc:492] T b946855345bc473aadd00cc4db5b7cd8 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
W20251025 07:52:26.204010 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 27651312cc2e42a8bfcd245f5d301679 has no consensus state
I20251025 07:52:26.204356 523 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 35ba9d23262248cda6fdf03c51be11a4
I20251025 07:52:26.205623 523 master_service.cc:954] ReplaceTablet: received request to replace tablet f3c87a83082b421cb4be546ac4ce7ad8 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:26.205717 1171 ts_tablet_manager.cc:1929] T 14fd790ba7054c1fb09cdfb19b001987 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:26.205860 1171 log.cc:1199] T 14fd790ba7054c1fb09cdfb19b001987 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/14fd790ba7054c1fb09cdfb19b001987
I20251025 07:52:26.206285 1171 ts_tablet_manager.cc:1950] T 14fd790ba7054c1fb09cdfb19b001987 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:26.206696 1169 tablet_bootstrap.cc:654] T b946855345bc473aadd00cc4db5b7cd8 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:26.207408 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 14fd790ba7054c1fb09cdfb19b001987 (table default.Table_43eec86aa9fc405fa761a8eaa89d2db0 [id=67beaeffaf2243ef86f6c45084781432]) successfully deleted
I20251025 07:52:26.207628 1171 tablet_replica.cc:333] T 89bfb7e2f051454aae0af5755a79ca58 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:26.207834 1171 raft_consensus.cc:2243] T 89bfb7e2f051454aae0af5755a79ca58 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:26.208061 1171 raft_consensus.cc:2272] T 89bfb7e2f051454aae0af5755a79ca58 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:26.208282 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 33c608beb4ef4a859a135b11e07c81a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33c608beb4ef4a859a135b11e07c81a4 already in progress: creating tablet
I20251025 07:52:26.208565 1171 ts_tablet_manager.cc:1916] T 89bfb7e2f051454aae0af5755a79ca58 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:26.208544 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet cf2cab1552bd4e7394797c93b149f450 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf2cab1552bd4e7394797c93b149f450 already in progress: creating tablet
I20251025 07:52:26.208323 522 catalog_manager.cc:5649] T 5680e1adaf3f4b7f8c5e05523dd0950a P db42be7ad3c741a0aa427f27e245993b reported cstate change: term changed from 0 to 1, leader changed from <none> to db42be7ad3c741a0aa427f27e245993b (127.0.104.3). New cstate: current_term: 1 leader_uuid: "db42be7ad3c741a0aa427f27e245993b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: UNKNOWN } } }
I20251025 07:52:26.209249 522 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 5680e1adaf3f4b7f8c5e05523dd0950a (table default.Table_31d7f6f73d6e4123b1ff1df86ff62205 [id=cb037f61ee0b4b6cb8fb613263a278fa]). Expected version 1 got 0
I20251025 07:52:26.210412 1166 tablet_bootstrap.cc:492] T a02b627401a64e1583f9e6123ef1be6f P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:26.216017 1171 ts_tablet_manager.cc:1929] T 89bfb7e2f051454aae0af5755a79ca58 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:26.216146 1171 log.cc:1199] T 89bfb7e2f051454aae0af5755a79ca58 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/89bfb7e2f051454aae0af5755a79ca58
I20251025 07:52:26.216504 1171 ts_tablet_manager.cc:1950] T 89bfb7e2f051454aae0af5755a79ca58 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:26.216521 1166 tablet_bootstrap.cc:654] T a02b627401a64e1583f9e6123ef1be6f P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:26.217410 1171 tablet_replica.cc:333] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:26.217607 1171 raft_consensus.cc:2243] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:26.217723 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 89bfb7e2f051454aae0af5755a79ca58 (table default.Table_5f94a1aa9fac44ae8a90d3554215d598 [id=d82b3e1317434cf08a0d51e8a9427224]) successfully deleted
I20251025 07:52:26.217777 1171 raft_consensus.cc:2272] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:26.218348 1171 ts_tablet_manager.cc:1916] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:26.218515 1021 tablet_service.cc:1505] Processing CreateTablet for tablet 467324e2432f4caca148951329caa292 (DEFAULT_TABLE table=default.Table_4cf617a8064c4be99402936ea4cf514b [id=11ce21ee2d434ce38aadc6152c02e2bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.218842 1167 tablet_bootstrap.cc:492] T d684da8134ce480693f470acdac176d4 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:26.218995 1167 ts_tablet_manager.cc:1403] T d684da8134ce480693f470acdac176d4 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.055s user 0.002s sys 0.001s
I20251025 07:52:26.219054 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 467324e2432f4caca148951329caa292. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.219086 894 tablet_service.cc:1505] Processing CreateTablet for tablet 467324e2432f4caca148951329caa292 (DEFAULT_TABLE table=default.Table_4cf617a8064c4be99402936ea4cf514b [id=11ce21ee2d434ce38aadc6152c02e2bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.219463 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 467324e2432f4caca148951329caa292. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.219835 1167 raft_consensus.cc:359] T d684da8134ce480693f470acdac176d4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.220026 1167 raft_consensus.cc:385] T d684da8134ce480693f470acdac176d4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:26.220119 1167 raft_consensus.cc:740] T d684da8134ce480693f470acdac176d4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:26.220379 1167 consensus_queue.cc:260] T d684da8134ce480693f470acdac176d4 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.221005 1167 ts_tablet_manager.cc:1434] T d684da8134ce480693f470acdac176d4 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20251025 07:52:26.221285 1167 tablet_bootstrap.cc:492] T b946855345bc473aadd00cc4db5b7cd8 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:26.222656 740 tablet_service.cc:1505] Processing CreateTablet for tablet 467324e2432f4caca148951329caa292 (DEFAULT_TABLE table=default.Table_4cf617a8064c4be99402936ea4cf514b [id=11ce21ee2d434ce38aadc6152c02e2bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.223038 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 467324e2432f4caca148951329caa292. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.225124 1218 raft_consensus.cc:493] T 70c3a36dda7b4919be20390f5797d09c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:26.225265 1218 raft_consensus.cc:515] T 70c3a36dda7b4919be20390f5797d09c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.226009 1218 leader_election.cc:290] T 70c3a36dda7b4919be20390f5797d09c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:26.226676 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "70c3a36dda7b4919be20390f5797d09c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:26.227223 850 leader_election.cc:343] T 70c3a36dda7b4919be20390f5797d09c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:26.227807 1171 ts_tablet_manager.cc:1929] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:26.227742 750 tablet_service.cc:1505] Processing CreateTablet for tablet 4037dee385d04e85bb78c8e77cb00379 (DEFAULT_TABLE table=default.Table_892da226f7054e61b5ffc882b1d5c35a [id=c13e40cbb3624b3997e907d4f2c1f706]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.227921 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "70c3a36dda7b4919be20390f5797d09c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:26.228152 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4037dee385d04e85bb78c8e77cb00379. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.228315 1170 ts_tablet_manager.cc:1916] T d684da8134ce480693f470acdac176d4 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:26.228621 847 leader_election.cc:343] T 70c3a36dda7b4919be20390f5797d09c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:26.228746 847 leader_election.cc:304] T 70c3a36dda7b4919be20390f5797d09c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:26.229121 1218 raft_consensus.cc:2749] T 70c3a36dda7b4919be20390f5797d09c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:26.230441 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 4037dee385d04e85bb78c8e77cb00379 (DEFAULT_TABLE table=default.Table_892da226f7054e61b5ffc882b1d5c35a [id=c13e40cbb3624b3997e907d4f2c1f706]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.230810 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4037dee385d04e85bb78c8e77cb00379. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.237133 519 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 5680e1adaf3f4b7f8c5e05523dd0950a (table default.Table_31d7f6f73d6e4123b1ff1df86ff62205 [id=cb037f61ee0b4b6cb8fb613263a278fa]). Expected version 1 got 0
I20251025 07:52:26.241250 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 94408a3a90224645ba3cacccf5de33f1
I20251025 07:52:26.242231 522 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 5680e1adaf3f4b7f8c5e05523dd0950a (table default.Table_31d7f6f73d6e4123b1ff1df86ff62205 [id=cb037f61ee0b4b6cb8fb613263a278fa]). Expected version 1 got 0
I20251025 07:52:26.242297 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 5caac9c943bf42a5b2ae522d87fdfe8e
I20251025 07:52:26.242324 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 14fd790ba7054c1fb09cdfb19b001987 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:26.242699 521 catalog_manager.cc:6719] ReplaceTablet: tablet 5caac9c943bf42a5b2ae522d87fdfe8e deleted and replaced by tablet 4f25c9caa05b4da4bd56a8a21fde423c
I20251025 07:52:26.236652 1171 log.cc:1199] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/e84dbbd4a0bd4e4dbf1212fefbf00fd4
I20251025 07:52:26.243647 1169 tablet_bootstrap.cc:492] T b946855345bc473aadd00cc4db5b7cd8 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:26.243753 1171 ts_tablet_manager.cc:1950] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:26.243829 1169 ts_tablet_manager.cc:1403] T b946855345bc473aadd00cc4db5b7cd8 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.003s
I20251025 07:52:26.244241 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet f3c87a83082b421cb4be546ac4ce7ad8
I20251025 07:52:26.244637 523 catalog_manager.cc:6719] ReplaceTablet: tablet f3c87a83082b421cb4be546ac4ce7ad8 deleted and replaced by tablet b17d2a9ad6be49d29f2413666fba7e69
I20251025 07:52:26.244701 1169 raft_consensus.cc:359] T b946855345bc473aadd00cc4db5b7cd8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.244966 1169 raft_consensus.cc:385] T b946855345bc473aadd00cc4db5b7cd8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:26.245066 1169 raft_consensus.cc:740] T b946855345bc473aadd00cc4db5b7cd8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:26.245345 1169 consensus_queue.cc:260] T b946855345bc473aadd00cc4db5b7cd8 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.245536 886 tablet_service.cc:1552] Processing DeleteTablet for tablet f3c87a83082b421cb4be546ac4ce7ad8 with delete_type TABLET_DATA_DELETED (replaced by tablet b17d2a9ad6be49d29f2413666fba7e69) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:26.245826 1169 ts_tablet_manager.cc:1434] T b946855345bc473aadd00cc4db5b7cd8 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:26.246172 740 tablet_service.cc:1552] Processing DeleteTablet for tablet 94408a3a90224645ba3cacccf5de33f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:26.246229 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet e84dbbd4a0bd4e4dbf1212fefbf00fd4 (table default.Table_826ad7d517d34a33a3759e733302ce66 [id=b6e173967bad47e6b306627221b80f94]) successfully deleted
I20251025 07:52:26.236817 1167 tablet_bootstrap.cc:654] T b946855345bc473aadd00cc4db5b7cd8 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:26.246500 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet d0fd00baa02d48278afaa5b8c19182f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0fd00baa02d48278afaa5b8c19182f0 already in progress: creating tablet
I20251025 07:52:26.246662 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 54cb4953ff264c13a0cc730b4c452ecc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:48830
W20251025 07:52:26.246703 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet f51ec427dd9349a48eb221db04cf242b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f51ec427dd9349a48eb221db04cf242b already in progress: creating tablet
I20251025 07:52:26.246917 754 tablet_service.cc:1552] Processing DeleteTablet for tablet f3c87a83082b421cb4be546ac4ce7ad8 with delete_type TABLET_DATA_DELETED (replaced by tablet b17d2a9ad6be49d29f2413666fba7e69) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:26.247301 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 5caac9c943bf42a5b2ae522d87fdfe8e with delete_type TABLET_DATA_DELETED (replaced by tablet 4f25c9caa05b4da4bd56a8a21fde423c) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:26.248298 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 5caac9c943bf42a5b2ae522d87fdfe8e with delete_type TABLET_DATA_DELETED (replaced by tablet 4f25c9caa05b4da4bd56a8a21fde423c) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:26.248541 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 94408a3a90224645ba3cacccf5de33f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:26.248629 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet f3c87a83082b421cb4be546ac4ce7ad8 with delete_type TABLET_DATA_DELETED (replaced by tablet b17d2a9ad6be49d29f2413666fba7e69) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:26.248577 886 tablet_service.cc:1505] Processing CreateTablet for tablet 4037dee385d04e85bb78c8e77cb00379 (DEFAULT_TABLE table=default.Table_892da226f7054e61b5ffc882b1d5c35a [id=c13e40cbb3624b3997e907d4f2c1f706]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.248955 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 5caac9c943bf42a5b2ae522d87fdfe8e with delete_type TABLET_DATA_DELETED (replaced by tablet 4f25c9caa05b4da4bd56a8a21fde423c) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:26.249040 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4037dee385d04e85bb78c8e77cb00379. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.249531 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 94408a3a90224645ba3cacccf5de33f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:26.249738 1169 tablet_bootstrap.cc:492] T 70c3a36dda7b4919be20390f5797d09c P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:26.251875 1169 tablet_bootstrap.cc:654] T 70c3a36dda7b4919be20390f5797d09c P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:26.252491 1170 ts_tablet_manager.cc:1929] T d684da8134ce480693f470acdac176d4 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:26.252635 1170 log.cc:1199] T d684da8134ce480693f470acdac176d4 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/d684da8134ce480693f470acdac176d4
I20251025 07:52:26.252789 1172 ts_tablet_manager.cc:1929] T 54cb4953ff264c13a0cc730b4c452ecc P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:26.252998 1170 ts_tablet_manager.cc:1950] T d684da8134ce480693f470acdac176d4 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:26.252867 740 tablet_service.cc:1552] Processing DeleteTablet for tablet d684da8134ce480693f470acdac176d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:48830
W20251025 07:52:26.255683 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 47284f96928749e6a5201103a8ee5f9f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47284f96928749e6a5201103a8ee5f9f already in progress: creating tablet
I20251025 07:52:26.256016 1251 raft_consensus.cc:493] T b946855345bc473aadd00cc4db5b7cd8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:26.256186 1251 raft_consensus.cc:515] T b946855345bc473aadd00cc4db5b7cd8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.257083 1251 leader_election.cc:290] T b946855345bc473aadd00cc4db5b7cd8 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:26.257570 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet d684da8134ce480693f470acdac176d4 (table default.Table_8a06efe638604f20bda568cdb8382a48 [id=4443da834fc34a01989b1d8cce759866]) successfully deleted
I20251025 07:52:26.257624 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b946855345bc473aadd00cc4db5b7cd8" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
W20251025 07:52:26.258090 982 leader_election.cc:343] T b946855345bc473aadd00cc4db5b7cd8 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:26.258227 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet f51ec427dd9349a48eb221db04cf242b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f51ec427dd9349a48eb221db04cf242b already in progress: creating tablet
W20251025 07:52:26.258387 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 79bc2678df8042e3994b0471afa2c304 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79bc2678df8042e3994b0471afa2c304 already in progress: creating tablet
W20251025 07:52:26.258755 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet d0fd00baa02d48278afaa5b8c19182f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0fd00baa02d48278afaa5b8c19182f0 already in progress: creating tablet
W20251025 07:52:26.259331 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 47284f96928749e6a5201103a8ee5f9f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47284f96928749e6a5201103a8ee5f9f already in progress: creating tablet
W20251025 07:52:26.259943 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 567a1f5c216a4a519249da66a2a03ea4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 567a1f5c216a4a519249da66a2a03ea4 already in progress: creating tablet
I20251025 07:52:26.260085 1167 tablet_bootstrap.cc:492] T b946855345bc473aadd00cc4db5b7cd8 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:26.260283 1167 ts_tablet_manager.cc:1403] T b946855345bc473aadd00cc4db5b7cd8 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.000s
W20251025 07:52:26.260588 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 79bc2678df8042e3994b0471afa2c304 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79bc2678df8042e3994b0471afa2c304 already in progress: creating tablet
W20251025 07:52:26.261157 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet dd898450f2604987b08639cf87d0e402 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dd898450f2604987b08639cf87d0e402 already in progress: creating tablet
I20251025 07:52:26.258090 1166 tablet_bootstrap.cc:492] T a02b627401a64e1583f9e6123ef1be6f P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:26.258044 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b946855345bc473aadd00cc4db5b7cd8" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:26.261577 1170 tablet_replica.cc:333] T b946855345bc473aadd00cc4db5b7cd8 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:26.261571 1166 ts_tablet_manager.cc:1403] T a02b627401a64e1583f9e6123ef1be6f P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.051s user 0.002s sys 0.002s
I20251025 07:52:26.261572 914 raft_consensus.cc:2468] T b946855345bc473aadd00cc4db5b7cd8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:26.261788 1167 raft_consensus.cc:359] T b946855345bc473aadd00cc4db5b7cd8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.261996 1167 raft_consensus.cc:385] T b946855345bc473aadd00cc4db5b7cd8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:26.262068 1167 raft_consensus.cc:740] T b946855345bc473aadd00cc4db5b7cd8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:26.261989 983 leader_election.cc:304] T b946855345bc473aadd00cc4db5b7cd8 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b
I20251025 07:52:26.262409 1247 raft_consensus.cc:493] T 54cb4953ff264c13a0cc730b4c452ecc P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:26.261868 1170 raft_consensus.cc:2243] T b946855345bc473aadd00cc4db5b7cd8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:26.262542 1247 raft_consensus.cc:515] T 54cb4953ff264c13a0cc730b4c452ecc P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.262758 1170 raft_consensus.cc:2272] T b946855345bc473aadd00cc4db5b7cd8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:26.262722 1166 raft_consensus.cc:359] T a02b627401a64e1583f9e6123ef1be6f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.262877 1166 raft_consensus.cc:385] T a02b627401a64e1583f9e6123ef1be6f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:26.262984 1166 raft_consensus.cc:740] T a02b627401a64e1583f9e6123ef1be6f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:26.263237 1166 consensus_queue.cc:260] T a02b627401a64e1583f9e6123ef1be6f P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.263435 1247 leader_election.cc:290] T 54cb4953ff264c13a0cc730b4c452ecc P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:26.264249 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54cb4953ff264c13a0cc730b4c452ecc" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:26.265025 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 35ba9d23262248cda6fdf03c51be11a4 (DEFAULT_TABLE table=default.Table_072e0092adb64ef0ac6834dc18688ccf [id=1d4cbe4a4f38445297ceeebb8ffdaa8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.265497 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35ba9d23262248cda6fdf03c51be11a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.266660 1166 ts_tablet_manager.cc:1434] T a02b627401a64e1583f9e6123ef1be6f P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251025 07:52:26.268296 889 tablet_service.cc:1505] Processing CreateTablet for tablet 02e4c4d6ffb34cecaf17beb5ba197e81 (DEFAULT_TABLE table=default.Table_40e767d985de4910930e0114543720fd [id=bf3a3f8dae184d999f2159dfcc7803b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.268402 886 tablet_service.cc:1505] Processing CreateTablet for tablet 35ba9d23262248cda6fdf03c51be11a4 (DEFAULT_TABLE table=default.Table_072e0092adb64ef0ac6834dc18688ccf [id=1d4cbe4a4f38445297ceeebb8ffdaa8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.268673 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02e4c4d6ffb34cecaf17beb5ba197e81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.269750 884 tablet_service.cc:1505] Processing CreateTablet for tablet b772025852794a49b072f37cccbd3291 (DEFAULT_TABLE table=default.Table_66a70e784e9b49cdb516f657733f2a38 [id=0827603890c8466082bc6c306c9c66af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.270102 894 tablet_service.cc:1552] Processing DeleteTablet for tablet a02b627401a64e1583f9e6123ef1be6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:26.270282 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54cb4953ff264c13a0cc730b4c452ecc" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:26.270530 775 raft_consensus.cc:2468] T 54cb4953ff264c13a0cc730b4c452ecc P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:26.270560 887 tablet_service.cc:1505] Processing CreateTablet for tablet 7fbae8be73bf4e18b5a5c4ec4cdebc12 (DEFAULT_TABLE table=default.Table_6921ba3d8feb4bd49739be0ba51dab61 [id=28bbaad474164a8c8c97a76bb8081791]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.270973 883 tablet_service.cc:1505] Processing CreateTablet for tablet d37758f606344127b3db74c87babe69c (DEFAULT_TABLE table=default.Table_92cbcd8230f249a8b2821f69a81b41f1 [id=08e53121e86d405dbbd02a4605695fec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.271523 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35ba9d23262248cda6fdf03c51be11a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.271670 1171 tablet_replica.cc:333] T 54cb4953ff264c13a0cc730b4c452ecc P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:26.271893 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fbae8be73bf4e18b5a5c4ec4cdebc12. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.271906 1171 raft_consensus.cc:2243] T 54cb4953ff264c13a0cc730b4c452ecc P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:26.272107 1171 raft_consensus.cc:2272] T 54cb4953ff264c13a0cc730b4c452ecc P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:26.272972 1171 ts_tablet_manager.cc:1916] T 54cb4953ff264c13a0cc730b4c452ecc P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:26.273234 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 567a1f5c216a4a519249da66a2a03ea4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 567a1f5c216a4a519249da66a2a03ea4 already in progress: creating tablet
W20251025 07:52:26.273516 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet dd898450f2604987b08639cf87d0e402 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dd898450f2604987b08639cf87d0e402 already in progress: creating tablet
W20251025 07:52:26.273703 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet b946855345bc473aadd00cc4db5b7cd8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b946855345bc473aadd00cc4db5b7cd8 already in progress: creating tablet
W20251025 07:52:26.273939 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 1b31177a062b467187d68ed2034f5690 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b31177a062b467187d68ed2034f5690 already in progress: creating tablet
W20251025 07:52:26.274154 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 08da9d2687704725aacd11cd92c6a9f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08da9d2687704725aacd11cd92c6a9f9 already in progress: creating tablet
W20251025 07:52:26.271801 983 leader_election.cc:343] T 54cb4953ff264c13a0cc730b4c452ecc P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: Tablet not RUNNING: STOPPED
I20251025 07:52:26.274268 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b772025852794a49b072f37cccbd3291. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:26.274333 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet b00153c97104470883f6089d71d23708 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b00153c97104470883f6089d71d23708 already in progress: creating tablet
I20251025 07:52:26.272142 1021 tablet_service.cc:1505] Processing CreateTablet for tablet d37758f606344127b3db74c87babe69c (DEFAULT_TABLE table=default.Table_92cbcd8230f249a8b2821f69a81b41f1 [id=08e53121e86d405dbbd02a4605695fec]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:26.274513 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet ac0ea298bc114c56a8093e12def21ce1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac0ea298bc114c56a8093e12def21ce1 already in progress: creating tablet
I20251025 07:52:26.274682 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d37758f606344127b3db74c87babe69c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:26.274696 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 94408a3a90224645ba3cacccf5de33f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94408a3a90224645ba3cacccf5de33f1 already in progress: creating tablet
I20251025 07:52:26.275203 750 tablet_service.cc:1505] Processing CreateTablet for tablet d37758f606344127b3db74c87babe69c (DEFAULT_TABLE table=default.Table_92cbcd8230f249a8b2821f69a81b41f1 [id=08e53121e86d405dbbd02a4605695fec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.275586 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d37758f606344127b3db74c87babe69c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.272406 1020 tablet_service.cc:1505] Processing CreateTablet for tablet 02e4c4d6ffb34cecaf17beb5ba197e81 (DEFAULT_TABLE table=default.Table_40e767d985de4910930e0114543720fd [id=bf3a3f8dae184d999f2159dfcc7803b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.278470 755 tablet_service.cc:1505] Processing CreateTablet for tablet b772025852794a49b072f37cccbd3291 (DEFAULT_TABLE table=default.Table_66a70e784e9b49cdb516f657733f2a38 [id=0827603890c8466082bc6c306c9c66af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.278564 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02e4c4d6ffb34cecaf17beb5ba197e81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.278833 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b772025852794a49b072f37cccbd3291. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.281371 1171 ts_tablet_manager.cc:1929] T 54cb4953ff264c13a0cc730b4c452ecc P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:26.281494 1171 log.cc:1199] T 54cb4953ff264c13a0cc730b4c452ecc P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/54cb4953ff264c13a0cc730b4c452ecc
I20251025 07:52:26.281826 1171 ts_tablet_manager.cc:1950] T 54cb4953ff264c13a0cc730b4c452ecc P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:26.272611 1026 tablet_service.cc:1505] Processing CreateTablet for tablet b772025852794a49b072f37cccbd3291 (DEFAULT_TABLE table=default.Table_66a70e784e9b49cdb516f657733f2a38 [id=0827603890c8466082bc6c306c9c66af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.282408 743 tablet_service.cc:1505] Processing CreateTablet for tablet 7fbae8be73bf4e18b5a5c4ec4cdebc12 (DEFAULT_TABLE table=default.Table_6921ba3d8feb4bd49739be0ba51dab61 [id=28bbaad474164a8c8c97a76bb8081791]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.282509 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b772025852794a49b072f37cccbd3291. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.282789 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fbae8be73bf4e18b5a5c4ec4cdebc12. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.282902 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 54cb4953ff264c13a0cc730b4c452ecc (table default.Table_ad812bcf86a04b0b89a3f0bc961062a9 [id=fe2800366dea4cdd85696d44df21a4c5]) successfully deleted
I20251025 07:52:26.284121 894 tablet_service.cc:1505] Processing CreateTablet for tablet 8c76d1b509f04ab1a10b83cff675c11e (DEFAULT_TABLE table=default.Table_cf686e1101314426a3ab24fd46a8974f [id=13fbed1c733c42e2b45e4ca69580b06d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.284533 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c76d1b509f04ab1a10b83cff675c11e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.272799 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 7fbae8be73bf4e18b5a5c4ec4cdebc12 (DEFAULT_TABLE table=default.Table_6921ba3d8feb4bd49739be0ba51dab61 [id=28bbaad474164a8c8c97a76bb8081791]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.286293 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d37758f606344127b3db74c87babe69c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.286449 739 tablet_service.cc:1505] Processing CreateTablet for tablet 8c76d1b509f04ab1a10b83cff675c11e (DEFAULT_TABLE table=default.Table_cf686e1101314426a3ab24fd46a8974f [id=13fbed1c733c42e2b45e4ca69580b06d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.286550 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fbae8be73bf4e18b5a5c4ec4cdebc12. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.286780 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c76d1b509f04ab1a10b83cff675c11e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.290536 754 tablet_service.cc:1505] Processing CreateTablet for tablet 02e4c4d6ffb34cecaf17beb5ba197e81 (DEFAULT_TABLE table=default.Table_40e767d985de4910930e0114543720fd [id=bf3a3f8dae184d999f2159dfcc7803b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.290656 982 leader_election.cc:304] T 54cb4953ff264c13a0cc730b4c452ecc P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters: 3aea2869ff814cc7b142012faa274382
I20251025 07:52:26.290867 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02e4c4d6ffb34cecaf17beb5ba197e81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.291555 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 8c76d1b509f04ab1a10b83cff675c11e (DEFAULT_TABLE table=default.Table_cf686e1101314426a3ab24fd46a8974f [id=13fbed1c733c42e2b45e4ca69580b06d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.291889 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c76d1b509f04ab1a10b83cff675c11e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.293313 1172 log.cc:1199] T 54cb4953ff264c13a0cc730b4c452ecc P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/54cb4953ff264c13a0cc730b4c452ecc
I20251025 07:52:26.293720 1172 ts_tablet_manager.cc:1950] T 54cb4953ff264c13a0cc730b4c452ecc P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:26.294481 741 tablet_service.cc:1505] Processing CreateTablet for tablet 27651312cc2e42a8bfcd245f5d301679 (DEFAULT_TABLE table=default.Table_46d719baf05a4de09e6bd95dc90e4d0f [id=b0caa78ee8c84aa8bea4818cf9feb70c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.294903 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27651312cc2e42a8bfcd245f5d301679. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:26.297574 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet f3c87a83082b421cb4be546ac4ce7ad8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3c87a83082b421cb4be546ac4ce7ad8 already in progress: creating tablet
I20251025 07:52:26.298295 1247 raft_consensus.cc:2804] T 54cb4953ff264c13a0cc730b4c452ecc P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:26.298436 1247 raft_consensus.cc:493] T 54cb4953ff264c13a0cc730b4c452ecc P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:26.298534 1247 raft_consensus.cc:3060] T 54cb4953ff264c13a0cc730b4c452ecc P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:26.298902 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_2253f6700c2f4a3a9847023ba98b80ea" } modify_external_catalogs: true
I20251025 07:52:26.299116 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_2253f6700c2f4a3a9847023ba98b80ea" } modify_external_catalogs: true
W20251025 07:52:26.299526 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 5caac9c943bf42a5b2ae522d87fdfe8e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5caac9c943bf42a5b2ae522d87fdfe8e already in progress: creating tablet
I20251025 07:52:26.299808 1171 tablet_replica.cc:333] T d684da8134ce480693f470acdac176d4 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:26.299927 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 27651312cc2e42a8bfcd245f5d301679 (DEFAULT_TABLE table=default.Table_46d719baf05a4de09e6bd95dc90e4d0f [id=b0caa78ee8c84aa8bea4818cf9feb70c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.300027 1171 raft_consensus.cc:2243] T d684da8134ce480693f470acdac176d4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:26.300217 1171 raft_consensus.cc:2272] T d684da8134ce480693f470acdac176d4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:26.300400 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27651312cc2e42a8bfcd245f5d301679. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.304210 740 tablet_service.cc:1505] Processing CreateTablet for tablet 35ba9d23262248cda6fdf03c51be11a4 (DEFAULT_TABLE table=default.Table_072e0092adb64ef0ac6834dc18688ccf [id=1d4cbe4a4f38445297ceeebb8ffdaa8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.304745 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35ba9d23262248cda6fdf03c51be11a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.310583 1166 tablet_bootstrap.cc:492] T 20e4178c16bd4f18b1c6bc969f6a0c6c P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:26.313091 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 31) took 1419 ms. Trace:
I20251025 07:52:26.313221 1172 rpcz_store.cc:276] 1025 07:52:24.893913 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:24.894657 (+ 744us) service_pool.cc:225] Handling call
1025 07:52:26.195395 (+1300738us) ts_tablet_manager.cc:1151] Deleting tablet 54cb4953ff264c13a0cc730b4c452ecc
1025 07:52:26.195411 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:26.249397 (+ 53986us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:26.252686 (+ 3289us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:26.313027 (+ 60341us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1300566}
I20251025 07:52:26.313591 1172 tablet_replica.cc:333] T 3be8063aee794d2abcfb1d6e24209c06 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:26.313777 1172 raft_consensus.cc:2243] T 3be8063aee794d2abcfb1d6e24209c06 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:26.313971 1172 raft_consensus.cc:2272] T 3be8063aee794d2abcfb1d6e24209c06 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:26.314731 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 54cb4953ff264c13a0cc730b4c452ecc (table default.Table_ad812bcf86a04b0b89a3f0bc961062a9 [id=fe2800366dea4cdd85696d44df21a4c5]) successfully deleted
I20251025 07:52:26.316705 1166 tablet_bootstrap.cc:654] T 20e4178c16bd4f18b1c6bc969f6a0c6c P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:26.317937 753 tablet_service.cc:1505] Processing CreateTablet for tablet 6897405bbc1c47a4ab0aa3af80e60e3d (DEFAULT_TABLE table=default.Table_c361b0312923449da58cd2d6a012be15 [id=ea892976886644d8aa025ae2edef718d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.318368 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6897405bbc1c47a4ab0aa3af80e60e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.318835 1251 raft_consensus.cc:2706] T b946855345bc473aadd00cc4db5b7cd8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Received pre-election callback for term 1 while not running: Illegal state: RaftConsensus is not running: State = Stopped
I20251025 07:52:26.301050 1171 ts_tablet_manager.cc:1916] T d684da8134ce480693f470acdac176d4 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:26.326660 884 tablet_service.cc:1505] Processing CreateTablet for tablet 27651312cc2e42a8bfcd245f5d301679 (DEFAULT_TABLE table=default.Table_46d719baf05a4de09e6bd95dc90e4d0f [id=b0caa78ee8c84aa8bea4818cf9feb70c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.327157 1172 ts_tablet_manager.cc:1916] T 3be8063aee794d2abcfb1d6e24209c06 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:26.327999 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27651312cc2e42a8bfcd245f5d301679. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.333468 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 6897405bbc1c47a4ab0aa3af80e60e3d (DEFAULT_TABLE table=default.Table_c361b0312923449da58cd2d6a012be15 [id=ea892976886644d8aa025ae2edef718d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.334074 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6897405bbc1c47a4ab0aa3af80e60e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.334721 1247 raft_consensus.cc:515] T 54cb4953ff264c13a0cc730b4c452ecc P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:26.336294 982 leader_election.cc:343] T 54cb4953ff264c13a0cc730b4c452ecc P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: 54cb4953ff264c13a0cc730b4c452ecc
I20251025 07:52:26.336807 1247 leader_election.cc:290] T 54cb4953ff264c13a0cc730b4c452ecc P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:26.337529 1170 ts_tablet_manager.cc:1916] T b946855345bc473aadd00cc4db5b7cd8 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:26.338557 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_f3ff22d1c2824deca3f8267fc3768f28" } new_table_name: "default.Table_33b6fa5be1ab40fba35b3d5ef9ad9668" modify_external_catalogs: true
I20251025 07:52:26.339948 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_46d719baf05a4de09e6bd95dc90e4d0f" } new_table_name: "default.Table_dac528b081844b7c86027169c6234c27" modify_external_catalogs: true
I20251025 07:52:26.341115 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_60bfcc345b8649e9bc73fe3d077f46c7" } modify_external_catalogs: true
I20251025 07:52:26.341260 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_60bfcc345b8649e9bc73fe3d077f46c7" } modify_external_catalogs: true
I20251025 07:52:26.342924 753 tablet_service.cc:1505] Processing CreateTablet for tablet 2f26667f053a4f16b5da330c0b5e40f9 (DEFAULT_TABLE table=default.Table_4b3d7b3a731942189cdc214938a45820 [id=88f1e8b1c5fa4913a5f7b5b2c6cbf6ec]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:26.343323 983 leader_election.cc:343] T 54cb4953ff264c13a0cc730b4c452ecc P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 54cb4953ff264c13a0cc730b4c452ecc
I20251025 07:52:26.343384 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f26667f053a4f16b5da330c0b5e40f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.343464 983 leader_election.cc:304] T 54cb4953ff264c13a0cc730b4c452ecc P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:26.345352 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_4bafafcc9e8641b687d438725186f639" } modify_external_catalogs: true
I20251025 07:52:26.345533 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_4bafafcc9e8641b687d438725186f639" } modify_external_catalogs: true
I20251025 07:52:26.346047 1170 ts_tablet_manager.cc:1929] T b946855345bc473aadd00cc4db5b7cd8 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:26.346189 1170 log.cc:1199] T b946855345bc473aadd00cc4db5b7cd8 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/b946855345bc473aadd00cc4db5b7cd8
I20251025 07:52:26.346897 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_ed69674fe05b48a184be60a6f44fdbea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:26.347420 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ed69674fe05b48a184be60a6f44fdbea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.348469 518 master_service.cc:954] ReplaceTablet: received request to replace tablet 14fd790ba7054c1fb09cdfb19b001987 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:26.349097 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_8a00e989d44946ea86b431683cafe2ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:26.350006 1170 ts_tablet_manager.cc:1950] T b946855345bc473aadd00cc4db5b7cd8 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
W20251025 07:52:26.350111 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a00e989d44946ea86b431683cafe2ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.350927 1167 consensus_queue.cc:260] T b946855345bc473aadd00cc4db5b7cd8 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.351608 1167 ts_tablet_manager.cc:1434] T b946855345bc473aadd00cc4db5b7cd8 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.091s user 0.000s sys 0.003s
I20251025 07:52:26.352542 1167 tablet_bootstrap.cc:492] T a02b627401a64e1583f9e6123ef1be6f P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:26.355805 740 tablet_service.cc:1505] Processing CreateTablet for tablet 7f29b64704e147aa9b0b5f6cda860f09 (DEFAULT_TABLE table=default.Table_2253f6700c2f4a3a9847023ba98b80ea [id=17c206c8e24647ab9f483d9944dd3191]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.356055 750 tablet_service.cc:1505] Processing CreateTablet for tablet 4f25c9caa05b4da4bd56a8a21fde423c (DEFAULT_TABLE table=default.Table_60bfcc345b8649e9bc73fe3d077f46c7 [id=ed06dbbb96a24ab39619f4186ba93191]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.356464 741 tablet_service.cc:1552] Processing DeleteTablet for tablet b946855345bc473aadd00cc4db5b7cd8 with delete_type TABLET_DATA_DELETED (replaced by tablet fab67b56eaa44b54886c17181d8c52b2) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:26.356810 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f29b64704e147aa9b0b5f6cda860f09. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.357074 1218 raft_consensus.cc:493] T 89bfb7e2f051454aae0af5755a79ca58 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:26.357254 1218 raft_consensus.cc:515] T 89bfb7e2f051454aae0af5755a79ca58 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.357450 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f25c9caa05b4da4bd56a8a21fde423c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.360112 741 tablet_service.cc:1505] Processing CreateTablet for tablet b17d2a9ad6be49d29f2413666fba7e69 (DEFAULT_TABLE table=default.Table_ac1708f4dd0545f4b3411b218a61b6a5 [id=f3794fa081364666b944c412415bdba0]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:26.360419 512 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 50433881c0eb4aa5a389b6f5b4a4337a (table default.Table_c73137b8706c43b2865a93af89cd428c [id=a5c73cb1e5a349ca8675f4ba4cb8a82d], current schema version=6) failed: Not found: failed to reset TS proxy: No leader found: Tablet 50433881c0eb4aa5a389b6f5b4a4337a consensus state has no leader
I20251025 07:52:26.360697 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_936e8f1bd9344704940225361305cf5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:26.361253 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_936e8f1bd9344704940225361305cf5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.360739 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_c57548f494f84e5d92c0e461a7dde7d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:26.361752 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b17d2a9ad6be49d29f2413666fba7e69. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:26.361994 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c57548f494f84e5d92c0e461a7dde7d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.363580 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_37ad28b796a342f3a6e962ead289ec5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:26.364347 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_37ad28b796a342f3a6e962ead289ec5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.365341 894 tablet_service.cc:1505] Processing CreateTablet for tablet 4f25c9caa05b4da4bd56a8a21fde423c (DEFAULT_TABLE table=default.Table_60bfcc345b8649e9bc73fe3d077f46c7 [id=ed06dbbb96a24ab39619f4186ba93191]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.365461 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 7f29b64704e147aa9b0b5f6cda860f09
I20251025 07:52:26.366698 886 tablet_service.cc:1505] Processing CreateTablet for tablet 7f29b64704e147aa9b0b5f6cda860f09 (DEFAULT_TABLE table=default.Table_2253f6700c2f4a3a9847023ba98b80ea [id=17c206c8e24647ab9f483d9944dd3191]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.367113 1218 leader_election.cc:290] T 89bfb7e2f051454aae0af5755a79ca58 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:26.367218 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 7f29b64704e147aa9b0b5f6cda860f09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:26.367980 889 tablet_service.cc:1505] Processing CreateTablet for tablet 6897405bbc1c47a4ab0aa3af80e60e3d (DEFAULT_TABLE table=default.Table_c361b0312923449da58cd2d6a012be15 [id=ea892976886644d8aa025ae2edef718d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.373302 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f25c9caa05b4da4bd56a8a21fde423c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.373711 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f29b64704e147aa9b0b5f6cda860f09. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.378301 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6897405bbc1c47a4ab0aa3af80e60e3d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:26.378504 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2fac7b819e2642378e0d3f9230d0259c consensus state has no leader
W20251025 07:52:26.379062 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 27651312cc2e42a8bfcd245f5d301679 consensus state has no leader
I20251025 07:52:26.379503 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 4f25c9caa05b4da4bd56a8a21fde423c
I20251025 07:52:26.380300 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 85e27fc52e9a4d62a8622645594f59c6
I20251025 07:52:26.385159 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_c1743cb60e4d45939f387235c1726ef3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:26.385931 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1743cb60e4d45939f387235c1726ef3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.387248 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f25c9caa05b4da4bd56a8a21fde423c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:26.387512 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 85e27fc52e9a4d62a8622645594f59c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:26.388446 887 tablet_service.cc:1505] Processing CreateTablet for tablet b17d2a9ad6be49d29f2413666fba7e69 (DEFAULT_TABLE table=default.Table_ac1708f4dd0545f4b3411b218a61b6a5 [id=f3794fa081364666b944c412415bdba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.388935 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b17d2a9ad6be49d29f2413666fba7e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.389432 883 tablet_service.cc:1505] Processing CreateTablet for tablet 2f26667f053a4f16b5da330c0b5e40f9 (DEFAULT_TABLE table=default.Table_4b3d7b3a731942189cdc214938a45820 [id=88f1e8b1c5fa4913a5f7b5b2c6cbf6ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.389987 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f26667f053a4f16b5da330c0b5e40f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.394335 1247 raft_consensus.cc:2749] T 54cb4953ff264c13a0cc730b4c452ecc P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251025 07:52:26.396768 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_49beae553d03426aa395b7cdb6dfec9a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:26.397385 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_49beae553d03426aa395b7cdb6dfec9a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.398880 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_e244b2d2db74449aabd10a8b28e4afd2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:26.399482 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e244b2d2db74449aabd10a8b28e4afd2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.400913 1167 tablet_bootstrap.cc:654] T a02b627401a64e1583f9e6123ef1be6f P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:26.400992 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_e58048181b0d473d9195088bbef35387"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:26.401480 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e58048181b0d473d9195088bbef35387 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.403174 1172 ts_tablet_manager.cc:1929] T 3be8063aee794d2abcfb1d6e24209c06 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:26.404086 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_ac1708f4dd0545f4b3411b218a61b6a5" } new_table_name: "default.Table_f34f271c2f9242efa3ef3e78302be7d6" modify_external_catalogs: true
I20251025 07:52:26.405407 1021 tablet_service.cc:1505] Processing CreateTablet for tablet 7f29b64704e147aa9b0b5f6cda860f09 (DEFAULT_TABLE table=default.Table_2253f6700c2f4a3a9847023ba98b80ea [id=17c206c8e24647ab9f483d9944dd3191]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.405843 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f29b64704e147aa9b0b5f6cda860f09. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.406080 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 2f26667f053a4f16b5da330c0b5e40f9 (DEFAULT_TABLE table=default.Table_4b3d7b3a731942189cdc214938a45820 [id=88f1e8b1c5fa4913a5f7b5b2c6cbf6ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.408350 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet b946855345bc473aadd00cc4db5b7cd8 (table default.Table_6ca844d1e78c4f19939985aec32a8afd [id=1e933863244e4792b4c1b4d6d5c674b7]) successfully deleted
I20251025 07:52:26.409868 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_692ab786006945128bee1c8578c2a0cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:26.410456 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 7f29b64704e147aa9b0b5f6cda860f09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:48830
W20251025 07:52:26.410550 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_692ab786006945128bee1c8578c2a0cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.410897 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_3f53ffa013f94faf941d634d22a37ea4" } new_table_name: "default.Table_21a2393beb5d49a8ba839143c8328569" modify_external_catalogs: true
I20251025 07:52:26.413431 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 7f29b64704e147aa9b0b5f6cda860f09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:26.413444 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 85e27fc52e9a4d62a8622645594f59c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:26.414196 739 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f25c9caa05b4da4bd56a8a21fde423c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:26.414201 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_5aff13b5fcbc4899800760718fb3cc90" } modify_external_catalogs: true
I20251025 07:52:26.414369 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_5aff13b5fcbc4899800760718fb3cc90" } modify_external_catalogs: true
I20251025 07:52:26.415220 1170 tablet_replica.cc:333] T 54cb4953ff264c13a0cc730b4c452ecc P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:26.415637 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 85e27fc52e9a4d62a8622645594f59c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:26.410548 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_0042201efe904a84936c293cd01cebde" } new_table_name: "default.Table_9f0ae2442a334cf8ab13f2eba387ebf3" modify_external_catalogs: true
I20251025 07:52:26.417526 1170 raft_consensus.cc:2243] T 54cb4953ff264c13a0cc730b4c452ecc P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:26.417725 1170 raft_consensus.cc:2272] T 54cb4953ff264c13a0cc730b4c452ecc P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:26.423378 1170 ts_tablet_manager.cc:1916] T 54cb4953ff264c13a0cc730b4c452ecc P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:26.423789 1169 tablet_bootstrap.cc:492] T 70c3a36dda7b4919be20390f5797d09c P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:26.423969 1169 ts_tablet_manager.cc:1403] T 70c3a36dda7b4919be20390f5797d09c P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.175s user 0.004s sys 0.000s
I20251025 07:52:26.424144 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 4f25c9caa05b4da4bd56a8a21fde423c (DEFAULT_TABLE table=default.Table_60bfcc345b8649e9bc73fe3d077f46c7 [id=ed06dbbb96a24ab39619f4186ba93191]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.426081 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f26667f053a4f16b5da330c0b5e40f9. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:26.411226 847 leader_election.cc:343] T 89bfb7e2f051454aae0af5755a79ca58 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: 89bfb7e2f051454aae0af5755a79ca58
W20251025 07:52:26.427157 850 leader_election.cc:343] T 89bfb7e2f051454aae0af5755a79ca58 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 89bfb7e2f051454aae0af5755a79ca58
I20251025 07:52:26.427599 850 leader_election.cc:304] T 89bfb7e2f051454aae0af5755a79ca58 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:26.428042 1218 raft_consensus.cc:2749] T 89bfb7e2f051454aae0af5755a79ca58 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:26.428655 1172 log.cc:1199] T 3be8063aee794d2abcfb1d6e24209c06 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/3be8063aee794d2abcfb1d6e24209c06
I20251025 07:52:26.429080 1172 ts_tablet_manager.cc:1950] T 3be8063aee794d2abcfb1d6e24209c06 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:26.429857 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 32) took 1466 ms. Trace:
I20251025 07:52:26.432514 520 master_service.cc:954] ReplaceTablet: received request to replace tablet 5caac9c943bf42a5b2ae522d87fdfe8e from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:26.433554 520 master_service.cc:954] ReplaceTablet: received request to replace tablet dd898450f2604987b08639cf87d0e402 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:26.434556 1166 tablet_bootstrap.cc:492] T 20e4178c16bd4f18b1c6bc969f6a0c6c P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:26.434724 1166 ts_tablet_manager.cc:1403] T 20e4178c16bd4f18b1c6bc969f6a0c6c P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.142s user 0.001s sys 0.006s
I20251025 07:52:26.435552 1166 raft_consensus.cc:359] T 20e4178c16bd4f18b1c6bc969f6a0c6c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.435758 1166 raft_consensus.cc:385] T 20e4178c16bd4f18b1c6bc969f6a0c6c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:26.435834 1166 raft_consensus.cc:740] T 20e4178c16bd4f18b1c6bc969f6a0c6c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:26.436079 1166 consensus_queue.cc:260] T 20e4178c16bd4f18b1c6bc969f6a0c6c P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.436568 1166 ts_tablet_manager.cc:1434] T 20e4178c16bd4f18b1c6bc969f6a0c6c P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 07:52:26.436805 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f25c9caa05b4da4bd56a8a21fde423c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.436919 1166 tablet_bootstrap.cc:492] T 9cf88862e0934634b3dd172e6b40a0ba P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:26.429991 1172 rpcz_store.cc:276] 1025 07:52:24.963126 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:24.963180 (+ 54us) service_pool.cc:225] Handling call
1025 07:52:26.313491 (+1350311us) ts_tablet_manager.cc:1151] Deleting tablet 3be8063aee794d2abcfb1d6e24209c06
1025 07:52:26.313501 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:26.367297 (+ 53796us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:26.403030 (+ 35733us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:26.429823 (+ 26793us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1350076}
I20251025 07:52:26.437443 1172 tablet_replica.cc:333] T 70c3a36dda7b4919be20390f5797d09c P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:26.437655 1172 raft_consensus.cc:2243] T 70c3a36dda7b4919be20390f5797d09c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:26.437846 1172 raft_consensus.cc:2272] T 70c3a36dda7b4919be20390f5797d09c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:26.438589 1172 ts_tablet_manager.cc:1916] T 70c3a36dda7b4919be20390f5797d09c P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:26.439796 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b17d2a9ad6be49d29f2413666fba7e69 consensus state has no leader
W20251025 07:52:26.440011 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9bf486d82b984d2c89ee32e84a85894f consensus state has no leader
I20251025 07:52:26.441393 1169 raft_consensus.cc:359] T 70c3a36dda7b4919be20390f5797d09c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.441632 1169 raft_consensus.cc:385] T 70c3a36dda7b4919be20390f5797d09c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:26.443326 1166 tablet_bootstrap.cc:654] T 9cf88862e0934634b3dd172e6b40a0ba P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:26.445495 1171 ts_tablet_manager.cc:1929] T d684da8134ce480693f470acdac176d4 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:26.446525 1171 log.cc:1199] T d684da8134ce480693f470acdac176d4 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/d684da8134ce480693f470acdac176d4
I20251025 07:52:26.447016 1171 ts_tablet_manager.cc:1950] T d684da8134ce480693f470acdac176d4 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:26.448313 1171 tablet_replica.cc:333] T b946855345bc473aadd00cc4db5b7cd8 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:26.448431 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f25c9caa05b4da4bd56a8a21fde423c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:26.448599 1171 raft_consensus.cc:2243] T b946855345bc473aadd00cc4db5b7cd8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:26.448617 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet d684da8134ce480693f470acdac176d4 (table default.Table_8a06efe638604f20bda568cdb8382a48 [id=4443da834fc34a01989b1d8cce759866]) successfully deleted
I20251025 07:52:26.448840 1171 raft_consensus.cc:2272] T b946855345bc473aadd00cc4db5b7cd8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:26.448943 1169 raft_consensus.cc:740] T 70c3a36dda7b4919be20390f5797d09c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:26.449275 1169 consensus_queue.cc:260] T 70c3a36dda7b4919be20390f5797d09c P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.449607 1171 ts_tablet_manager.cc:1916] T b946855345bc473aadd00cc4db5b7cd8 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:26.449965 1169 ts_tablet_manager.cc:1434] T 70c3a36dda7b4919be20390f5797d09c P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
I20251025 07:52:26.450608 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_31d7f6f73d6e4123b1ff1df86ff62205" } modify_external_catalogs: true
I20251025 07:52:26.450836 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_31d7f6f73d6e4123b1ff1df86ff62205" } modify_external_catalogs: true
I20251025 07:52:26.456183 1167 tablet_bootstrap.cc:492] T a02b627401a64e1583f9e6123ef1be6f P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:26.456367 1167 ts_tablet_manager.cc:1403] T a02b627401a64e1583f9e6123ef1be6f P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.104s user 0.002s sys 0.002s
I20251025 07:52:26.457244 1167 raft_consensus.cc:359] T a02b627401a64e1583f9e6123ef1be6f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.457437 1167 raft_consensus.cc:385] T a02b627401a64e1583f9e6123ef1be6f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:26.457523 1167 raft_consensus.cc:740] T a02b627401a64e1583f9e6123ef1be6f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:26.457749 1167 consensus_queue.cc:260] T a02b627401a64e1583f9e6123ef1be6f P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.458333 1167 ts_tablet_manager.cc:1434] T a02b627401a64e1583f9e6123ef1be6f P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:26.458654 1167 tablet_bootstrap.cc:492] T 70c3a36dda7b4919be20390f5797d09c P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:26.461117 740 tablet_service.cc:1552] Processing DeleteTablet for tablet a02b627401a64e1583f9e6123ef1be6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:26.462368 1167 tablet_bootstrap.cc:654] T 70c3a36dda7b4919be20390f5797d09c P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:26.463397 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 2784bbfb97d5466f9ec856add7d0b29a
W20251025 07:52:26.468477 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 50ffbcdd4a9247bf99b8e2361bc02ef5 consensus state has no leader
I20251025 07:52:26.469192 740 tablet_service.cc:1552] Processing DeleteTablet for tablet 2784bbfb97d5466f9ec856add7d0b29a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:26.475045 517 master_service.cc:954] ReplaceTablet: received request to replace tablet 47284f96928749e6a5201103a8ee5f9f from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:26.477617 1166 tablet_bootstrap.cc:492] T 9cf88862e0934634b3dd172e6b40a0ba P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:26.477766 1166 ts_tablet_manager.cc:1403] T 9cf88862e0934634b3dd172e6b40a0ba P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.001s
I20251025 07:52:26.478535 519 master_service.cc:954] ReplaceTablet: received request to replace tablet 198db9a45257435f86df2605992ccb34 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:26.478575 1166 raft_consensus.cc:359] T 9cf88862e0934634b3dd172e6b40a0ba P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.478782 1166 raft_consensus.cc:385] T 9cf88862e0934634b3dd172e6b40a0ba P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:26.478875 1166 raft_consensus.cc:740] T 9cf88862e0934634b3dd172e6b40a0ba P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:26.479116 1166 consensus_queue.cc:260] T 9cf88862e0934634b3dd172e6b40a0ba P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.482264 1167 tablet_bootstrap.cc:492] T 70c3a36dda7b4919be20390f5797d09c P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:26.482455 1166 ts_tablet_manager.cc:1434] T 9cf88862e0934634b3dd172e6b40a0ba P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251025 07:52:26.482499 1167 ts_tablet_manager.cc:1403] T 70c3a36dda7b4919be20390f5797d09c P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.002s
I20251025 07:52:26.483897 1167 raft_consensus.cc:359] T 70c3a36dda7b4919be20390f5797d09c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.484151 1167 raft_consensus.cc:385] T 70c3a36dda7b4919be20390f5797d09c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:26.484275 1167 raft_consensus.cc:740] T 70c3a36dda7b4919be20390f5797d09c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:26.484481 1166 tablet_bootstrap.cc:492] T 785d0ce85b234826ba8021fbd182bf3c P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:26.485289 1167 consensus_queue.cc:260] T 70c3a36dda7b4919be20390f5797d09c P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.485414 1020 tablet_service.cc:1505] Processing CreateTablet for tablet b17d2a9ad6be49d29f2413666fba7e69 (DEFAULT_TABLE table=default.Table_ac1708f4dd0545f4b3411b218a61b6a5 [id=f3794fa081364666b944c412415bdba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.486603 1166 tablet_bootstrap.cc:654] T 785d0ce85b234826ba8021fbd182bf3c P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:26.488009 1171 ts_tablet_manager.cc:1929] T b946855345bc473aadd00cc4db5b7cd8 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:26.488705 1172 ts_tablet_manager.cc:1929] T 70c3a36dda7b4919be20390f5797d09c P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:26.489514 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 70c3a36dda7b4919be20390f5797d09c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:26.489562 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet 2784bbfb97d5466f9ec856add7d0b29a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:26.489753 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b17d2a9ad6be49d29f2413666fba7e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.489856 1169 tablet_bootstrap.cc:492] T 20e4178c16bd4f18b1c6bc969f6a0c6c P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:26.489965 1171 log.cc:1199] T b946855345bc473aadd00cc4db5b7cd8 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/b946855345bc473aadd00cc4db5b7cd8
I20251025 07:52:26.490417 1171 ts_tablet_manager.cc:1950] T b946855345bc473aadd00cc4db5b7cd8 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:26.490597 1172 log.cc:1199] T 70c3a36dda7b4919be20390f5797d09c P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/70c3a36dda7b4919be20390f5797d09c
I20251025 07:52:26.490993 1172 ts_tablet_manager.cc:1950] T 70c3a36dda7b4919be20390f5797d09c P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:26.492180 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet b946855345bc473aadd00cc4db5b7cd8 (table default.Table_6ca844d1e78c4f19939985aec32a8afd [id=1e933863244e4792b4c1b4d6d5c674b7]) successfully deleted
I20251025 07:52:26.492339 1171 tablet_replica.cc:333] T a02b627401a64e1583f9e6123ef1be6f P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:26.492479 1171 raft_consensus.cc:2243] T a02b627401a64e1583f9e6123ef1be6f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
W20251025 07:52:26.492496 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 7f29b64704e147aa9b0b5f6cda860f09 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f29b64704e147aa9b0b5f6cda860f09 already in progress: creating tablet
I20251025 07:52:26.492591 1171 raft_consensus.cc:2272] T a02b627401a64e1583f9e6123ef1be6f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:26.493124 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 4f25c9caa05b4da4bd56a8a21fde423c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f25c9caa05b4da4bd56a8a21fde423c already in progress: creating tablet
I20251025 07:52:26.493373 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 39) took 1510 ms. Trace:
W20251025 07:52:26.493387 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 85e27fc52e9a4d62a8622645594f59c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 85e27fc52e9a4d62a8622645594f59c6 already in progress: creating tablet
I20251025 07:52:26.493443 1172 rpcz_store.cc:276] 1025 07:52:24.982619 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:24.982676 (+ 57us) service_pool.cc:225] Handling call
1025 07:52:26.437371 (+1454695us) ts_tablet_manager.cc:1151] Deleting tablet 70c3a36dda7b4919be20390f5797d09c
1025 07:52:26.437381 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:26.471072 (+ 33691us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:26.488602 (+ 17530us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:26.493327 (+ 4725us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1454471}
I20251025 07:52:26.488166 1167 ts_tablet_manager.cc:1434] T 70c3a36dda7b4919be20390f5797d09c P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251025 07:52:26.488816 740 tablet_service.cc:1552] Processing DeleteTablet for tablet 70c3a36dda7b4919be20390f5797d09c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:26.495097 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 2784bbfb97d5466f9ec856add7d0b29a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:26.500471 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 70c3a36dda7b4919be20390f5797d09c (table default.Table_42908396d76f4df8b63e3ce11dcc2dfb [id=5e424c1c1c7c470f8640c7a7a571a11d]) successfully deleted
I20251025 07:52:26.500686 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 41) took 1516 ms. Trace:
I20251025 07:52:26.500777 1172 rpcz_store.cc:276] 1025 07:52:24.984054 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:24.984103 (+ 49us) service_pool.cc:225] Handling call
1025 07:52:26.500517 (+1516414us) ts_tablet_manager.cc:1151] Deleting tablet 713ae1dd444544bdabdd4a771e109d90
1025 07:52:26.500529 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:26.500638 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1514099}
W20251025 07:52:26.501129 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 713ae1dd444544bdabdd4a771e109d90 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 713ae1dd444544bdabdd4a771e109d90 already in progress: creating tablet
I20251025 07:52:26.501590 1171 ts_tablet_manager.cc:1916] T a02b627401a64e1583f9e6123ef1be6f P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:26.502265 1167 tablet_bootstrap.cc:492] T 3be8063aee794d2abcfb1d6e24209c06 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:26.504333 1172 tablet_replica.cc:333] T 20e4178c16bd4f18b1c6bc969f6a0c6c P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:26.504575 1172 raft_consensus.cc:2243] T 20e4178c16bd4f18b1c6bc969f6a0c6c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:26.504745 1167 tablet_bootstrap.cc:654] T 3be8063aee794d2abcfb1d6e24209c06 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:26.504822 1172 raft_consensus.cc:2272] T 20e4178c16bd4f18b1c6bc969f6a0c6c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:26.505780 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 5680e1adaf3f4b7f8c5e05523dd0950a
I20251025 07:52:26.505960 1166 tablet_bootstrap.cc:492] T 785d0ce85b234826ba8021fbd182bf3c P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:26.506114 1166 ts_tablet_manager.cc:1403] T 785d0ce85b234826ba8021fbd182bf3c P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20251025 07:52:26.506961 1166 raft_consensus.cc:359] T 785d0ce85b234826ba8021fbd182bf3c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.507179 1166 raft_consensus.cc:385] T 785d0ce85b234826ba8021fbd182bf3c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:26.507257 1166 raft_consensus.cc:740] T 785d0ce85b234826ba8021fbd182bf3c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:26.507514 1166 consensus_queue.cc:260] T 785d0ce85b234826ba8021fbd182bf3c P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.510535 1166 ts_tablet_manager.cc:1434] T 785d0ce85b234826ba8021fbd182bf3c P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251025 07:52:26.512653 1166 tablet_bootstrap.cc:492] T 2fac7b819e2642378e0d3f9230d0259c P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:26.515306 1166 tablet_bootstrap.cc:654] T 2fac7b819e2642378e0d3f9230d0259c P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:26.518688 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 5680e1adaf3f4b7f8c5e05523dd0950a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:26.519003 887 tablet_service.cc:1505] Processing CreateTablet for tablet fa77085e81c7440d9e255c44d1e80c26 (DEFAULT_TABLE table=default.Table_936e8f1bd9344704940225361305cf5b [id=4998c2abb8fb48db83483bc0aa0170eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.519176 883 tablet_service.cc:1505] Processing CreateTablet for tablet a6ddefa1738f4ff98586900c344359d2 (DEFAULT_TABLE table=default.Table_ed69674fe05b48a184be60a6f44fdbea [id=56e169f79a37465d82dd1d1c3c285740]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.519440 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa77085e81c7440d9e255c44d1e80c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.519646 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6ddefa1738f4ff98586900c344359d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.519961 741 tablet_service.cc:1505] Processing CreateTablet for tablet fa77085e81c7440d9e255c44d1e80c26 (DEFAULT_TABLE table=default.Table_936e8f1bd9344704940225361305cf5b [id=4998c2abb8fb48db83483bc0aa0170eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.520367 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa77085e81c7440d9e255c44d1e80c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.522789 884 tablet_service.cc:1505] Processing CreateTablet for tablet 0b21b7da1e2f47829c3f6cdaee689e8a (DEFAULT_TABLE table=default.Table_8a00e989d44946ea86b431683cafe2ed [id=6d26779bac7243f19d534277dd1c986c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.523177 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b21b7da1e2f47829c3f6cdaee689e8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.526446 750 tablet_service.cc:1505] Processing CreateTablet for tablet 0b21b7da1e2f47829c3f6cdaee689e8a (DEFAULT_TABLE table=default.Table_8a00e989d44946ea86b431683cafe2ed [id=6d26779bac7243f19d534277dd1c986c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.526829 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b21b7da1e2f47829c3f6cdaee689e8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.530522 754 tablet_service.cc:1505] Processing CreateTablet for tablet a6ddefa1738f4ff98586900c344359d2 (DEFAULT_TABLE table=default.Table_ed69674fe05b48a184be60a6f44fdbea [id=56e169f79a37465d82dd1d1c3c285740]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.531064 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6ddefa1738f4ff98586900c344359d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.531383 886 tablet_service.cc:1505] Processing CreateTablet for tablet 887e7c7a5d0e4aaf88f26246306df19c (DEFAULT_TABLE table=default.Table_c57548f494f84e5d92c0e461a7dde7d3 [id=8e07d0ad2db84d1b8a3a3a3efa2d7f69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.531756 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 887e7c7a5d0e4aaf88f26246306df19c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.533453 1171 ts_tablet_manager.cc:1929] T a02b627401a64e1583f9e6123ef1be6f P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:26.533608 1171 log.cc:1199] T a02b627401a64e1583f9e6123ef1be6f P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/a02b627401a64e1583f9e6123ef1be6f
I20251025 07:52:26.533974 1171 ts_tablet_manager.cc:1950] T a02b627401a64e1583f9e6123ef1be6f P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:26.534490 739 tablet_service.cc:1505] Processing CreateTablet for tablet 887e7c7a5d0e4aaf88f26246306df19c (DEFAULT_TABLE table=default.Table_c57548f494f84e5d92c0e461a7dde7d3 [id=8e07d0ad2db84d1b8a3a3a3efa2d7f69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.534679 1172 ts_tablet_manager.cc:1916] T 20e4178c16bd4f18b1c6bc969f6a0c6c P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:26.534973 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 887e7c7a5d0e4aaf88f26246306df19c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.538712 740 tablet_service.cc:1552] Processing DeleteTablet for tablet 5680e1adaf3f4b7f8c5e05523dd0950a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:26.541227 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 198db9a45257435f86df2605992ccb34
I20251025 07:52:26.541539 1172 ts_tablet_manager.cc:1929] T 20e4178c16bd4f18b1c6bc969f6a0c6c P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:26.541661 1172 log.cc:1199] T 20e4178c16bd4f18b1c6bc969f6a0c6c P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/20e4178c16bd4f18b1c6bc969f6a0c6c
I20251025 07:52:26.541795 519 catalog_manager.cc:6719] ReplaceTablet: tablet 198db9a45257435f86df2605992ccb34 deleted and replaced by tablet 71432a6848944a0da06ef9b81f160dc4
I20251025 07:52:26.543283 1169 tablet_bootstrap.cc:654] T 20e4178c16bd4f18b1c6bc969f6a0c6c P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:26.543354 740 tablet_service.cc:1552] Processing DeleteTablet for tablet 198db9a45257435f86df2605992ccb34 with delete_type TABLET_DATA_DELETED (replaced by tablet 71432a6848944a0da06ef9b81f160dc4) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:26.542563 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 198db9a45257435f86df2605992ccb34 with delete_type TABLET_DATA_DELETED (replaced by tablet 71432a6848944a0da06ef9b81f160dc4) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:26.544220 1166 tablet_bootstrap.cc:492] T 2fac7b819e2642378e0d3f9230d0259c P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:26.544598 1166 ts_tablet_manager.cc:1403] T 2fac7b819e2642378e0d3f9230d0259c P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.001s
I20251025 07:52:26.545514 1166 raft_consensus.cc:359] T 2fac7b819e2642378e0d3f9230d0259c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.545677 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet a02b627401a64e1583f9e6123ef1be6f (table default.Table_1437bb6ce9154a15b1817a029151c71a [id=9fd173d153294521a4f14c390b316594]) successfully deleted
I20251025 07:52:26.545779 1166 raft_consensus.cc:385] T 2fac7b819e2642378e0d3f9230d0259c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:26.545863 1166 raft_consensus.cc:740] T 2fac7b819e2642378e0d3f9230d0259c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:26.546113 1166 consensus_queue.cc:260] T 2fac7b819e2642378e0d3f9230d0259c P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.546973 1166 ts_tablet_manager.cc:1434] T 2fac7b819e2642378e0d3f9230d0259c P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251025 07:52:26.547312 1166 tablet_bootstrap.cc:492] T 713ae1dd444544bdabdd4a771e109d90 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:26.549086 1166 tablet_bootstrap.cc:654] T 713ae1dd444544bdabdd4a771e109d90 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:26.551687 1022 tablet_service.cc:1505] Processing CreateTablet for tablet fa77085e81c7440d9e255c44d1e80c26 (DEFAULT_TABLE table=default.Table_936e8f1bd9344704940225361305cf5b [id=4998c2abb8fb48db83483bc0aa0170eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.551960 1024 tablet_service.cc:1505] Processing CreateTablet for tablet a6ddefa1738f4ff98586900c344359d2 (DEFAULT_TABLE table=default.Table_ed69674fe05b48a184be60a6f44fdbea [id=56e169f79a37465d82dd1d1c3c285740]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.552129 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa77085e81c7440d9e255c44d1e80c26. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:26.552469 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 2784bbfb97d5466f9ec856add7d0b29a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2784bbfb97d5466f9ec856add7d0b29a already in progress: creating tablet
I20251025 07:52:26.552742 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 5680e1adaf3f4b7f8c5e05523dd0950a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:26.552999 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6ddefa1738f4ff98586900c344359d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.544638 1172 ts_tablet_manager.cc:1950] T 20e4178c16bd4f18b1c6bc969f6a0c6c P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:26.556556 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 42) took 1569 ms. Trace:
I20251025 07:52:26.557476 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 0b21b7da1e2f47829c3f6cdaee689e8a (DEFAULT_TABLE table=default.Table_8a00e989d44946ea86b431683cafe2ed [id=6d26779bac7243f19d534277dd1c986c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.556721 1172 rpcz_store.cc:276] 1025 07:52:24.986695 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:24.986748 (+ 53us) service_pool.cc:225] Handling call
1025 07:52:26.504252 (+1517504us) ts_tablet_manager.cc:1151] Deleting tablet 20e4178c16bd4f18b1c6bc969f6a0c6c
1025 07:52:26.504262 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:26.538366 (+ 34104us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:26.541417 (+ 3051us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:26.556146 (+ 14729us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1517327}
I20251025 07:52:26.558509 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 20e4178c16bd4f18b1c6bc969f6a0c6c (table default.Table_c73137b8706c43b2865a93af89cd428c [id=a5c73cb1e5a349ca8675f4ba4cb8a82d]) successfully deleted
I20251025 07:52:26.558563 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 54) took 1253 ms. Trace:
I20251025 07:52:26.558660 1172 rpcz_store.cc:276] 1025 07:52:25.305399 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:25.305486 (+ 87us) service_pool.cc:225] Handling call
1025 07:52:26.558405 (+1252919us) ts_tablet_manager.cc:1151] Deleting tablet a231f3d31b204f7b82f018892ccc8435
1025 07:52:26.558415 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:26.558530 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1240515}
I20251025 07:52:26.559000 1171 tablet_replica.cc:333] T 70c3a36dda7b4919be20390f5797d09c P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
W20251025 07:52:26.559056 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet a231f3d31b204f7b82f018892ccc8435 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a231f3d31b204f7b82f018892ccc8435 already in progress: creating tablet
I20251025 07:52:26.559264 1171 raft_consensus.cc:2243] T 70c3a36dda7b4919be20390f5797d09c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:26.559371 1172 tablet_replica.cc:333] T 9cf88862e0934634b3dd172e6b40a0ba P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:26.559474 1171 raft_consensus.cc:2272] T 70c3a36dda7b4919be20390f5797d09c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:26.559671 1172 raft_consensus.cc:2243] T 9cf88862e0934634b3dd172e6b40a0ba P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:26.559882 1172 raft_consensus.cc:2272] T 9cf88862e0934634b3dd172e6b40a0ba P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:26.560433 1171 ts_tablet_manager.cc:1916] T 70c3a36dda7b4919be20390f5797d09c P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:26.560847 1172 ts_tablet_manager.cc:1916] T 9cf88862e0934634b3dd172e6b40a0ba P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:26.560925 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 198db9a45257435f86df2605992ccb34 with delete_type TABLET_DATA_DELETED (replaced by tablet 71432a6848944a0da06ef9b81f160dc4) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:26.561092 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b21b7da1e2f47829c3f6cdaee689e8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.562868 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 887e7c7a5d0e4aaf88f26246306df19c (DEFAULT_TABLE table=default.Table_c57548f494f84e5d92c0e461a7dde7d3 [id=8e07d0ad2db84d1b8a3a3a3efa2d7f69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.563262 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 887e7c7a5d0e4aaf88f26246306df19c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.571455 1166 tablet_bootstrap.cc:492] T 713ae1dd444544bdabdd4a771e109d90 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:26.571709 1166 ts_tablet_manager.cc:1403] T 713ae1dd444544bdabdd4a771e109d90 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.003s
I20251025 07:52:26.572820 1166 raft_consensus.cc:359] T 713ae1dd444544bdabdd4a771e109d90 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.573104 1166 raft_consensus.cc:385] T 713ae1dd444544bdabdd4a771e109d90 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:26.573203 1166 raft_consensus.cc:740] T 713ae1dd444544bdabdd4a771e109d90 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:26.573511 1166 consensus_queue.cc:260] T 713ae1dd444544bdabdd4a771e109d90 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.574375 1166 ts_tablet_manager.cc:1434] T 713ae1dd444544bdabdd4a771e109d90 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20251025 07:52:26.574877 1166 tablet_bootstrap.cc:492] T 299572b2dbe244918dcebc5ea5f3c45a P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:26.576848 1166 tablet_bootstrap.cc:654] T 299572b2dbe244918dcebc5ea5f3c45a P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:26.578840 1172 ts_tablet_manager.cc:1929] T 9cf88862e0934634b3dd172e6b40a0ba P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:26.582561 1170 ts_tablet_manager.cc:1929] T 54cb4953ff264c13a0cc730b4c452ecc P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:26.582723 1170 log.cc:1199] T 54cb4953ff264c13a0cc730b4c452ecc P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/54cb4953ff264c13a0cc730b4c452ecc
I20251025 07:52:26.582872 1172 log.cc:1199] T 9cf88862e0934634b3dd172e6b40a0ba P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/9cf88862e0934634b3dd172e6b40a0ba
I20251025 07:52:26.583117 1170 ts_tablet_manager.cc:1950] T 54cb4953ff264c13a0cc730b4c452ecc P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:26.584357 1172 ts_tablet_manager.cc:1950] T 9cf88862e0934634b3dd172e6b40a0ba P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:26.584360 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_349ac535104c4b4dbaaf963de2640242" } new_table_name: "default.Table_bd5089a7892041649dd05b7fb0961255" modify_external_catalogs: true
I20251025 07:52:26.584518 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_e0bf55e24b084fdbb62f09cce80b50c5" } modify_external_catalogs: true
I20251025 07:52:26.584702 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_e0bf55e24b084fdbb62f09cce80b50c5" } modify_external_catalogs: true
I20251025 07:52:26.585570 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 55) took 1278 ms. Trace:
I20251025 07:52:26.585624 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_b6f727a336084bbf8d7d9c25a95105ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:26.585726 1172 rpcz_store.cc:276] 1025 07:52:25.306894 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:25.306940 (+ 46us) service_pool.cc:225] Handling call
1025 07:52:26.559313 (+1252373us) ts_tablet_manager.cc:1151] Deleting tablet 9cf88862e0934634b3dd172e6b40a0ba
1025 07:52:26.559322 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:26.567337 (+ 8015us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:26.578730 (+ 11393us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:26.585514 (+ 6784us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1241126}
W20251025 07:52:26.586215 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b6f727a336084bbf8d7d9c25a95105ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.586508 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 9cf88862e0934634b3dd172e6b40a0ba (table default.Table_1eb640b2196849a8b0136b390deea1c2 [id=ec85197b425f466ba7cd69e27caf6c93]) successfully deleted
I20251025 07:52:26.587019 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 56) took 1273 ms. Trace:
I20251025 07:52:26.587163 1172 rpcz_store.cc:276] 1025 07:52:25.313595 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:25.313645 (+ 50us) service_pool.cc:225] Handling call
1025 07:52:26.586844 (+1273199us) ts_tablet_manager.cc:1151] Deleting tablet 576c51d9a22d45f6a3642a7296b5bf23
1025 07:52:26.586855 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:26.586970 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1268388}
W20251025 07:52:26.587673 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 576c51d9a22d45f6a3642a7296b5bf23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 576c51d9a22d45f6a3642a7296b5bf23 already in progress: creating tablet
I20251025 07:52:26.588045 1172 tablet_replica.cc:333] T 785d0ce85b234826ba8021fbd182bf3c P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:26.588280 1172 raft_consensus.cc:2243] T 785d0ce85b234826ba8021fbd182bf3c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:26.588524 1172 raft_consensus.cc:2272] T 785d0ce85b234826ba8021fbd182bf3c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:26.590312 1171 ts_tablet_manager.cc:1929] T 70c3a36dda7b4919be20390f5797d09c P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:26.590492 1171 log.cc:1199] T 70c3a36dda7b4919be20390f5797d09c P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/70c3a36dda7b4919be20390f5797d09c
I20251025 07:52:26.590938 1171 ts_tablet_manager.cc:1950] T 70c3a36dda7b4919be20390f5797d09c P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:26.591675 1172 ts_tablet_manager.cc:1916] T 785d0ce85b234826ba8021fbd182bf3c P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:26.592234 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 70c3a36dda7b4919be20390f5797d09c (table default.Table_42908396d76f4df8b63e3ce11dcc2dfb [id=5e424c1c1c7c470f8640c7a7a571a11d]) successfully deleted
I20251025 07:52:26.592293 1171 tablet_replica.cc:333] T 5680e1adaf3f4b7f8c5e05523dd0950a P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:26.592497 1171 raft_consensus.cc:2243] T 5680e1adaf3f4b7f8c5e05523dd0950a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:26.592737 1171 raft_consensus.cc:2272] T 5680e1adaf3f4b7f8c5e05523dd0950a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:26.593003 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 299572b2dbe244918dcebc5ea5f3c45a consensus state has no leader
I20251025 07:52:26.595328 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_ebcfacbbec3147a189ff08cec113fc55" } modify_external_catalogs: true
I20251025 07:52:26.595515 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_ebcfacbbec3147a189ff08cec113fc55" } modify_external_catalogs: true
I20251025 07:52:26.596406 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_8faecb0cb37949a39f4ef5d9d2740c3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:26.597019 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8faecb0cb37949a39f4ef5d9d2740c3d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.600178 1166 tablet_bootstrap.cc:492] T 299572b2dbe244918dcebc5ea5f3c45a P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:26.600400 1166 ts_tablet_manager.cc:1403] T 299572b2dbe244918dcebc5ea5f3c45a P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.004s
I20251025 07:52:26.601620 1166 raft_consensus.cc:359] T 299572b2dbe244918dcebc5ea5f3c45a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.601971 1166 raft_consensus.cc:385] T 299572b2dbe244918dcebc5ea5f3c45a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:26.602502 1166 raft_consensus.cc:740] T 299572b2dbe244918dcebc5ea5f3c45a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:26.602991 1166 consensus_queue.cc:260] T 299572b2dbe244918dcebc5ea5f3c45a P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.604067 1166 ts_tablet_manager.cc:1434] T 299572b2dbe244918dcebc5ea5f3c45a P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251025 07:52:26.604530 1166 tablet_bootstrap.cc:492] T 576c51d9a22d45f6a3642a7296b5bf23 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:26.606585 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_40e767d985de4910930e0114543720fd" } new_table_name: "default.Table_cfa54e1a04154e2eb8747fd3a4812883" modify_external_catalogs: true
I20251025 07:52:26.608757 1171 ts_tablet_manager.cc:1916] T 5680e1adaf3f4b7f8c5e05523dd0950a P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:26.609730 1166 tablet_bootstrap.cc:654] T 576c51d9a22d45f6a3642a7296b5bf23 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:26.612587 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_590cc281d8b042b19ef12c7f6436752a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:26.613114 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_590cc281d8b042b19ef12c7f6436752a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.617125 1172 ts_tablet_manager.cc:1929] T 785d0ce85b234826ba8021fbd182bf3c P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:26.617312 1172 log.cc:1199] T 785d0ce85b234826ba8021fbd182bf3c P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/785d0ce85b234826ba8021fbd182bf3c
I20251025 07:52:26.617753 1172 ts_tablet_manager.cc:1950] T 785d0ce85b234826ba8021fbd182bf3c P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:26.618302 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_cf686e1101314426a3ab24fd46a8974f" } modify_external_catalogs: true
I20251025 07:52:26.618618 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 60) took 1287 ms. Trace:
I20251025 07:52:26.618742 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_cf686e1101314426a3ab24fd46a8974f" } modify_external_catalogs: true
I20251025 07:52:26.618739 1172 rpcz_store.cc:276] 1025 07:52:25.331319 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:25.331371 (+ 52us) service_pool.cc:225] Handling call
1025 07:52:26.587960 (+1256589us) ts_tablet_manager.cc:1151] Deleting tablet 785d0ce85b234826ba8021fbd182bf3c
1025 07:52:26.587971 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:26.607457 (+ 19486us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:26.616982 (+ 9525us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:26.618579 (+ 1597us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1251096}
I20251025 07:52:26.619222 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 785d0ce85b234826ba8021fbd182bf3c (table default.Table_e0e89ced2ea94c139ed635849e78cb35 [id=807c428924b74fde847c2becad78d4ef]) successfully deleted
I20251025 07:52:26.619637 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 61) took 1283 ms. Trace:
I20251025 07:52:26.619774 1172 rpcz_store.cc:276] 1025 07:52:25.336112 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:25.336153 (+ 41us) service_pool.cc:225] Handling call
1025 07:52:26.619490 (+1283337us) ts_tablet_manager.cc:1151] Deleting tablet 2947502815d742e89cacb474b7fbb249
1025 07:52:26.619499 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:26.619605 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1280847}
W20251025 07:52:26.620240 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 2947502815d742e89cacb474b7fbb249 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2947502815d742e89cacb474b7fbb249 already in progress: creating tablet
I20251025 07:52:26.620534 1172 tablet_replica.cc:333] T 14fd790ba7054c1fb09cdfb19b001987 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:26.620759 1172 raft_consensus.cc:2243] T 14fd790ba7054c1fb09cdfb19b001987 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:26.620941 1172 raft_consensus.cc:2272] T 14fd790ba7054c1fb09cdfb19b001987 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:26.625339 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 02e4c4d6ffb34cecaf17beb5ba197e81 consensus state has no leader
I20251025 07:52:26.626581 1166 tablet_bootstrap.cc:492] T 576c51d9a22d45f6a3642a7296b5bf23 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:26.626856 1166 ts_tablet_manager.cc:1403] T 576c51d9a22d45f6a3642a7296b5bf23 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.001s
I20251025 07:52:26.627595 1171 ts_tablet_manager.cc:1929] T 5680e1adaf3f4b7f8c5e05523dd0950a P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:26.627802 1172 ts_tablet_manager.cc:1916] T 14fd790ba7054c1fb09cdfb19b001987 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:26.628010 1166 raft_consensus.cc:359] T 576c51d9a22d45f6a3642a7296b5bf23 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.628479 1166 raft_consensus.cc:385] T 576c51d9a22d45f6a3642a7296b5bf23 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:26.628639 1166 raft_consensus.cc:740] T 576c51d9a22d45f6a3642a7296b5bf23 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:26.631690 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_c9bc68f9b5c54a33b854f9ee8a1ad311"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:26.632671 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9bc68f9b5c54a33b854f9ee8a1ad311 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.634295 1171 log.cc:1199] T 5680e1adaf3f4b7f8c5e05523dd0950a P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/5680e1adaf3f4b7f8c5e05523dd0950a
I20251025 07:52:26.634526 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_acca538f4b584f3ea4a55921a708d5a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:26.634842 1171 ts_tablet_manager.cc:1950] T 5680e1adaf3f4b7f8c5e05523dd0950a P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
W20251025 07:52:26.635309 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_acca538f4b584f3ea4a55921a708d5a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.636041 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 5680e1adaf3f4b7f8c5e05523dd0950a (table default.Table_31d7f6f73d6e4123b1ff1df86ff62205 [id=cb037f61ee0b4b6cb8fb613263a278fa]) successfully deleted
I20251025 07:52:26.629091 1166 consensus_queue.cc:260] T 576c51d9a22d45f6a3642a7296b5bf23 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.646711 1218 raft_consensus.cc:493] T 576c51d9a22d45f6a3642a7296b5bf23 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:26.646955 1218 raft_consensus.cc:515] T 576c51d9a22d45f6a3642a7296b5bf23 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.648088 1218 leader_election.cc:290] T 576c51d9a22d45f6a3642a7296b5bf23 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:26.650053 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "576c51d9a22d45f6a3642a7296b5bf23" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:26.650657 850 leader_election.cc:343] T 576c51d9a22d45f6a3642a7296b5bf23 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:26.651013 850 consensus_peers.cc:597] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e84dbbd4a0bd4e4dbf1212fefbf00fd4. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:26.651134 1170 tablet_replica.cc:333] T 14fd790ba7054c1fb09cdfb19b001987 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:26.651315 1170 raft_consensus.cc:2243] T 14fd790ba7054c1fb09cdfb19b001987 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:26.651476 1170 raft_consensus.cc:2272] T 14fd790ba7054c1fb09cdfb19b001987 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:26.651997 1170 ts_tablet_manager.cc:1916] T 14fd790ba7054c1fb09cdfb19b001987 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:26.652182 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_8a00e989d44946ea86b431683cafe2ed" } new_table_name: "default.Table_e61ae9065c604ad8b93af8146748cec9" modify_external_catalogs: true
I20251025 07:52:26.654049 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_e49cd337b7254c18ba468dddbb318d4c" } new_table_name: "default.Table_10e57b42f97544208675708b1e028846" modify_external_catalogs: true
I20251025 07:52:26.654420 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 54cb4953ff264c13a0cc730b4c452ecc (table default.Table_ad812bcf86a04b0b89a3f0bc961062a9 [id=fe2800366dea4cdd85696d44df21a4c5]) successfully deleted
W20251025 07:52:26.654717 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 1b31177a062b467187d68ed2034f5690 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b31177a062b467187d68ed2034f5690 already in progress: creating tablet
W20251025 07:52:26.654963 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 08da9d2687704725aacd11cd92c6a9f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08da9d2687704725aacd11cd92c6a9f9 already in progress: creating tablet
W20251025 07:52:26.655857 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet ac0ea298bc114c56a8093e12def21ce1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac0ea298bc114c56a8093e12def21ce1 already in progress: creating tablet
W20251025 07:52:26.656153 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet b00153c97104470883f6089d71d23708 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b00153c97104470883f6089d71d23708 already in progress: creating tablet
I20251025 07:52:26.656276 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_3cf5eecb57c54611bdcc3dfca5fcdca1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:26.657361 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_133ad13fd9b340ffbab4f64a48f0474c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:26.661069 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_4b63840ca3ac4296b23104bf46030bb4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:26.665748 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 198db9a45257435f86df2605992ccb34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 198db9a45257435f86df2605992ccb34 already in progress: creating tablet
I20251025 07:52:26.670002 1166 ts_tablet_manager.cc:1434] T 576c51d9a22d45f6a3642a7296b5bf23 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.042s user 0.004s sys 0.000s
I20251025 07:52:26.674628 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_04074ebcb0224168862553e3cf731987"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:26.678450 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0b21b7da1e2f47829c3f6cdaee689e8a consensus state has no leader
I20251025 07:52:26.679013 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_c1743cb60e4d45939f387235c1726ef3" } modify_external_catalogs: true
I20251025 07:52:26.679245 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_c1743cb60e4d45939f387235c1726ef3" } modify_external_catalogs: true
W20251025 07:52:26.679829 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8e6d76b1a97945cfb6c3055e1b622f8a consensus state has no leader
I20251025 07:52:26.680241 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_98454703d7234750b3cb2aab947ef63e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:26.681108 1172 ts_tablet_manager.cc:1929] T 14fd790ba7054c1fb09cdfb19b001987 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251025 07:52:26.682502 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_133ad13fd9b340ffbab4f64a48f0474c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.682811 1172 log.cc:1199] T 14fd790ba7054c1fb09cdfb19b001987 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/14fd790ba7054c1fb09cdfb19b001987
I20251025 07:52:26.683503 1172 ts_tablet_manager.cc:1950] T 14fd790ba7054c1fb09cdfb19b001987 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
W20251025 07:52:26.683482 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_04074ebcb0224168862553e3cf731987 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:26.684094 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3cf5eecb57c54611bdcc3dfca5fcdca1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.685016 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 64) took 1191 ms. Trace:
W20251025 07:52:26.685349 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98454703d7234750b3cb2aab947ef63e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:26.693361 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b63840ca3ac4296b23104bf46030bb4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.697615 1169 tablet_bootstrap.cc:492] T 20e4178c16bd4f18b1c6bc969f6a0c6c P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:26.697806 1169 ts_tablet_manager.cc:1403] T 20e4178c16bd4f18b1c6bc969f6a0c6c P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.225s user 0.009s sys 0.001s
W20251025 07:52:26.698968 511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:37611: connect: Connection refused (error 111) [suppressed 7 similar messages]
I20251025 07:52:26.699342 1169 raft_consensus.cc:359] T 20e4178c16bd4f18b1c6bc969f6a0c6c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.699581 1169 raft_consensus.cc:385] T 20e4178c16bd4f18b1c6bc969f6a0c6c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:26.699651 1169 raft_consensus.cc:740] T 20e4178c16bd4f18b1c6bc969f6a0c6c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:26.686151 1166 tablet_bootstrap.cc:492] T 4829ec41391145cc84327d60048d9b0c P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:26.699926 1169 consensus_queue.cc:260] T 20e4178c16bd4f18b1c6bc969f6a0c6c P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.702292 1169 ts_tablet_manager.cc:1434] T 20e4178c16bd4f18b1c6bc969f6a0c6c P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20251025 07:52:26.702442 1166 tablet_bootstrap.cc:654] T 4829ec41391145cc84327d60048d9b0c P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:26.703819 1172 rpcz_store.cc:276] 1025 07:52:25.493245 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:25.502942 (+ 9697us) service_pool.cc:225] Handling call
1025 07:52:26.620475 (+1117533us) ts_tablet_manager.cc:1151] Deleting tablet 14fd790ba7054c1fb09cdfb19b001987
1025 07:52:26.620485 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:26.631566 (+ 11081us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:26.680966 (+ 49400us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:26.684963 (+ 3997us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1117284}
I20251025 07:52:26.704419 1169 tablet_bootstrap.cc:492] T 3be8063aee794d2abcfb1d6e24209c06 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:26.704447 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 66) took 1181 ms. Trace:
I20251025 07:52:26.704636 1172 rpcz_store.cc:276] 1025 07:52:25.523248 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:25.523301 (+ 53us) service_pool.cc:225] Handling call
1025 07:52:26.704254 (+1180953us) ts_tablet_manager.cc:1151] Deleting tablet cf2cab1552bd4e7394797c93b149f450
1025 07:52:26.704265 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:26.704391 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1179987}
I20251025 07:52:26.705178 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 65) took 1182 ms. Trace:
I20251025 07:52:26.705332 1172 rpcz_store.cc:276] 1025 07:52:25.523065 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:25.523123 (+ 58us) service_pool.cc:225] Handling call
1025 07:52:26.704983 (+1181860us) ts_tablet_manager.cc:1151] Deleting tablet 33c608beb4ef4a859a135b11e07c81a4
1025 07:52:26.704992 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:26.705113 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1180419}
I20251025 07:52:26.705736 1172 tablet_replica.cc:333] T 89bfb7e2f051454aae0af5755a79ca58 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:26.705982 1172 raft_consensus.cc:2243] T 89bfb7e2f051454aae0af5755a79ca58 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:26.706266 1169 tablet_bootstrap.cc:654] T 3be8063aee794d2abcfb1d6e24209c06 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:26.706284 1172 raft_consensus.cc:2272] T 89bfb7e2f051454aae0af5755a79ca58 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:26.706830 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_66a70e784e9b49cdb516f657733f2a38" } new_table_name: "default.Table_808d3c7f83104ccbbda7ee2e03db42a2" modify_external_catalogs: true
I20251025 07:52:26.712136 519 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 2fac7b819e2642378e0d3f9230d0259c (table default.Table_33b6fa5be1ab40fba35b3d5ef9ad9668 [id=6605e11e11804d408a6a4a9d403dde95]). Expected version 1 got 0
I20251025 07:52:26.706928 1172 ts_tablet_manager.cc:1916] T 89bfb7e2f051454aae0af5755a79ca58 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:26.713945 1166 tablet_bootstrap.cc:492] T 4829ec41391145cc84327d60048d9b0c P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:26.714112 1166 ts_tablet_manager.cc:1403] T 4829ec41391145cc84327d60048d9b0c P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.003s
I20251025 07:52:26.715451 1166 raft_consensus.cc:359] T 4829ec41391145cc84327d60048d9b0c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.715647 1166 raft_consensus.cc:385] T 4829ec41391145cc84327d60048d9b0c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:26.715751 1166 raft_consensus.cc:740] T 4829ec41391145cc84327d60048d9b0c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:26.716030 1166 consensus_queue.cc:260] T 4829ec41391145cc84327d60048d9b0c P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.716553 1166 ts_tablet_manager.cc:1434] T 4829ec41391145cc84327d60048d9b0c P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251025 07:52:26.717432 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_6ca844d1e78c4f19939985aec32a8afd" } modify_external_catalogs: true
I20251025 07:52:26.717643 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_6ca844d1e78c4f19939985aec32a8afd" } modify_external_catalogs: true
I20251025 07:52:26.717794 1166 tablet_bootstrap.cc:492] T 3d5e0dd14e474fe09993a03851f8b1f0 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:26.720526 1166 tablet_bootstrap.cc:654] T 3d5e0dd14e474fe09993a03851f8b1f0 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:26.721072 739 tablet_service.cc:1505] Processing CreateTablet for tablet f0bdaa8cfd764f1b83f47881353d26b6 (DEFAULT_TABLE table=default.Table_49beae553d03426aa395b7cdb6dfec9a [id=ab3328aa6f4148bfa2877cd4f0c6d324]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.721500 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0bdaa8cfd764f1b83f47881353d26b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.722955 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 14fd790ba7054c1fb09cdfb19b001987 (table default.Table_43eec86aa9fc405fa761a8eaa89d2db0 [id=67beaeffaf2243ef86f6c45084781432]) successfully deleted
I20251025 07:52:26.723007 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "576c51d9a22d45f6a3642a7296b5bf23" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
W20251025 07:52:26.723201 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet cf2cab1552bd4e7394797c93b149f450 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf2cab1552bd4e7394797c93b149f450 already in progress: creating tablet
W20251025 07:52:26.723409 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 33c608beb4ef4a859a135b11e07c81a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33c608beb4ef4a859a135b11e07c81a4 already in progress: creating tablet
I20251025 07:52:26.723418 886 tablet_service.cc:1505] Processing CreateTablet for tablet bef0ef23d90d480d9c99aef21d7271fd (DEFAULT_TABLE table=default.Table_e58048181b0d473d9195088bbef35387 [id=ce5f22039d104e518ef6f40468b404cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.723611 750 tablet_service.cc:1505] Processing CreateTablet for tablet bef0ef23d90d480d9c99aef21d7271fd (DEFAULT_TABLE table=default.Table_e58048181b0d473d9195088bbef35387 [id=ce5f22039d104e518ef6f40468b404cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.723801 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bef0ef23d90d480d9c99aef21d7271fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.723950 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bef0ef23d90d480d9c99aef21d7271fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.726397 754 tablet_service.cc:1505] Processing CreateTablet for tablet cb824bd780524832b2b60f5618a39da5 (DEFAULT_TABLE table=default.Table_c1743cb60e4d45939f387235c1726ef3 [id=18e29b94f8a6443fb8e6a876119fd9a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.726776 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb824bd780524832b2b60f5618a39da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.726841 1170 ts_tablet_manager.cc:1929] T 14fd790ba7054c1fb09cdfb19b001987 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:26.726972 1170 log.cc:1199] T 14fd790ba7054c1fb09cdfb19b001987 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/14fd790ba7054c1fb09cdfb19b001987
I20251025 07:52:26.727340 1170 ts_tablet_manager.cc:1950] T 14fd790ba7054c1fb09cdfb19b001987 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:26.731359 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 14fd790ba7054c1fb09cdfb19b001987 (table default.Table_43eec86aa9fc405fa761a8eaa89d2db0 [id=67beaeffaf2243ef86f6c45084781432]) successfully deleted
I20251025 07:52:26.732769 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 3775a2e08d4e40bd9f57e4cda759d535 (DEFAULT_TABLE table=default.Table_37ad28b796a342f3a6e962ead289ec5b [id=a4d25738410b4a168aa0c76094daf69b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.733201 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3775a2e08d4e40bd9f57e4cda759d535. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.733713 741 tablet_service.cc:1505] Processing CreateTablet for tablet 3775a2e08d4e40bd9f57e4cda759d535 (DEFAULT_TABLE table=default.Table_37ad28b796a342f3a6e962ead289ec5b [id=a4d25738410b4a168aa0c76094daf69b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.734153 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3775a2e08d4e40bd9f57e4cda759d535. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.736006 524 master_service.cc:954] ReplaceTablet: received request to replace tablet 713ae1dd444544bdabdd4a771e109d90 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:26.736522 524 master_service.cc:954] ReplaceTablet: received request to replace tablet a02b627401a64e1583f9e6123ef1be6f from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:26.739004 740 tablet_service.cc:1505] Processing CreateTablet for tablet 2209089667dd4937abd96615f6929449 (DEFAULT_TABLE table=default.Table_692ab786006945128bee1c8578c2a0cd [id=1780f0b073574d5dad0e1489d1f777e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.739451 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2209089667dd4937abd96615f6929449. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.740772 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 2209089667dd4937abd96615f6929449 (DEFAULT_TABLE table=default.Table_692ab786006945128bee1c8578c2a0cd [id=1780f0b073574d5dad0e1489d1f777e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.741168 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2209089667dd4937abd96615f6929449. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.742442 753 tablet_service.cc:1505] Processing CreateTablet for tablet 8ff73ffc42434f74a4949f156771d5a4 (DEFAULT_TABLE table=default.Table_e244b2d2db74449aabd10a8b28e4afd2 [id=8bb9f6cbc77241f0b64c7ab15d1fe266]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.742794 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ff73ffc42434f74a4949f156771d5a4. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:26.751209 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet f3c87a83082b421cb4be546ac4ce7ad8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3c87a83082b421cb4be546ac4ce7ad8 already in progress: creating tablet
I20251025 07:52:26.751633 1021 tablet_service.cc:1505] Processing CreateTablet for tablet 8ff73ffc42434f74a4949f156771d5a4 (DEFAULT_TABLE table=default.Table_e244b2d2db74449aabd10a8b28e4afd2 [id=8bb9f6cbc77241f0b64c7ab15d1fe266]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.752048 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ff73ffc42434f74a4949f156771d5a4. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:26.752717 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 94408a3a90224645ba3cacccf5de33f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94408a3a90224645ba3cacccf5de33f1 already in progress: creating tablet
W20251025 07:52:26.753299 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 85e27fc52e9a4d62a8622645594f59c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 85e27fc52e9a4d62a8622645594f59c6 already in progress: creating tablet
W20251025 07:52:26.753880 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 7f29b64704e147aa9b0b5f6cda860f09 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f29b64704e147aa9b0b5f6cda860f09 already in progress: creating tablet
I20251025 07:52:26.754415 1169 tablet_bootstrap.cc:492] T 3be8063aee794d2abcfb1d6e24209c06 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
W20251025 07:52:26.754515 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 4f25c9caa05b4da4bd56a8a21fde423c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f25c9caa05b4da4bd56a8a21fde423c already in progress: creating tablet
I20251025 07:52:26.754673 1169 ts_tablet_manager.cc:1403] T 3be8063aee794d2abcfb1d6e24209c06 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.002s
I20251025 07:52:26.754886 1170 tablet_replica.cc:333] T 70c3a36dda7b4919be20390f5797d09c P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:26.755141 1170 raft_consensus.cc:2243] T 70c3a36dda7b4919be20390f5797d09c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:26.755777 1169 raft_consensus.cc:359] T 3be8063aee794d2abcfb1d6e24209c06 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.756021 1169 raft_consensus.cc:385] T 3be8063aee794d2abcfb1d6e24209c06 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:26.756158 1169 raft_consensus.cc:740] T 3be8063aee794d2abcfb1d6e24209c06 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:26.756462 1169 consensus_queue.cc:260] T 3be8063aee794d2abcfb1d6e24209c06 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.756973 1169 ts_tablet_manager.cc:1434] T 3be8063aee794d2abcfb1d6e24209c06 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:26.758993 1169 tablet_bootstrap.cc:492] T a02b627401a64e1583f9e6123ef1be6f P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:26.759243 1167 tablet_bootstrap.cc:492] T 3be8063aee794d2abcfb1d6e24209c06 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:26.759455 1167 ts_tablet_manager.cc:1403] T 3be8063aee794d2abcfb1d6e24209c06 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.264s user 0.044s sys 0.083s
I20251025 07:52:26.759886 519 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 299572b2dbe244918dcebc5ea5f3c45a (table default.Table_bd5089a7892041649dd05b7fb0961255 [id=840b3ffcb44144548843457c48f46380]). Expected version 1 got 0
I20251025 07:52:26.760141 519 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 4829ec41391145cc84327d60048d9b0c (table default.Table_45b1ec3be7e9460296fe5b819beaa6ea [id=94a9a4e752b547afb3d4d0b6806bdb54]). Expected version 2 got 0
I20251025 07:52:26.760519 1167 raft_consensus.cc:359] T 3be8063aee794d2abcfb1d6e24209c06 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.760774 1169 tablet_bootstrap.cc:654] T a02b627401a64e1583f9e6123ef1be6f P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:26.760805 1167 raft_consensus.cc:385] T 3be8063aee794d2abcfb1d6e24209c06 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:26.760916 1167 raft_consensus.cc:740] T 3be8063aee794d2abcfb1d6e24209c06 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:26.755376 1170 raft_consensus.cc:2272] T 70c3a36dda7b4919be20390f5797d09c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:26.761159 1167 consensus_queue.cc:260] T 3be8063aee794d2abcfb1d6e24209c06 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.762001 1170 ts_tablet_manager.cc:1916] T 70c3a36dda7b4919be20390f5797d09c P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:26.762488 1268 raft_consensus.cc:493] T 2fac7b819e2642378e0d3f9230d0259c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:26.762646 1268 raft_consensus.cc:515] T 2fac7b819e2642378e0d3f9230d0259c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.763351 1268 leader_election.cc:290] T 2fac7b819e2642378e0d3f9230d0259c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:26.763849 1026 tablet_service.cc:1505] Processing CreateTablet for tablet cb824bd780524832b2b60f5618a39da5 (DEFAULT_TABLE table=default.Table_c1743cb60e4d45939f387235c1726ef3 [id=18e29b94f8a6443fb8e6a876119fd9a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.764263 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb824bd780524832b2b60f5618a39da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.765169 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fac7b819e2642378e0d3f9230d0259c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:26.767781 1167 ts_tablet_manager.cc:1434] T 3be8063aee794d2abcfb1d6e24209c06 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251025 07:52:26.768805 889 tablet_service.cc:1505] Processing CreateTablet for tablet 2209089667dd4937abd96615f6929449 (DEFAULT_TABLE table=default.Table_692ab786006945128bee1c8578c2a0cd [id=1780f0b073574d5dad0e1489d1f777e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.769223 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2209089667dd4937abd96615f6929449. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.770624 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fac7b819e2642378e0d3f9230d0259c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:26.772384 1167 tablet_bootstrap.cc:492] T 20e4178c16bd4f18b1c6bc969f6a0c6c P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:26.773766 887 tablet_service.cc:1505] Processing CreateTablet for tablet cb824bd780524832b2b60f5618a39da5 (DEFAULT_TABLE table=default.Table_c1743cb60e4d45939f387235c1726ef3 [id=18e29b94f8a6443fb8e6a876119fd9a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.755081 1022 tablet_service.cc:1505] Processing CreateTablet for tablet bef0ef23d90d480d9c99aef21d7271fd (DEFAULT_TABLE table=default.Table_e58048181b0d473d9195088bbef35387 [id=ce5f22039d104e518ef6f40468b404cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.774459 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bef0ef23d90d480d9c99aef21d7271fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.774819 1167 tablet_bootstrap.cc:654] T 20e4178c16bd4f18b1c6bc969f6a0c6c P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:26.777119 1247 raft_consensus.cc:493] T 3be8063aee794d2abcfb1d6e24209c06 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:26.777311 1247 raft_consensus.cc:515] T 3be8063aee794d2abcfb1d6e24209c06 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.778101 1247 leader_election.cc:290] T 3be8063aee794d2abcfb1d6e24209c06 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:26.779094 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3be8063aee794d2abcfb1d6e24209c06" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:26.779340 775 raft_consensus.cc:2468] T 3be8063aee794d2abcfb1d6e24209c06 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:26.779860 982 leader_election.cc:304] T 3be8063aee794d2abcfb1d6e24209c06 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:26.780221 1247 raft_consensus.cc:2804] T 3be8063aee794d2abcfb1d6e24209c06 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:26.780332 1247 raft_consensus.cc:493] T 3be8063aee794d2abcfb1d6e24209c06 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:26.780452 1247 raft_consensus.cc:3060] T 3be8063aee794d2abcfb1d6e24209c06 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:26.782235 883 tablet_service.cc:1505] Processing CreateTablet for tablet 3775a2e08d4e40bd9f57e4cda759d535 (DEFAULT_TABLE table=default.Table_37ad28b796a342f3a6e962ead289ec5b [id=a4d25738410b4a168aa0c76094daf69b]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:26.782739 850 leader_election.cc:343] T 2fac7b819e2642378e0d3f9230d0259c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:26.784505 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb824bd780524832b2b60f5618a39da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.783658 884 tablet_service.cc:1505] Processing CreateTablet for tablet f0bdaa8cfd764f1b83f47881353d26b6 (DEFAULT_TABLE table=default.Table_49beae553d03426aa395b7cdb6dfec9a [id=ab3328aa6f4148bfa2877cd4f0c6d324]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.785916 1167 tablet_bootstrap.cc:492] T 20e4178c16bd4f18b1c6bc969f6a0c6c P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:26.786558 1167 ts_tablet_manager.cc:1403] T 20e4178c16bd4f18b1c6bc969f6a0c6c P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.005s
W20251025 07:52:26.787127 847 leader_election.cc:343] T 576c51d9a22d45f6a3642a7296b5bf23 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:26.787271 847 leader_election.cc:304] T 576c51d9a22d45f6a3642a7296b5bf23 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:26.787823 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 1aeaad5c5d01478bb997254ac372d9a6
I20251025 07:52:26.787892 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 1483c40eee834f20a92e6b4cd89775b6
I20251025 07:52:26.788025 1268 raft_consensus.cc:493] T 299572b2dbe244918dcebc5ea5f3c45a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:26.788189 1218 raft_consensus.cc:2749] T 576c51d9a22d45f6a3642a7296b5bf23 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:26.788419 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3775a2e08d4e40bd9f57e4cda759d535. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.788461 1167 raft_consensus.cc:359] T 20e4178c16bd4f18b1c6bc969f6a0c6c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.788653 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 8c76d1b509f04ab1a10b83cff675c11e
I20251025 07:52:26.789443 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet cb824bd780524832b2b60f5618a39da5
I20251025 07:52:26.790223 1167 raft_consensus.cc:385] T 20e4178c16bd4f18b1c6bc969f6a0c6c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:26.790259 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet fab67b56eaa44b54886c17181d8c52b2
I20251025 07:52:26.790601 740 tablet_service.cc:1552] Processing DeleteTablet for tablet 1483c40eee834f20a92e6b4cd89775b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:26.790795 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c76d1b509f04ab1a10b83cff675c11e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:26.790885 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 1aeaad5c5d01478bb997254ac372d9a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:26.790990 755 tablet_service.cc:1552] Processing DeleteTablet for tablet cb824bd780524832b2b60f5618a39da5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:48830
W20251025 07:52:26.792205 983 leader_election.cc:343] T 3be8063aee794d2abcfb1d6e24209c06 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 3be8063aee794d2abcfb1d6e24209c06
I20251025 07:52:26.792312 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 1483c40eee834f20a92e6b4cd89775b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:26.792520 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c76d1b509f04ab1a10b83cff675c11e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:26.792644 1167 raft_consensus.cc:740] T 20e4178c16bd4f18b1c6bc969f6a0c6c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:26.792702 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet cb824bd780524832b2b60f5618a39da5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:26.792227 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet fab67b56eaa44b54886c17181d8c52b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:26.793418 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 1aeaad5c5d01478bb997254ac372d9a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:26.793318 1167 consensus_queue.cc:260] T 20e4178c16bd4f18b1c6bc969f6a0c6c P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:26.794332 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b772025852794a49b072f37cccbd3291 consensus state has no leader
I20251025 07:52:26.794250 1268 raft_consensus.cc:515] T 299572b2dbe244918dcebc5ea5f3c45a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.794389 755 tablet_service.cc:1552] Processing DeleteTablet for tablet fab67b56eaa44b54886c17181d8c52b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:48830
W20251025 07:52:26.794795 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 1483c40eee834f20a92e6b4cd89775b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1483c40eee834f20a92e6b4cd89775b6 already in progress: creating tablet
I20251025 07:52:26.794891 1268 leader_election.cc:290] T 299572b2dbe244918dcebc5ea5f3c45a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
W20251025 07:52:26.795073 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 8c76d1b509f04ab1a10b83cff675c11e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c76d1b509f04ab1a10b83cff675c11e already in progress: creating tablet
W20251025 07:52:26.795305 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 1aeaad5c5d01478bb997254ac372d9a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1aeaad5c5d01478bb997254ac372d9a6 already in progress: creating tablet
W20251025 07:52:26.795338 847 consensus_peers.cc:597] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382 -> Peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Couldn't send request to peer 23eeffd3de254ce18db5febfde971c6b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e84dbbd4a0bd4e4dbf1212fefbf00fd4. This is attempt 1: this message will repeat every 5th retry.
W20251025 07:52:26.795533 510 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet cb824bd780524832b2b60f5618a39da5 because the tablet was not found. No further retry: Not found: Tablet not found: cb824bd780524832b2b60f5618a39da5
I20251025 07:52:26.796694 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "299572b2dbe244918dcebc5ea5f3c45a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:26.798604 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet fab67b56eaa44b54886c17181d8c52b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fab67b56eaa44b54886c17181d8c52b2 already in progress: creating tablet
W20251025 07:52:26.798765 850 leader_election.cc:343] T 299572b2dbe244918dcebc5ea5f3c45a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:26.798957 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0bdaa8cfd764f1b83f47881353d26b6. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:26.799741 847 leader_election.cc:343] T 2fac7b819e2642378e0d3f9230d0259c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:26.800225 1167 ts_tablet_manager.cc:1434] T 20e4178c16bd4f18b1c6bc969f6a0c6c P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20251025 07:52:26.801015 847 leader_election.cc:304] T 2fac7b819e2642378e0d3f9230d0259c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:26.802268 1247 raft_consensus.cc:515] T 3be8063aee794d2abcfb1d6e24209c06 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.802521 1268 raft_consensus.cc:2749] T 2fac7b819e2642378e0d3f9230d0259c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:26.803222 894 tablet_service.cc:1505] Processing CreateTablet for tablet 8ff73ffc42434f74a4949f156771d5a4 (DEFAULT_TABLE table=default.Table_e244b2d2db74449aabd10a8b28e4afd2 [id=8bb9f6cbc77241f0b64c7ab15d1fe266]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.803736 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 20e4178c16bd4f18b1c6bc969f6a0c6c with delete_type TABLET_DATA_DELETED (replaced by tablet 50433881c0eb4aa5a389b6f5b4a4337a) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:26.803934 1167 tablet_bootstrap.cc:492] T 785d0ce85b234826ba8021fbd182bf3c P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:26.804032 1171 tablet_replica.cc:333] T 20e4178c16bd4f18b1c6bc969f6a0c6c P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:26.804210 1171 raft_consensus.cc:2243] T 20e4178c16bd4f18b1c6bc969f6a0c6c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:26.804419 1171 raft_consensus.cc:2272] T 20e4178c16bd4f18b1c6bc969f6a0c6c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:26.805070 1171 ts_tablet_manager.cc:1916] T 20e4178c16bd4f18b1c6bc969f6a0c6c P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:26.805274 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 713ae1dd444544bdabdd4a771e109d90 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:26.807168 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_133ad13fd9b340ffbab4f64a48f0474c" } new_table_name: "default.Table_e1d698cd782e4d58b29e9cc7d1f56703" modify_external_catalogs: true
I20251025 07:52:26.807403 741 tablet_service.cc:1552] Processing DeleteTablet for tablet 3be8063aee794d2abcfb1d6e24209c06 with delete_type TABLET_DATA_DELETED (replaced by tablet 2947502815d742e89cacb474b7fbb249) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:26.808063 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 1483c40eee834f20a92e6b4cd89775b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:26.811884 1171 ts_tablet_manager.cc:1929] T 20e4178c16bd4f18b1c6bc969f6a0c6c P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:26.812073 1171 log.cc:1199] T 20e4178c16bd4f18b1c6bc969f6a0c6c P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/20e4178c16bd4f18b1c6bc969f6a0c6c
I20251025 07:52:26.812568 1171 ts_tablet_manager.cc:1950] T 20e4178c16bd4f18b1c6bc969f6a0c6c P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:26.812572 1247 leader_election.cc:290] T 3be8063aee794d2abcfb1d6e24209c06 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:26.807829 1029 tablet_service.cc:1505] Processing CreateTablet for tablet f0bdaa8cfd764f1b83f47881353d26b6 (DEFAULT_TABLE table=default.Table_49beae553d03426aa395b7cdb6dfec9a [id=ab3328aa6f4148bfa2877cd4f0c6d324]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.813102 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3be8063aee794d2abcfb1d6e24209c06" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:26.813234 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0bdaa8cfd764f1b83f47881353d26b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.813313 775 raft_consensus.cc:3060] T 3be8063aee794d2abcfb1d6e24209c06 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:26.814105 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 1aeaad5c5d01478bb997254ac372d9a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:26.814602 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c76d1b509f04ab1a10b83cff675c11e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:26.814759 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 3be8063aee794d2abcfb1d6e24209c06 with delete_type TABLET_DATA_DELETED (replaced by tablet 2947502815d742e89cacb474b7fbb249) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:26.815951 775 raft_consensus.cc:2468] T 3be8063aee794d2abcfb1d6e24209c06 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:26.816874 1171 tablet_replica.cc:333] T 3be8063aee794d2abcfb1d6e24209c06 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:26.817191 1172 ts_tablet_manager.cc:1929] T 89bfb7e2f051454aae0af5755a79ca58 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:26.817291 1171 raft_consensus.cc:2243] T 3be8063aee794d2abcfb1d6e24209c06 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:26.817548 1171 raft_consensus.cc:2272] T 3be8063aee794d2abcfb1d6e24209c06 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:26.817435 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_8a4e7e8d20264b62b1db8a111b1285b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:26.817822 890 tablet_service.cc:1552] Processing DeleteTablet for tablet cb824bd780524832b2b60f5618a39da5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:56590
W20251025 07:52:26.818066 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a4e7e8d20264b62b1db8a111b1285b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.818078 885 tablet_service.cc:1552] Processing DeleteTablet for tablet fab67b56eaa44b54886c17181d8c52b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:26.818470 1171 ts_tablet_manager.cc:1916] T 3be8063aee794d2abcfb1d6e24209c06 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:26.820760 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_5abdea4ed6414edb9d539f20c19af0c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:26.821281 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5abdea4ed6414edb9d539f20c19af0c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:26.823830 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet db99a1e68daf4e948a3d66ad276cc5f9 has no consensus state
I20251025 07:52:26.823945 1167 tablet_bootstrap.cc:654] T 785d0ce85b234826ba8021fbd182bf3c P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:26.824941 982 leader_election.cc:304] T 3be8063aee794d2abcfb1d6e24209c06 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:26.825663 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 20e4178c16bd4f18b1c6bc969f6a0c6c (table default.Table_c73137b8706c43b2865a93af89cd428c [id=a5c73cb1e5a349ca8675f4ba4cb8a82d]) successfully deleted
I20251025 07:52:26.825662 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "299572b2dbe244918dcebc5ea5f3c45a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:26.828089 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_a58a5fd6549343d5bb4c540a9dde185a" } new_table_name: "default.Table_6b937442c9614efeb8cc1562ac306174" modify_external_catalogs: true
W20251025 07:52:26.829849 983 leader_election.cc:343] T 3be8063aee794d2abcfb1d6e24209c06 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 3be8063aee794d2abcfb1d6e24209c06
I20251025 07:52:26.829478 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_f3cbba9b75614002b5464930ccab8c7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:26.830744 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3cbba9b75614002b5464930ccab8c7f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.830829 1172 log.cc:1199] T 89bfb7e2f051454aae0af5755a79ca58 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/89bfb7e2f051454aae0af5755a79ca58
I20251025 07:52:26.831540 1169 tablet_bootstrap.cc:492] T a02b627401a64e1583f9e6123ef1be6f P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:26.831627 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ff73ffc42434f74a4949f156771d5a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.831809 1169 ts_tablet_manager.cc:1403] T a02b627401a64e1583f9e6123ef1be6f P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.073s user 0.002s sys 0.002s
I20251025 07:52:26.831974 1171 ts_tablet_manager.cc:1929] T 3be8063aee794d2abcfb1d6e24209c06 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:26.831976 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet 20e4178c16bd4f18b1c6bc969f6a0c6c with delete_type TABLET_DATA_DELETED (replaced by tablet 50433881c0eb4aa5a389b6f5b4a4337a) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:26.832247 519 master_service.cc:954] ReplaceTablet: received request to replace tablet d0fd00baa02d48278afaa5b8c19182f0 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:26.832926 1169 raft_consensus.cc:359] T a02b627401a64e1583f9e6123ef1be6f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.833220 1169 raft_consensus.cc:385] T a02b627401a64e1583f9e6123ef1be6f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:26.833349 1169 raft_consensus.cc:740] T a02b627401a64e1583f9e6123ef1be6f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:26.833617 1169 consensus_queue.cc:260] T a02b627401a64e1583f9e6123ef1be6f P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.834733 1247 raft_consensus.cc:2804] T 3be8063aee794d2abcfb1d6e24209c06 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:26.833993 1167 tablet_bootstrap.cc:492] T 785d0ce85b234826ba8021fbd182bf3c P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:26.835067 1167 ts_tablet_manager.cc:1403] T 785d0ce85b234826ba8021fbd182bf3c P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.002s
I20251025 07:52:26.836071 1167 raft_consensus.cc:359] T 785d0ce85b234826ba8021fbd182bf3c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.836761 1170 ts_tablet_manager.cc:1929] T 70c3a36dda7b4919be20390f5797d09c P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:26.837239 1167 raft_consensus.cc:385] T 785d0ce85b234826ba8021fbd182bf3c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:26.835253 1171 log.cc:1199] T 3be8063aee794d2abcfb1d6e24209c06 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/3be8063aee794d2abcfb1d6e24209c06
I20251025 07:52:26.837924 1167 raft_consensus.cc:740] T 785d0ce85b234826ba8021fbd182bf3c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:26.838459 1171 ts_tablet_manager.cc:1950] T 3be8063aee794d2abcfb1d6e24209c06 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:26.839458 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 3be8063aee794d2abcfb1d6e24209c06 (table default.Table_c96e62972b3c4ee48b6c3ea90831a34f [id=e7a5b463914840c980a306b7d46ea871]) successfully deleted
I20251025 07:52:26.839911 1167 consensus_queue.cc:260] T 785d0ce85b234826ba8021fbd182bf3c P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.843575 1172 ts_tablet_manager.cc:1950] T 89bfb7e2f051454aae0af5755a79ca58 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:26.843954 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 67) took 1319 ms. Trace:
I20251025 07:52:26.844210 1172 rpcz_store.cc:276] 1025 07:52:25.524758 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:25.527474 (+ 2716us) service_pool.cc:225] Handling call
1025 07:52:26.705635 (+1178161us) ts_tablet_manager.cc:1151] Deleting tablet 89bfb7e2f051454aae0af5755a79ca58
1025 07:52:26.705648 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:26.773500 (+ 67852us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:26.807657 (+ 34157us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:26.843924 (+ 36267us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1177872}
I20251025 07:52:26.844672 1172 tablet_replica.cc:333] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:26.845249 1172 raft_consensus.cc:2243] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:26.846056 1172 pending_rounds.cc:70] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382: Trying to abort 1 pending ops.
W20251025 07:52:26.846514 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 627a10a0714f4be8bdbced503fd6388d consensus state has no leader
I20251025 07:52:26.848248 1167 ts_tablet_manager.cc:1434] T 785d0ce85b234826ba8021fbd182bf3c P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20251025 07:52:26.849876 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_77761ef857464942be15bf6e67127d6d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:26.850205 755 tablet_service.cc:1505] Processing CreateTablet for tablet 1d261e432cf5403ea162129706bc4d4d (DEFAULT_TABLE table=default.Table_590cc281d8b042b19ef12c7f6436752a [id=b1eee5c0243a48c4bbdecc0b4f38ab2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.850456 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 576c51d9a22d45f6a3642a7296b5bf23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:56590
W20251025 07:52:26.850575 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77761ef857464942be15bf6e67127d6d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.850630 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d261e432cf5403ea162129706bc4d4d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:26.851472 847 leader_election.cc:343] T 299572b2dbe244918dcebc5ea5f3c45a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:26.851936 847 leader_election.cc:304] T 299572b2dbe244918dcebc5ea5f3c45a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:26.851920 741 tablet_service.cc:1505] Processing CreateTablet for tablet 8956ad28b35d45f28c683ac055bba441 (DEFAULT_TABLE table=default.Table_3cf5eecb57c54611bdcc3dfca5fcdca1 [id=39a63c8186374189ad21fbf8492d5d2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.852216 1268 raft_consensus.cc:2749] T 299572b2dbe244918dcebc5ea5f3c45a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:26.852272 1167 tablet_bootstrap.cc:492] T 9cf88862e0934634b3dd172e6b40a0ba P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:26.852339 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8956ad28b35d45f28c683ac055bba441. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.855341 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 89bfb7e2f051454aae0af5755a79ca58 (table default.Table_5f94a1aa9fac44ae8a90d3554215d598 [id=d82b3e1317434cf08a0d51e8a9427224]) successfully deleted
I20251025 07:52:26.857698 523 master_service.cc:954] ReplaceTablet: received request to replace tablet 21e52928580c4762b0ae7152490ef955 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:26.857811 1172 pending_rounds.cc:77] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214607342381957120 op_type: NO_OP noop_request { }
I20251025 07:52:26.858700 1170 log.cc:1199] T 70c3a36dda7b4919be20390f5797d09c P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/70c3a36dda7b4919be20390f5797d09c
I20251025 07:52:26.860427 1170 ts_tablet_manager.cc:1950] T 70c3a36dda7b4919be20390f5797d09c P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:26.860484 889 tablet_service.cc:1505] Processing CreateTablet for tablet 8956ad28b35d45f28c683ac055bba441 (DEFAULT_TABLE table=default.Table_3cf5eecb57c54611bdcc3dfca5fcdca1 [id=39a63c8186374189ad21fbf8492d5d2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.862006 753 tablet_service.cc:1505] Processing CreateTablet for tablet 87268e27540c4ec58ce1e62df819f19d (DEFAULT_TABLE table=default.Table_b6f727a336084bbf8d7d9c25a95105ba [id=96b0e47c666c4d8f982741aea7b17ae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.862475 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87268e27540c4ec58ce1e62df819f19d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.863307 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8956ad28b35d45f28c683ac055bba441. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.863260 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 70c3a36dda7b4919be20390f5797d09c (table default.Table_42908396d76f4df8b63e3ce11dcc2dfb [id=5e424c1c1c7c470f8640c7a7a571a11d]) successfully deleted
I20251025 07:52:26.847718 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 8956ad28b35d45f28c683ac055bba441 (DEFAULT_TABLE table=default.Table_3cf5eecb57c54611bdcc3dfca5fcdca1 [id=39a63c8186374189ad21fbf8492d5d2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.863692 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8956ad28b35d45f28c683ac055bba441. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.861395 1020 tablet_service.cc:1505] Processing CreateTablet for tablet 1d261e432cf5403ea162129706bc4d4d (DEFAULT_TABLE table=default.Table_590cc281d8b042b19ef12c7f6436752a [id=b1eee5c0243a48c4bbdecc0b4f38ab2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.865218 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d261e432cf5403ea162129706bc4d4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.868463 522 master_service.cc:954] ReplaceTablet: received request to replace tablet d37758f606344127b3db74c87babe69c from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:26.871167 1172 raft_consensus.cc:2889] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 07:52:26.871697 1172 raft_consensus.cc:2272] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:26.872316 1172 ts_tablet_manager.cc:1916] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:26.862164 1169 ts_tablet_manager.cc:1434] T a02b627401a64e1583f9e6123ef1be6f P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.030s user 0.001s sys 0.001s
I20251025 07:52:26.873026 887 tablet_service.cc:1505] Processing CreateTablet for tablet 1d261e432cf5403ea162129706bc4d4d (DEFAULT_TABLE table=default.Table_590cc281d8b042b19ef12c7f6436752a [id=b1eee5c0243a48c4bbdecc0b4f38ab2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.873915 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 21e52928580c4762b0ae7152490ef955
I20251025 07:52:26.874606 739 tablet_service.cc:1552] Processing DeleteTablet for tablet 21e52928580c4762b0ae7152490ef955 with delete_type TABLET_DATA_DELETED (replaced by tablet 9d6a9937d10a4cc2813671dd1bfa1664) from {username='slave'} at 127.0.0.1:48830
W20251025 07:52:26.875272 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 21e52928580c4762b0ae7152490ef955 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21e52928580c4762b0ae7152490ef955 already in progress: creating tablet
I20251025 07:52:26.875671 1167 tablet_bootstrap.cc:654] T 9cf88862e0934634b3dd172e6b40a0ba P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:26.876319 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet d37758f606344127b3db74c87babe69c
I20251025 07:52:26.876724 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d261e432cf5403ea162129706bc4d4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.877456 753 tablet_service.cc:1552] Processing DeleteTablet for tablet d37758f606344127b3db74c87babe69c with delete_type TABLET_DATA_DELETED (replaced by tablet 07898860aa594a05a211fc89768cb76d) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:26.877604 522 catalog_manager.cc:6719] ReplaceTablet: tablet d37758f606344127b3db74c87babe69c deleted and replaced by tablet 07898860aa594a05a211fc89768cb76d
I20251025 07:52:26.878399 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet d37758f606344127b3db74c87babe69c with delete_type TABLET_DATA_DELETED (replaced by tablet 07898860aa594a05a211fc89768cb76d) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:26.879348 1169 tablet_bootstrap.cc:492] T 785d0ce85b234826ba8021fbd182bf3c P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
W20251025 07:52:26.880110 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet d37758f606344127b3db74c87babe69c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d37758f606344127b3db74c87babe69c already in progress: creating tablet
I20251025 07:52:26.880195 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_04074ebcb0224168862553e3cf731987" } new_table_name: "default.Table_45b5db2f6c964ec583835ac9ca079e81" modify_external_catalogs: true
I20251025 07:52:26.883541 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_808d3c7f83104ccbbda7ee2e03db42a2" } new_table_name: "default.Table_04c3923fa1624c52b34d968919e406ae" modify_external_catalogs: true
I20251025 07:52:26.885108 523 catalog_manager.cc:6719] ReplaceTablet: tablet 21e52928580c4762b0ae7152490ef955 deleted and replaced by tablet 9d6a9937d10a4cc2813671dd1bfa1664
I20251025 07:52:26.886024 886 tablet_service.cc:1552] Processing DeleteTablet for tablet 21e52928580c4762b0ae7152490ef955 with delete_type TABLET_DATA_DELETED (replaced by tablet 9d6a9937d10a4cc2813671dd1bfa1664) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:26.886490 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 21e52928580c4762b0ae7152490ef955 with delete_type TABLET_DATA_DELETED (replaced by tablet 9d6a9937d10a4cc2813671dd1bfa1664) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:26.886760 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet a02b627401a64e1583f9e6123ef1be6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:26.888408 1169 tablet_bootstrap.cc:654] T 785d0ce85b234826ba8021fbd182bf3c P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:26.888934 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_f6e21e14e6fb4649af45c7d10a5f993e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:26.889528 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f6e21e14e6fb4649af45c7d10a5f993e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.892356 1167 tablet_bootstrap.cc:492] T 9cf88862e0934634b3dd172e6b40a0ba P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:26.892592 1167 ts_tablet_manager.cc:1403] T 9cf88862e0934634b3dd172e6b40a0ba P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.040s user 0.006s sys 0.002s
I20251025 07:52:26.893267 1172 ts_tablet_manager.cc:1929] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:26.893409 1167 raft_consensus.cc:359] T 9cf88862e0934634b3dd172e6b40a0ba P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:26.893610 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 896bd7870d87408e8adfdb7a78998111 has no consensus state
I20251025 07:52:26.894088 1167 raft_consensus.cc:385] T 9cf88862e0934634b3dd172e6b40a0ba P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:26.894248 1167 raft_consensus.cc:740] T 9cf88862e0934634b3dd172e6b40a0ba P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:26.894374 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 87268e27540c4ec58ce1e62df819f19d (DEFAULT_TABLE table=default.Table_b6f727a336084bbf8d7d9c25a95105ba [id=96b0e47c666c4d8f982741aea7b17ae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.894817 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87268e27540c4ec58ce1e62df819f19d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.894920 1167 consensus_queue.cc:260] T 9cf88862e0934634b3dd172e6b40a0ba P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.895989 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 785d0ce85b234826ba8021fbd182bf3c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:26.900969 1172 log.cc:1199] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/e84dbbd4a0bd4e4dbf1212fefbf00fd4
I20251025 07:52:26.901643 886 tablet_service.cc:1505] Processing CreateTablet for tablet 87268e27540c4ec58ce1e62df819f19d (DEFAULT_TABLE table=default.Table_b6f727a336084bbf8d7d9c25a95105ba [id=96b0e47c666c4d8f982741aea7b17ae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.902042 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87268e27540c4ec58ce1e62df819f19d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.902668 1024 tablet_service.cc:1505] Processing CreateTablet for tablet be5ab0db6ac34c2c9a370445e251eb46 (DEFAULT_TABLE table=default.Table_8faecb0cb37949a39f4ef5d9d2740c3d [id=f03dd08b39184812884db77faf4bb39e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.903071 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be5ab0db6ac34c2c9a370445e251eb46. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:26.907035 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b772025852794a49b072f37cccbd3291 consensus state has no leader
I20251025 07:52:26.900795 889 tablet_service.cc:1552] Processing DeleteTablet for tablet d37758f606344127b3db74c87babe69c with delete_type TABLET_DATA_DELETED (replaced by tablet 07898860aa594a05a211fc89768cb76d) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:26.911422 1172 ts_tablet_manager.cc:1950] T e84dbbd4a0bd4e4dbf1212fefbf00fd4 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:26.913647 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_208cd3f3c07744ed92b1260ebb401310"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:26.914052 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 68) took 1377 ms. Trace:
W20251025 07:52:26.914275 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_208cd3f3c07744ed92b1260ebb401310 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.914486 1172 rpcz_store.cc:276] 1025 07:52:25.536391 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:25.538784 (+ 2393us) service_pool.cc:225] Handling call
1025 07:52:26.844605 (+1305821us) ts_tablet_manager.cc:1151] Deleting tablet e84dbbd4a0bd4e4dbf1212fefbf00fd4
1025 07:52:26.844613 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:26.876504 (+ 31891us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:26.893125 (+ 16621us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:26.914006 (+ 20881us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1305582}
I20251025 07:52:26.915570 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_490b6bd20f694746afabf7be7f74fbc0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:26.916457 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_490b6bd20f694746afabf7be7f74fbc0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.917130 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 75) took 1243 ms. Trace:
I20251025 07:52:26.917302 1172 rpcz_store.cc:276] 1025 07:52:25.673417 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:25.673628 (+ 211us) service_pool.cc:225] Handling call
1025 07:52:26.916971 (+1243343us) ts_tablet_manager.cc:1151] Deleting tablet d0fd00baa02d48278afaa5b8c19182f0
1025 07:52:26.916980 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:26.917088 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1243108}
I20251025 07:52:26.917886 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 76) took 1244 ms. Trace:
I20251025 07:52:26.918051 1172 rpcz_store.cc:276] 1025 07:52:25.673577 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:25.673896 (+ 319us) service_pool.cc:225] Handling call
1025 07:52:26.917733 (+1243837us) ts_tablet_manager.cc:1151] Deleting tablet f51ec427dd9349a48eb221db04cf242b
1025 07:52:26.917742 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:26.917836 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1243643}
I20251025 07:52:26.918290 1170 tablet_replica.cc:333] T 5680e1adaf3f4b7f8c5e05523dd0950a P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:26.918525 1170 raft_consensus.cc:2243] T 5680e1adaf3f4b7f8c5e05523dd0950a P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:26.918818 1170 pending_rounds.cc:70] T 5680e1adaf3f4b7f8c5e05523dd0950a P db42be7ad3c741a0aa427f27e245993b: Trying to abort 1 pending ops.
I20251025 07:52:26.918920 1170 pending_rounds.cc:77] T 5680e1adaf3f4b7f8c5e05523dd0950a P db42be7ad3c741a0aa427f27e245993b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214607344166817792 op_type: NO_OP noop_request { }
I20251025 07:52:26.919072 1170 raft_consensus.cc:2889] T 5680e1adaf3f4b7f8c5e05523dd0950a P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 07:52:26.919209 1170 raft_consensus.cc:2272] T 5680e1adaf3f4b7f8c5e05523dd0950a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:26.920011 1170 ts_tablet_manager.cc:1916] T 5680e1adaf3f4b7f8c5e05523dd0950a P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:26.920488 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 91) took 1056 ms. Trace:
I20251025 07:52:26.920650 1172 rpcz_store.cc:276] 1025 07:52:25.864256 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:25.864313 (+ 57us) service_pool.cc:225] Handling call
1025 07:52:26.920326 (+1056013us) ts_tablet_manager.cc:1151] Deleting tablet 79bc2678df8042e3994b0471afa2c304
1025 07:52:26.920334 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:26.920453 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1055813}
I20251025 07:52:26.917019 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet e84dbbd4a0bd4e4dbf1212fefbf00fd4 (table default.Table_826ad7d517d34a33a3759e733302ce66 [id=b6e173967bad47e6b306627221b80f94]) successfully deleted
W20251025 07:52:26.923596 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet d0fd00baa02d48278afaa5b8c19182f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0fd00baa02d48278afaa5b8c19182f0 already in progress: creating tablet
W20251025 07:52:26.923887 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet f51ec427dd9349a48eb221db04cf242b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f51ec427dd9349a48eb221db04cf242b already in progress: creating tablet
I20251025 07:52:26.917574 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_517752d265854c7cbd706e2742536620"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:26.918290 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 2784bbfb97d5466f9ec856add7d0b29a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2784bbfb97d5466f9ec856add7d0b29a already in progress: creating tablet
W20251025 07:52:26.925671 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_517752d265854c7cbd706e2742536620 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.927115 886 tablet_service.cc:1505] Processing CreateTablet for tablet be5ab0db6ac34c2c9a370445e251eb46 (DEFAULT_TABLE table=default.Table_8faecb0cb37949a39f4ef5d9d2740c3d [id=f03dd08b39184812884db77faf4bb39e]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:26.927558 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 79bc2678df8042e3994b0471afa2c304 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79bc2678df8042e3994b0471afa2c304 already in progress: creating tablet
I20251025 07:52:26.927803 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be5ab0db6ac34c2c9a370445e251eb46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.928828 887 tablet_service.cc:1505] Processing CreateTablet for tablet f3001314b56345f5a70314777cae0125 (DEFAULT_TABLE table=default.Table_98454703d7234750b3cb2aab947ef63e [id=0a62837e6dda43879cc075d289326c5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.929298 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3001314b56345f5a70314777cae0125. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.929236 1021 tablet_service.cc:1505] Processing CreateTablet for tablet f3001314b56345f5a70314777cae0125 (DEFAULT_TABLE table=default.Table_98454703d7234750b3cb2aab947ef63e [id=0a62837e6dda43879cc075d289326c5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.929836 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3001314b56345f5a70314777cae0125. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.933524 889 tablet_service.cc:1505] Processing CreateTablet for tablet 71432a6848944a0da06ef9b81f160dc4 (DEFAULT_TABLE table=default.Table_79969e4ace6f4227b2748b59136027d6 [id=a1135611169142ba9ef987e3e9f95ce8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.932585 1169 tablet_bootstrap.cc:492] T 785d0ce85b234826ba8021fbd182bf3c P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:26.933818 1169 ts_tablet_manager.cc:1403] T 785d0ce85b234826ba8021fbd182bf3c P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.055s user 0.001s sys 0.002s
I20251025 07:52:26.933977 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71432a6848944a0da06ef9b81f160dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.934020 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 71432a6848944a0da06ef9b81f160dc4 (DEFAULT_TABLE table=default.Table_79969e4ace6f4227b2748b59136027d6 [id=a1135611169142ba9ef987e3e9f95ce8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.935000 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71432a6848944a0da06ef9b81f160dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.935274 1169 raft_consensus.cc:359] T 785d0ce85b234826ba8021fbd182bf3c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.935596 1169 raft_consensus.cc:385] T 785d0ce85b234826ba8021fbd182bf3c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:26.935743 1169 raft_consensus.cc:740] T 785d0ce85b234826ba8021fbd182bf3c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:26.936094 1169 consensus_queue.cc:260] T 785d0ce85b234826ba8021fbd182bf3c P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.936937 1169 ts_tablet_manager.cc:1434] T 785d0ce85b234826ba8021fbd182bf3c P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251025 07:52:26.937399 1169 tablet_bootstrap.cc:492] T 9cf88862e0934634b3dd172e6b40a0ba P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:26.940728 1268 raft_consensus.cc:493] T a02b627401a64e1583f9e6123ef1be6f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:26.940798 1171 tablet_replica.cc:333] T 785d0ce85b234826ba8021fbd182bf3c P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:26.940893 1268 raft_consensus.cc:515] T a02b627401a64e1583f9e6123ef1be6f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.941329 1167 ts_tablet_manager.cc:1434] T 9cf88862e0934634b3dd172e6b40a0ba P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.048s user 0.002s sys 0.000s
I20251025 07:52:26.941668 1268 leader_election.cc:290] T a02b627401a64e1583f9e6123ef1be6f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:26.942235 1167 tablet_bootstrap.cc:492] T 2fac7b819e2642378e0d3f9230d0259c P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:26.942364 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a02b627401a64e1583f9e6123ef1be6f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:26.942603 1048 raft_consensus.cc:2468] T a02b627401a64e1583f9e6123ef1be6f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:26.943344 850 leader_election.cc:304] T a02b627401a64e1583f9e6123ef1be6f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:26.943918 1268 raft_consensus.cc:2804] T a02b627401a64e1583f9e6123ef1be6f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:26.944036 1268 raft_consensus.cc:493] T a02b627401a64e1583f9e6123ef1be6f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:26.944131 1268 raft_consensus.cc:3060] T a02b627401a64e1583f9e6123ef1be6f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:26.944415 1171 raft_consensus.cc:2243] T 785d0ce85b234826ba8021fbd182bf3c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:26.944617 1171 raft_consensus.cc:2272] T 785d0ce85b234826ba8021fbd182bf3c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:26.945371 1171 ts_tablet_manager.cc:1916] T 785d0ce85b234826ba8021fbd182bf3c P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:26.945979 1020 tablet_service.cc:1505] Processing CreateTablet for tablet c4610b0c62e741eca28c21f63cc3de81 (DEFAULT_TABLE table=default.Table_c9bc68f9b5c54a33b854f9ee8a1ad311 [id=d8f616c04cc54b86af0af3cc073c5e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.946393 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4610b0c62e741eca28c21f63cc3de81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.946997 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 9cf88862e0934634b3dd172e6b40a0ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:26.948590 755 tablet_service.cc:1505] Processing CreateTablet for tablet 71432a6848944a0da06ef9b81f160dc4 (DEFAULT_TABLE table=default.Table_79969e4ace6f4227b2748b59136027d6 [id=a1135611169142ba9ef987e3e9f95ce8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.949235 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71432a6848944a0da06ef9b81f160dc4. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:26.949246 847 leader_election.cc:343] T a02b627401a64e1583f9e6123ef1be6f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: a02b627401a64e1583f9e6123ef1be6f
I20251025 07:52:26.949576 1247 raft_consensus.cc:697] T 3be8063aee794d2abcfb1d6e24209c06 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:26.949891 884 tablet_service.cc:1505] Processing CreateTablet for tablet c4610b0c62e741eca28c21f63cc3de81 (DEFAULT_TABLE table=default.Table_c9bc68f9b5c54a33b854f9ee8a1ad311 [id=d8f616c04cc54b86af0af3cc073c5e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.949860 1247 consensus_queue.cc:237] T 3be8063aee794d2abcfb1d6e24209c06 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.950452 753 tablet_service.cc:1505] Processing CreateTablet for tablet f3001314b56345f5a70314777cae0125 (DEFAULT_TABLE table=default.Table_98454703d7234750b3cb2aab947ef63e [id=0a62837e6dda43879cc075d289326c5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.950939 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3001314b56345f5a70314777cae0125. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.951057 1169 tablet_bootstrap.cc:654] T 9cf88862e0934634b3dd172e6b40a0ba P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:26.951208 1268 raft_consensus.cc:515] T a02b627401a64e1583f9e6123ef1be6f P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:26.952219 739 tablet_service.cc:1505] Processing CreateTablet for tablet c4610b0c62e741eca28c21f63cc3de81 (DEFAULT_TABLE table=default.Table_c9bc68f9b5c54a33b854f9ee8a1ad311 [id=d8f616c04cc54b86af0af3cc073c5e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.952636 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4610b0c62e741eca28c21f63cc3de81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.952602 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a02b627401a64e1583f9e6123ef1be6f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:26.952766 1048 raft_consensus.cc:3060] T a02b627401a64e1583f9e6123ef1be6f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:26.955097 750 tablet_service.cc:1505] Processing CreateTablet for tablet db99a1e68daf4e948a3d66ad276cc5f9 (DEFAULT_TABLE table=default.Table_e1d698cd782e4d58b29e9cc7d1f56703 [id=e8dcb9f9f6554f5ebcdfee63c6259f03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.955245 1167 tablet_bootstrap.cc:654] T 2fac7b819e2642378e0d3f9230d0259c P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:26.955528 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db99a1e68daf4e948a3d66ad276cc5f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.955649 1048 raft_consensus.cc:2468] T a02b627401a64e1583f9e6123ef1be6f P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:26.956038 850 leader_election.cc:304] T a02b627401a64e1583f9e6123ef1be6f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:26.956327 1218 raft_consensus.cc:2804] T a02b627401a64e1583f9e6123ef1be6f P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:26.956624 1024 tablet_service.cc:1505] Processing CreateTablet for tablet db99a1e68daf4e948a3d66ad276cc5f9 (DEFAULT_TABLE table=default.Table_e1d698cd782e4d58b29e9cc7d1f56703 [id=e8dcb9f9f6554f5ebcdfee63c6259f03]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:26.956966 847 leader_election.cc:343] T a02b627401a64e1583f9e6123ef1be6f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: a02b627401a64e1583f9e6123ef1be6f
I20251025 07:52:26.957305 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db99a1e68daf4e948a3d66ad276cc5f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.957383 894 tablet_service.cc:1505] Processing CreateTablet for tablet db99a1e68daf4e948a3d66ad276cc5f9 (DEFAULT_TABLE table=default.Table_e1d698cd782e4d58b29e9cc7d1f56703 [id=e8dcb9f9f6554f5ebcdfee63c6259f03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.958096 883 tablet_service.cc:1505] Processing CreateTablet for tablet c59f6e6eb9594a8e83f109a09b034045 (DEFAULT_TABLE table=default.Table_acca538f4b584f3ea4a55921a708d5a9 [id=71ac1bb83b444af2aede5644bf5619a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.958395 754 tablet_service.cc:1505] Processing CreateTablet for tablet be5ab0db6ac34c2c9a370445e251eb46 (DEFAULT_TABLE table=default.Table_8faecb0cb37949a39f4ef5d9d2740c3d [id=f03dd08b39184812884db77faf4bb39e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.958491 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_30224020f4384b0283e26621e412f889"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:26.959295 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be5ab0db6ac34c2c9a370445e251eb46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.959375 1022 tablet_service.cc:1505] Processing CreateTablet for tablet c59f6e6eb9594a8e83f109a09b034045 (DEFAULT_TABLE table=default.Table_acca538f4b584f3ea4a55921a708d5a9 [id=71ac1bb83b444af2aede5644bf5619a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.959820 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c59f6e6eb9594a8e83f109a09b034045. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.959854 1268 leader_election.cc:290] T a02b627401a64e1583f9e6123ef1be6f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:26.964864 1218 raft_consensus.cc:697] T a02b627401a64e1583f9e6123ef1be6f P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:26.965229 1218 consensus_queue.cc:237] T a02b627401a64e1583f9e6123ef1be6f P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:26.965751 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30224020f4384b0283e26621e412f889 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:26.969615 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4610b0c62e741eca28c21f63cc3de81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.970389 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db99a1e68daf4e948a3d66ad276cc5f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.974332 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c59f6e6eb9594a8e83f109a09b034045. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:26.975186 1170 ts_tablet_manager.cc:1929] T 5680e1adaf3f4b7f8c5e05523dd0950a P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:26.975327 1170 log.cc:1199] T 5680e1adaf3f4b7f8c5e05523dd0950a P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/5680e1adaf3f4b7f8c5e05523dd0950a
I20251025 07:52:26.975737 1170 ts_tablet_manager.cc:1950] T 5680e1adaf3f4b7f8c5e05523dd0950a P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:26.976753 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 5680e1adaf3f4b7f8c5e05523dd0950a (table default.Table_31d7f6f73d6e4123b1ff1df86ff62205 [id=cb037f61ee0b4b6cb8fb613263a278fa]) successfully deleted
I20251025 07:52:26.979807 743 tablet_service.cc:1505] Processing CreateTablet for tablet c59f6e6eb9594a8e83f109a09b034045 (DEFAULT_TABLE table=default.Table_acca538f4b584f3ea4a55921a708d5a9 [id=71ac1bb83b444af2aede5644bf5619a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:26.980195 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c59f6e6eb9594a8e83f109a09b034045. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:26.982750 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 198db9a45257435f86df2605992ccb34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 198db9a45257435f86df2605992ccb34 already in progress: creating tablet
W20251025 07:52:26.985396 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 1483c40eee834f20a92e6b4cd89775b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1483c40eee834f20a92e6b4cd89775b6 already in progress: creating tablet
W20251025 07:52:26.985968 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 8c76d1b509f04ab1a10b83cff675c11e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c76d1b509f04ab1a10b83cff675c11e already in progress: creating tablet
W20251025 07:52:26.986531 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet cb824bd780524832b2b60f5618a39da5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb824bd780524832b2b60f5618a39da5 already in progress: creating tablet
I20251025 07:52:26.989892 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 92) took 1073 ms. Trace:
I20251025 07:52:26.990199 1172 rpcz_store.cc:276] 1025 07:52:25.916626 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:25.916692 (+ 66us) service_pool.cc:225] Handling call
1025 07:52:26.940631 (+1023939us) ts_tablet_manager.cc:1151] Deleting tablet dd898450f2604987b08639cf87d0e402
1025 07:52:26.940642 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:26.989847 (+ 49205us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1023724}
W20251025 07:52:26.990617 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet fab67b56eaa44b54886c17181d8c52b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fab67b56eaa44b54886c17181d8c52b2 already in progress: creating tablet
I20251025 07:52:26.991187 1171 ts_tablet_manager.cc:1929] T 785d0ce85b234826ba8021fbd182bf3c P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:26.991379 1171 log.cc:1199] T 785d0ce85b234826ba8021fbd182bf3c P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/785d0ce85b234826ba8021fbd182bf3c
W20251025 07:52:26.991674 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 1aeaad5c5d01478bb997254ac372d9a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1aeaad5c5d01478bb997254ac372d9a6 already in progress: creating tablet
I20251025 07:52:26.991860 1171 ts_tablet_manager.cc:1950] T 785d0ce85b234826ba8021fbd182bf3c P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:26.992018 1170 tablet_replica.cc:333] T 3be8063aee794d2abcfb1d6e24209c06 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:26.992934 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 785d0ce85b234826ba8021fbd182bf3c (table default.Table_e0e89ced2ea94c139ed635849e78cb35 [id=807c428924b74fde847c2becad78d4ef]) successfully deleted
I20251025 07:52:26.993280 1171 tablet_replica.cc:333] T 9cf88862e0934634b3dd172e6b40a0ba P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:26.993521 1171 raft_consensus.cc:2243] T 9cf88862e0934634b3dd172e6b40a0ba P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:26.993765 1171 raft_consensus.cc:2272] T 9cf88862e0934634b3dd172e6b40a0ba P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:26.994585 1171 ts_tablet_manager.cc:1916] T 9cf88862e0934634b3dd172e6b40a0ba P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:26.995255 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_45b1ec3be7e9460296fe5b819beaa6ea" } modify_external_catalogs: true
I20251025 07:52:26.995427 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_45b1ec3be7e9460296fe5b819beaa6ea" } modify_external_catalogs: true
W20251025 07:52:26.995733 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet dd898450f2604987b08639cf87d0e402 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dd898450f2604987b08639cf87d0e402 already in progress: creating tablet
I20251025 07:52:26.995808 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_e61ae9065c604ad8b93af8146748cec9" } modify_external_catalogs: true
I20251025 07:52:26.995944 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_e61ae9065c604ad8b93af8146748cec9" } modify_external_catalogs: true
I20251025 07:52:26.996763 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_c73137b8706c43b2865a93af89cd428c" } new_table_name: "default.Table_b42187d20a7e4f8783a8d25acb249350" modify_external_catalogs: true
I20251025 07:52:26.995939 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 90) took 1147 ms. Trace:
I20251025 07:52:26.997561 1172 rpcz_store.cc:276] 1025 07:52:25.848323 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:25.951756 (+103433us) service_pool.cc:225] Handling call
1025 07:52:26.995788 (+1044032us) ts_tablet_manager.cc:1151] Deleting tablet 567a1f5c216a4a519249da66a2a03ea4
1025 07:52:26.995796 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:26.995896 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1043825}
W20251025 07:52:26.997963 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 567a1f5c216a4a519249da66a2a03ea4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 567a1f5c216a4a519249da66a2a03ea4 already in progress: creating tablet
I20251025 07:52:26.998385 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 89) took 1150 ms. Trace:
I20251025 07:52:26.998507 1172 rpcz_store.cc:276] 1025 07:52:25.848145 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:25.952538 (+104393us) service_pool.cc:225] Handling call
1025 07:52:26.998248 (+1045710us) ts_tablet_manager.cc:1151] Deleting tablet 47284f96928749e6a5201103a8ee5f9f
1025 07:52:26.998257 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:26.998355 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1045512}
W20251025 07:52:26.998875 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 47284f96928749e6a5201103a8ee5f9f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47284f96928749e6a5201103a8ee5f9f already in progress: creating tablet
I20251025 07:52:26.999207 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_f3cbba9b75614002b5464930ccab8c7f" } modify_external_catalogs: true
I20251025 07:52:26.999418 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_f3cbba9b75614002b5464930ccab8c7f" } modify_external_catalogs: true
I20251025 07:52:26.999492 1172 tablet_replica.cc:333] T b946855345bc473aadd00cc4db5b7cd8 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:26.999673 1172 raft_consensus.cc:2243] T b946855345bc473aadd00cc4db5b7cd8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:26.999827 1172 raft_consensus.cc:2272] T b946855345bc473aadd00cc4db5b7cd8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:27.001708 1166 tablet_bootstrap.cc:492] T 3d5e0dd14e474fe09993a03851f8b1f0 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:27.001878 1166 ts_tablet_manager.cc:1403] T 3d5e0dd14e474fe09993a03851f8b1f0 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.284s user 0.024s sys 0.044s
W20251025 07:52:27.005681 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 50433881c0eb4aa5a389b6f5b4a4337a consensus state has no leader
I20251025 07:52:27.005676 1166 raft_consensus.cc:359] T 3d5e0dd14e474fe09993a03851f8b1f0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.005882 1166 raft_consensus.cc:385] T 3d5e0dd14e474fe09993a03851f8b1f0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:27.005967 1166 raft_consensus.cc:740] T 3d5e0dd14e474fe09993a03851f8b1f0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:27.006423 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_b1c43f4e0ede4adcba8b010f04bdb72a" } modify_external_catalogs: true
I20251025 07:52:27.006642 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_b1c43f4e0ede4adcba8b010f04bdb72a" } modify_external_catalogs: true
I20251025 07:52:27.010103 743 tablet_service.cc:1505] Processing CreateTablet for tablet 9979cae195e849908677d78c6c33be71 (DEFAULT_TABLE table=default.Table_208cd3f3c07744ed92b1260ebb401310 [id=d6c5598979e84e36a3bf1996f37f9c28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.010501 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9979cae195e849908677d78c6c33be71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.010540 753 tablet_service.cc:1505] Processing CreateTablet for tablet 896bd7870d87408e8adfdb7a78998111 (DEFAULT_TABLE table=default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.011106 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 896bd7870d87408e8adfdb7a78998111. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.013718 1020 tablet_service.cc:1505] Processing CreateTablet for tablet f170424be2f643e7af499834955472b0 (DEFAULT_TABLE table=default.Table_f3cbba9b75614002b5464930ccab8c7f [id=4f5379fa0372460ea325339f39289a10]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.011487 1172 ts_tablet_manager.cc:1916] T b946855345bc473aadd00cc4db5b7cd8 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:27.015024 739 tablet_service.cc:1505] Processing CreateTablet for tablet f170424be2f643e7af499834955472b0 (DEFAULT_TABLE table=default.Table_f3cbba9b75614002b5464930ccab8c7f [id=4f5379fa0372460ea325339f39289a10]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.015388 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f170424be2f643e7af499834955472b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.018352 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_9b8e648894e14f8595796e4698407495" } modify_external_catalogs: true
I20251025 07:52:27.018590 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_9b8e648894e14f8595796e4698407495" } modify_external_catalogs: true
I20251025 07:52:27.021325 883 tablet_service.cc:1505] Processing CreateTablet for tablet f170424be2f643e7af499834955472b0 (DEFAULT_TABLE table=default.Table_f3cbba9b75614002b5464930ccab8c7f [id=4f5379fa0372460ea325339f39289a10]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.022313 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f170424be2f643e7af499834955472b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.023274 1169 tablet_bootstrap.cc:492] T 9cf88862e0934634b3dd172e6b40a0ba P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:27.023584 1169 ts_tablet_manager.cc:1403] T 9cf88862e0934634b3dd172e6b40a0ba P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.086s user 0.005s sys 0.000s
I20251025 07:52:27.024380 1169 raft_consensus.cc:359] T 9cf88862e0934634b3dd172e6b40a0ba P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.025080 1169 raft_consensus.cc:385] T 9cf88862e0934634b3dd172e6b40a0ba P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:27.025224 1169 raft_consensus.cc:740] T 9cf88862e0934634b3dd172e6b40a0ba P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:27.025575 884 tablet_service.cc:1505] Processing CreateTablet for tablet 9979cae195e849908677d78c6c33be71 (DEFAULT_TABLE table=default.Table_208cd3f3c07744ed92b1260ebb401310 [id=d6c5598979e84e36a3bf1996f37f9c28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.025520 1169 consensus_queue.cc:260] T 9cf88862e0934634b3dd172e6b40a0ba P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.025959 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9979cae195e849908677d78c6c33be71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.026332 1169 ts_tablet_manager.cc:1434] T 9cf88862e0934634b3dd172e6b40a0ba P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251025 07:52:27.026690 1169 tablet_bootstrap.cc:492] T 2fac7b819e2642378e0d3f9230d0259c P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:27.028647 1170 raft_consensus.cc:2243] T 3be8063aee794d2abcfb1d6e24209c06 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:27.029070 1170 pending_rounds.cc:70] T 3be8063aee794d2abcfb1d6e24209c06 P db42be7ad3c741a0aa427f27e245993b: Trying to abort 1 pending ops.
I20251025 07:52:27.029158 1170 pending_rounds.cc:77] T 3be8063aee794d2abcfb1d6e24209c06 P db42be7ad3c741a0aa427f27e245993b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214607347509510144 op_type: NO_OP noop_request { }
I20251025 07:52:27.029313 1170 raft_consensus.cc:2889] T 3be8063aee794d2abcfb1d6e24209c06 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 07:52:27.029422 1170 raft_consensus.cc:2272] T 3be8063aee794d2abcfb1d6e24209c06 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:27.032263 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 896bd7870d87408e8adfdb7a78998111 (DEFAULT_TABLE table=default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.034476 523 master_service.cc:954] ReplaceTablet: received request to replace tablet 02e4c4d6ffb34cecaf17beb5ba197e81 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:27.034919 525 master_service.cc:954] ReplaceTablet: received request to replace tablet fab67b56eaa44b54886c17181d8c52b2 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:27.036371 1029 tablet_service.cc:1505] Processing CreateTablet for tablet d5e5e5bbcf9f43118eea502b69880af5 (DEFAULT_TABLE table=default.Table_5abdea4ed6414edb9d539f20c19af0c4 [id=4d9e99d6474b4d00bc66e26a33694e69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.036829 750 tablet_service.cc:1505] Processing CreateTablet for tablet b97e39aaac3048ed93dbfe873b757254 (DEFAULT_TABLE table=default.Table_8a4e7e8d20264b62b1db8a111b1285b8 [id=c8059ab6cec146efa631d06799d969a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.037597 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b97e39aaac3048ed93dbfe873b757254. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.037770 1167 tablet_bootstrap.cc:492] T 2fac7b819e2642378e0d3f9230d0259c P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:27.037962 1167 ts_tablet_manager.cc:1403] T 2fac7b819e2642378e0d3f9230d0259c P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.096s user 0.006s sys 0.004s
I20251025 07:52:27.038208 889 tablet_service.cc:1505] Processing CreateTablet for tablet c88c46e022ff41a58b02538f2151e260 (DEFAULT_TABLE table=default.Table_f6e21e14e6fb4649af45c7d10a5f993e [id=c5eda563853a4e15b846d063608d42ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.038573 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c88c46e022ff41a58b02538f2151e260. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.038547 1023 tablet_service.cc:1505] Processing CreateTablet for tablet c88c46e022ff41a58b02538f2151e260 (DEFAULT_TABLE table=default.Table_f6e21e14e6fb4649af45c7d10a5f993e [id=c5eda563853a4e15b846d063608d42ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.038553 1026 tablet_service.cc:1505] Processing CreateTablet for tablet b97e39aaac3048ed93dbfe873b757254 (DEFAULT_TABLE table=default.Table_8a4e7e8d20264b62b1db8a111b1285b8 [id=c8059ab6cec146efa631d06799d969a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.039279 755 tablet_service.cc:1505] Processing CreateTablet for tablet abf94c32ff2945119b710905c075d63c (DEFAULT_TABLE table=default.Table_4b63840ca3ac4296b23104bf46030bb4 [id=0010b5f16dd14d64b8c93d941803cc75]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.039244 1167 raft_consensus.cc:359] T 2fac7b819e2642378e0d3f9230d0259c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.039565 1167 raft_consensus.cc:385] T 2fac7b819e2642378e0d3f9230d0259c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:27.039700 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abf94c32ff2945119b710905c075d63c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.039819 1167 raft_consensus.cc:740] T 2fac7b819e2642378e0d3f9230d0259c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:27.040132 1167 consensus_queue.cc:260] T 2fac7b819e2642378e0d3f9230d0259c P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.044899 1167 ts_tablet_manager.cc:1434] T 2fac7b819e2642378e0d3f9230d0259c P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251025 07:52:27.045651 1021 tablet_service.cc:1505] Processing CreateTablet for tablet 9979cae195e849908677d78c6c33be71 (DEFAULT_TABLE table=default.Table_208cd3f3c07744ed92b1260ebb401310 [id=d6c5598979e84e36a3bf1996f37f9c28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.048025 1167 tablet_bootstrap.cc:492] T 576c51d9a22d45f6a3642a7296b5bf23 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:27.054538 743 tablet_service.cc:1505] Processing CreateTablet for tablet d5e5e5bbcf9f43118eea502b69880af5 (DEFAULT_TABLE table=default.Table_5abdea4ed6414edb9d539f20c19af0c4 [id=4d9e99d6474b4d00bc66e26a33694e69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.048293 1166 consensus_queue.cc:260] T 3d5e0dd14e474fe09993a03851f8b1f0 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.055034 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5e5e5bbcf9f43118eea502b69880af5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.059511 1166 ts_tablet_manager.cc:1434] T 3d5e0dd14e474fe09993a03851f8b1f0 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.058s user 0.003s sys 0.000s
I20251025 07:52:27.059484 754 tablet_service.cc:1505] Processing CreateTablet for tablet c88c46e022ff41a58b02538f2151e260 (DEFAULT_TABLE table=default.Table_f6e21e14e6fb4649af45c7d10a5f993e [id=c5eda563853a4e15b846d063608d42ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.061095 1167 tablet_bootstrap.cc:654] T 576c51d9a22d45f6a3642a7296b5bf23 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:27.061168 524 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 2fac7b819e2642378e0d3f9230d0259c (table default.Table_33b6fa5be1ab40fba35b3d5ef9ad9668 [id=6605e11e11804d408a6a4a9d403dde95]). Expected version 1 got 0
I20251025 07:52:27.062494 524 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 3d5e0dd14e474fe09993a03851f8b1f0 (table default.Table_028b24c9775b49ea8adf1a62fe9df71d [id=1b5db2f5053c4cb3a3b40eb109d5d5d9]). Expected version 1 got 0
I20251025 07:52:27.063625 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_30224020f4384b0283e26621e412f889" } new_table_name: "default.Table_3f081b7bb4e1420c8121ea4115439f3d" modify_external_catalogs: true
I20251025 07:52:27.063515 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c88c46e022ff41a58b02538f2151e260. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.064113 1166 tablet_bootstrap.cc:492] T 33c608beb4ef4a859a135b11e07c81a4 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:27.064193 1170 ts_tablet_manager.cc:1916] T 3be8063aee794d2abcfb1d6e24209c06 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:27.065397 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_5abdea4ed6414edb9d539f20c19af0c4" } new_table_name: "default.Table_c82c2dad12af41788d8959f7bd9f92b0" modify_external_catalogs: true
I20251025 07:52:27.066682 1166 tablet_bootstrap.cc:654] T 33c608beb4ef4a859a135b11e07c81a4 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:27.067792 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_39bd22f18d6540d3be94520671683032"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:27.069931 1218 raft_consensus.cc:493] T 4829ec41391145cc84327d60048d9b0c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:27.070117 1218 raft_consensus.cc:515] T 4829ec41391145cc84327d60048d9b0c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.070387 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f170424be2f643e7af499834955472b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.070665 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b97e39aaac3048ed93dbfe873b757254. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.070815 1218 leader_election.cc:290] T 4829ec41391145cc84327d60048d9b0c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:27.071880 883 tablet_service.cc:1505] Processing CreateTablet for tablet abf94c32ff2945119b710905c075d63c (DEFAULT_TABLE table=default.Table_4b63840ca3ac4296b23104bf46030bb4 [id=0010b5f16dd14d64b8c93d941803cc75]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.072702 1169 tablet_bootstrap.cc:654] T 2fac7b819e2642378e0d3f9230d0259c P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:27.074373 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4829ec41391145cc84327d60048d9b0c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:27.077066 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c88c46e022ff41a58b02538f2151e260. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:27.078687 850 leader_election.cc:343] T 4829ec41391145cc84327d60048d9b0c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:27.079087 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5e5e5bbcf9f43118eea502b69880af5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.081151 1171 ts_tablet_manager.cc:1929] T 9cf88862e0934634b3dd172e6b40a0ba P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:27.081908 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 896bd7870d87408e8adfdb7a78998111. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.082216 1171 log.cc:1199] T 9cf88862e0934634b3dd172e6b40a0ba P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/9cf88862e0934634b3dd172e6b40a0ba
I20251025 07:52:27.082176 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4829ec41391145cc84327d60048d9b0c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
W20251025 07:52:27.082751 847 leader_election.cc:343] T 4829ec41391145cc84327d60048d9b0c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:27.082895 847 leader_election.cc:304] T 4829ec41391145cc84327d60048d9b0c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:27.083338 1218 raft_consensus.cc:2749] T 4829ec41391145cc84327d60048d9b0c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:27.083678 1171 ts_tablet_manager.cc:1950] T 9cf88862e0934634b3dd172e6b40a0ba P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:27.084702 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 9cf88862e0934634b3dd172e6b40a0ba (table default.Table_1eb640b2196849a8b0136b390deea1c2 [id=ec85197b425f466ba7cd69e27caf6c93]) successfully deleted
I20251025 07:52:27.086117 1167 tablet_bootstrap.cc:492] T 576c51d9a22d45f6a3642a7296b5bf23 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:27.086648 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_aa90c6fe1e744173a6174066750dc583"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:27.088541 1167 ts_tablet_manager.cc:1403] T 576c51d9a22d45f6a3642a7296b5bf23 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.005s
W20251025 07:52:27.089114 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa90c6fe1e744173a6174066750dc583 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:27.089223 1170 ts_tablet_manager.cc:1929] T 3be8063aee794d2abcfb1d6e24209c06 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:27.090090 1170 log.cc:1199] T 3be8063aee794d2abcfb1d6e24209c06 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/3be8063aee794d2abcfb1d6e24209c06
W20251025 07:52:27.092113 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39bd22f18d6540d3be94520671683032 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:27.069689 1022 tablet_service.cc:1505] Processing CreateTablet for tablet abf94c32ff2945119b710905c075d63c (DEFAULT_TABLE table=default.Table_4b63840ca3ac4296b23104bf46030bb4 [id=0010b5f16dd14d64b8c93d941803cc75]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.096127 1167 raft_consensus.cc:359] T 576c51d9a22d45f6a3642a7296b5bf23 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.098661 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 0b21b7da1e2f47829c3f6cdaee689e8a
I20251025 07:52:27.098719 894 tablet_service.cc:1505] Processing CreateTablet for tablet 07898860aa594a05a211fc89768cb76d (DEFAULT_TABLE table=default.Table_92cbcd8230f249a8b2821f69a81b41f1 [id=08e53121e86d405dbbd02a4605695fec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.099610 755 tablet_service.cc:1505] Processing CreateTablet for tablet 07898860aa594a05a211fc89768cb76d (DEFAULT_TABLE table=default.Table_92cbcd8230f249a8b2821f69a81b41f1 [id=08e53121e86d405dbbd02a4605695fec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.097206 1167 raft_consensus.cc:385] T 576c51d9a22d45f6a3642a7296b5bf23 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:27.101099 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet f170424be2f643e7af499834955472b0
I20251025 07:52:27.100732 1167 raft_consensus.cc:740] T 576c51d9a22d45f6a3642a7296b5bf23 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:27.100890 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07898860aa594a05a211fc89768cb76d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.102785 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b21b7da1e2f47829c3f6cdaee689e8a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:27.103264 741 tablet_service.cc:1505] Processing CreateTablet for tablet 53198283718d4b45b30df65b5bde96a6 (DEFAULT_TABLE table=default.Table_77761ef857464942be15bf6e67127d6d [id=d9d0e8bb796a45a8b17336c04b12cd01]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:27.103381 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 0b21b7da1e2f47829c3f6cdaee689e8a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b21b7da1e2f47829c3f6cdaee689e8a already in progress: creating tablet
I20251025 07:52:27.103617 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53198283718d4b45b30df65b5bde96a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.102092 522 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 2fac7b819e2642378e0d3f9230d0259c (table default.Table_33b6fa5be1ab40fba35b3d5ef9ad9668 [id=6605e11e11804d408a6a4a9d403dde95]). Expected version 1 got 0
I20251025 07:52:27.105371 522 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 3d5e0dd14e474fe09993a03851f8b1f0 (table default.Table_028b24c9775b49ea8adf1a62fe9df71d [id=1b5db2f5053c4cb3a3b40eb109d5d5d9]). Expected version 1 got 0
I20251025 07:52:27.106092 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_400f06d166054eeb97890ad56453afb2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:27.106561 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b21b7da1e2f47829c3f6cdaee689e8a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:27.106986 890 tablet_service.cc:1505] Processing CreateTablet for tablet 9d6a9937d10a4cc2813671dd1bfa1664 (DEFAULT_TABLE table=default.Table_e7d32e91b9c64ce1b3fcff3d806e959f [id=bab869e5001c4d999daa22083fca128a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.106987 753 tablet_service.cc:1505] Processing CreateTablet for tablet 9d6a9937d10a4cc2813671dd1bfa1664 (DEFAULT_TABLE table=default.Table_e7d32e91b9c64ce1b3fcff3d806e959f [id=bab869e5001c4d999daa22083fca128a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.107390 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d6a9937d10a4cc2813671dd1bfa1664. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:27.107429 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_400f06d166054eeb97890ad56453afb2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:27.107427 892 tablet_service.cc:1552] Processing DeleteTablet for tablet f170424be2f643e7af499834955472b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:27.108952 1172 ts_tablet_manager.cc:1929] T b946855345bc473aadd00cc4db5b7cd8 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:27.109215 1172 log.cc:1199] T b946855345bc473aadd00cc4db5b7cd8 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/b946855345bc473aadd00cc4db5b7cd8
I20251025 07:52:27.110250 887 tablet_service.cc:1505] Processing CreateTablet for tablet 896bd7870d87408e8adfdb7a78998111 (DEFAULT_TABLE table=default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.108927 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_4bb5ba285e1c46f8b698e4915e37fab9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:27.110893 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4bb5ba285e1c46f8b698e4915e37fab9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:27.112321 1172 ts_tablet_manager.cc:1950] T b946855345bc473aadd00cc4db5b7cd8 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:27.113278 739 tablet_service.cc:1552] Processing DeleteTablet for tablet f170424be2f643e7af499834955472b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:27.095711 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9979cae195e849908677d78c6c33be71. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:27.113862 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet f170424be2f643e7af499834955472b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f170424be2f643e7af499834955472b0 already in progress: creating tablet
I20251025 07:52:27.115527 889 tablet_service.cc:1505] Processing CreateTablet for tablet 53198283718d4b45b30df65b5bde96a6 (DEFAULT_TABLE table=default.Table_77761ef857464942be15bf6e67127d6d [id=d9d0e8bb796a45a8b17336c04b12cd01]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.120638 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abf94c32ff2945119b710905c075d63c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.121031 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07898860aa594a05a211fc89768cb76d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.121330 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d6a9937d10a4cc2813671dd1bfa1664. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.123677 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abf94c32ff2945119b710905c075d63c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.124331 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 785d0ce85b234826ba8021fbd182bf3c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:27.124514 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 9cf88862e0934634b3dd172e6b40a0ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:27.124610 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b21b7da1e2f47829c3f6cdaee689e8a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:27.124851 1018 tablet_service.cc:1505] Processing CreateTablet for tablet 9d6a9937d10a4cc2813671dd1bfa1664 (DEFAULT_TABLE table=default.Table_e7d32e91b9c64ce1b3fcff3d806e959f [id=bab869e5001c4d999daa22083fca128a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.125528 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d6a9937d10a4cc2813671dd1bfa1664. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.127785 1023 tablet_service.cc:1505] Processing CreateTablet for tablet 07898860aa594a05a211fc89768cb76d (DEFAULT_TABLE table=default.Table_92cbcd8230f249a8b2821f69a81b41f1 [id=08e53121e86d405dbbd02a4605695fec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.128291 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07898860aa594a05a211fc89768cb76d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.132050 1167 consensus_queue.cc:260] T 576c51d9a22d45f6a3642a7296b5bf23 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.132696 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 53198283718d4b45b30df65b5bde96a6 (DEFAULT_TABLE table=default.Table_77761ef857464942be15bf6e67127d6d [id=d9d0e8bb796a45a8b17336c04b12cd01]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.133071 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53198283718d4b45b30df65b5bde96a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.118912 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 97) took 1128 ms. Trace:
I20251025 07:52:27.136246 1167 ts_tablet_manager.cc:1434] T 576c51d9a22d45f6a3642a7296b5bf23 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.047s user 0.002s sys 0.001s
I20251025 07:52:27.136574 1167 tablet_bootstrap.cc:492] T 4829ec41391145cc84327d60048d9b0c P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:27.138796 1167 tablet_bootstrap.cc:654] T 4829ec41391145cc84327d60048d9b0c P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:27.140846 1170 ts_tablet_manager.cc:1950] T 3be8063aee794d2abcfb1d6e24209c06 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:27.145653 1172 rpcz_store.cc:276] 1025 07:52:25.989560 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:25.989619 (+ 59us) service_pool.cc:225] Handling call
1025 07:52:26.999403 (+1009784us) ts_tablet_manager.cc:1151] Deleting tablet b946855345bc473aadd00cc4db5b7cd8
1025 07:52:26.999411 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:27.017484 (+ 18073us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:27.108739 (+ 91255us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:27.118258 (+ 9519us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1006829}
I20251025 07:52:27.146673 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 3be8063aee794d2abcfb1d6e24209c06 (table default.Table_c96e62972b3c4ee48b6c3ea90831a34f [id=e7a5b463914840c980a306b7d46ea871]) successfully deleted
I20251025 07:52:27.147478 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet f170424be2f643e7af499834955472b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:27.151788 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 3c3eb43209f24592820638fc25486b78
I20251025 07:52:27.152112 1166 tablet_bootstrap.cc:492] T 33c608beb4ef4a859a135b11e07c81a4 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:27.152280 1166 ts_tablet_manager.cc:1403] T 33c608beb4ef4a859a135b11e07c81a4 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.088s user 0.000s sys 0.011s
I20251025 07:52:27.153661 1169 tablet_bootstrap.cc:492] T 2fac7b819e2642378e0d3f9230d0259c P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:27.153767 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_e58048181b0d473d9195088bbef35387" } new_table_name: "default.Table_e3b368ac698c46f4a8c947d02f9a2680" modify_external_catalogs: true
I20251025 07:52:27.153913 1169 ts_tablet_manager.cc:1403] T 2fac7b819e2642378e0d3f9230d0259c P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.127s user 0.007s sys 0.005s
I20251025 07:52:27.154349 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 4829ec41391145cc84327d60048d9b0c
I20251025 07:52:27.154892 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_826ad7d517d34a33a3759e733302ce66" } new_table_name: "default.Table_11117cabb5ca4540a58aac25d02964b2" modify_external_catalogs: true
I20251025 07:52:27.155694 884 tablet_service.cc:1505] Processing CreateTablet for tablet d5e5e5bbcf9f43118eea502b69880af5 (DEFAULT_TABLE table=default.Table_5abdea4ed6414edb9d539f20c19af0c4 [id=4d9e99d6474b4d00bc66e26a33694e69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.156180 886 tablet_service.cc:1505] Processing CreateTablet for tablet b97e39aaac3048ed93dbfe873b757254 (DEFAULT_TABLE table=default.Table_8a4e7e8d20264b62b1db8a111b1285b8 [id=c8059ab6cec146efa631d06799d969a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.156363 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 19918a3c1f1d4ed6a78d5b4f0f4e38b3
I20251025 07:52:27.156772 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 106) took 1050 ms. Trace:
I20251025 07:52:27.156876 1172 rpcz_store.cc:276] 1025 07:52:26.097551 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:26.097849 (+ 298us) service_pool.cc:225] Handling call
1025 07:52:27.146873 (+1049024us) ts_tablet_manager.cc:1151] Deleting tablet 90764275825c4ffb8712cb883d37a551
1025 07:52:27.146896 (+ 23us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:27.148267 (+ 1371us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1029555}
I20251025 07:52:27.157070 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_56d474131b4047f49432e55467b20cdd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:27.157635 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56d474131b4047f49432e55467b20cdd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:27.157739 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 104) took 1060 ms. Trace:
I20251025 07:52:27.157825 1172 rpcz_store.cc:276] 1025 07:52:26.097219 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:26.104022 (+ 6803us) service_pool.cc:225] Handling call
1025 07:52:27.157267 (+1053245us) ts_tablet_manager.cc:1151] Deleting tablet 08da9d2687704725aacd11cd92c6a9f9
1025 07:52:27.157276 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:27.157363 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1036767}
I20251025 07:52:27.158413 1170 tablet_replica.cc:333] T 20e4178c16bd4f18b1c6bc969f6a0c6c P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:27.158742 1170 raft_consensus.cc:2243] T 20e4178c16bd4f18b1c6bc969f6a0c6c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:27.158881 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 896bd7870d87408e8adfdb7a78998111. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.159379 1170 raft_consensus.cc:2272] T 20e4178c16bd4f18b1c6bc969f6a0c6c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:27.159543 1166 raft_consensus.cc:359] T 33c608beb4ef4a859a135b11e07c81a4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:27.159664 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 08da9d2687704725aacd11cd92c6a9f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08da9d2687704725aacd11cd92c6a9f9 already in progress: creating tablet
I20251025 07:52:27.159719 1166 raft_consensus.cc:385] T 33c608beb4ef4a859a135b11e07c81a4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:27.159996 1166 raft_consensus.cc:740] T 33c608beb4ef4a859a135b11e07c81a4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:27.160269 1166 consensus_queue.cc:260] T 33c608beb4ef4a859a135b11e07c81a4 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.162526 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b97e39aaac3048ed93dbfe873b757254. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.163813 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c3eb43209f24592820638fc25486b78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:48830
W20251025 07:52:27.164618 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 3c3eb43209f24592820638fc25486b78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c3eb43209f24592820638fc25486b78 already in progress: creating tablet
I20251025 07:52:27.165494 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 19918a3c1f1d4ed6a78d5b4f0f4e38b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:48830
W20251025 07:52:27.166075 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 19918a3c1f1d4ed6a78d5b4f0f4e38b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19918a3c1f1d4ed6a78d5b4f0f4e38b3 already in progress: creating tablet
I20251025 07:52:27.167155 741 tablet_service.cc:1552] Processing DeleteTablet for tablet 4829ec41391145cc84327d60048d9b0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:48830
W20251025 07:52:27.167742 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 4829ec41391145cc84327d60048d9b0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4829ec41391145cc84327d60048d9b0c already in progress: creating tablet
I20251025 07:52:27.169052 1166 ts_tablet_manager.cc:1434] T 33c608beb4ef4a859a135b11e07c81a4 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251025 07:52:27.169504 1166 tablet_bootstrap.cc:492] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:27.169085 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53198283718d4b45b30df65b5bde96a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.170090 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 4829ec41391145cc84327d60048d9b0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:27.170823 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c3eb43209f24592820638fc25486b78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:27.171082 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 4829ec41391145cc84327d60048d9b0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:27.171284 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 19918a3c1f1d4ed6a78d5b4f0f4e38b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:27.173614 1170 ts_tablet_manager.cc:1916] T 20e4178c16bd4f18b1c6bc969f6a0c6c P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:27.174281 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 576c51d9a22d45f6a3642a7296b5bf23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:27.174566 1171 tablet_replica.cc:333] T 576c51d9a22d45f6a3642a7296b5bf23 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:27.174748 1171 raft_consensus.cc:2243] T 576c51d9a22d45f6a3642a7296b5bf23 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:27.174949 1171 raft_consensus.cc:2272] T 576c51d9a22d45f6a3642a7296b5bf23 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:27.176745 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 19918a3c1f1d4ed6a78d5b4f0f4e38b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:27.176791 1167 tablet_bootstrap.cc:492] T 4829ec41391145cc84327d60048d9b0c P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:27.176941 1167 ts_tablet_manager.cc:1403] T 4829ec41391145cc84327d60048d9b0c P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.040s user 0.000s sys 0.004s
I20251025 07:52:27.177698 1167 raft_consensus.cc:359] T 4829ec41391145cc84327d60048d9b0c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.177886 1167 raft_consensus.cc:385] T 4829ec41391145cc84327d60048d9b0c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:27.177985 1167 raft_consensus.cc:740] T 4829ec41391145cc84327d60048d9b0c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:27.178651 1167 consensus_queue.cc:260] T 4829ec41391145cc84327d60048d9b0c P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.179410 1169 raft_consensus.cc:359] T 2fac7b819e2642378e0d3f9230d0259c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.179625 1169 raft_consensus.cc:385] T 2fac7b819e2642378e0d3f9230d0259c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:27.180734 1167 ts_tablet_manager.cc:1434] T 4829ec41391145cc84327d60048d9b0c P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251025 07:52:27.180981 1169 raft_consensus.cc:740] T 2fac7b819e2642378e0d3f9230d0259c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:27.181111 1167 tablet_bootstrap.cc:492] T 299572b2dbe244918dcebc5ea5f3c45a P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:27.181274 1169 consensus_queue.cc:260] T 2fac7b819e2642378e0d3f9230d0259c P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.182796 1171 ts_tablet_manager.cc:1916] T 576c51d9a22d45f6a3642a7296b5bf23 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:27.184919 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d5e5e5bbcf9f43118eea502b69880af5 consensus state has no leader
I20251025 07:52:27.185724 1169 ts_tablet_manager.cc:1434] T 2fac7b819e2642378e0d3f9230d0259c P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.032s user 0.001s sys 0.001s
I20251025 07:52:27.186030 1169 tablet_bootstrap.cc:492] T 576c51d9a22d45f6a3642a7296b5bf23 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:27.187773 1169 tablet_bootstrap.cc:654] T 576c51d9a22d45f6a3642a7296b5bf23 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:27.188086 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_73279bdee234454e9def34441da9579a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:27.188675 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73279bdee234454e9def34441da9579a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:27.190644 1171 ts_tablet_manager.cc:1929] T 576c51d9a22d45f6a3642a7296b5bf23 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:27.190757 1171 log.cc:1199] T 576c51d9a22d45f6a3642a7296b5bf23 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/576c51d9a22d45f6a3642a7296b5bf23
I20251025 07:52:27.191114 1171 ts_tablet_manager.cc:1950] T 576c51d9a22d45f6a3642a7296b5bf23 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:27.191135 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_97c3fbaf37cd4533bb3f4d8f55a0f945"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:27.191672 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97c3fbaf37cd4533bb3f4d8f55a0f945 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:27.192101 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 576c51d9a22d45f6a3642a7296b5bf23 (table default.Table_14cbf20f7ff449499612663b54f0acd9 [id=e9343178deb142339649b0cc6d26a9af]) successfully deleted
I20251025 07:52:27.192417 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 4829ec41391145cc84327d60048d9b0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:27.192682 1171 tablet_replica.cc:333] T 4829ec41391145cc84327d60048d9b0c P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:27.192868 1171 raft_consensus.cc:2243] T 4829ec41391145cc84327d60048d9b0c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:27.193046 1171 raft_consensus.cc:2272] T 4829ec41391145cc84327d60048d9b0c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:27.193121 517 master_service.cc:954] ReplaceTablet: received request to replace tablet 50433881c0eb4aa5a389b6f5b4a4337a from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:27.193446 1167 tablet_bootstrap.cc:654] T 299572b2dbe244918dcebc5ea5f3c45a P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:27.194247 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 742b3ec3f6604647b9e397532f1673dd consensus state has no leader
I20251025 07:52:27.194577 1171 ts_tablet_manager.cc:1916] T 4829ec41391145cc84327d60048d9b0c P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:27.194548 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_4976d4f5d05540258d144751d129bdb8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:27.195523 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4976d4f5d05540258d144751d129bdb8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:27.196717 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_5487882c2d0c4af4ad17df240fa90277"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:27.197153 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5487882c2d0c4af4ad17df240fa90277 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:27.200650 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 02e4c4d6ffb34cecaf17beb5ba197e81
W20251025 07:52:27.204780 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 509c94f2a422493ea4390ecee576db72 has no consensus state
I20251025 07:52:27.205214 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_ac2ef60285674f918695309a38ad40b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:27.205776 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac2ef60285674f918695309a38ad40b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:27.210983 1170 ts_tablet_manager.cc:1929] T 20e4178c16bd4f18b1c6bc969f6a0c6c P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:27.211117 1170 log.cc:1199] T 20e4178c16bd4f18b1c6bc969f6a0c6c P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/20e4178c16bd4f18b1c6bc969f6a0c6c
I20251025 07:52:27.211515 1170 ts_tablet_manager.cc:1950] T 20e4178c16bd4f18b1c6bc969f6a0c6c P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
W20251025 07:52:27.211923 510 catalog_manager.cc:5101] could not convert tablet's 5680e1adaf3f4b7f8c5e05523dd0950a schema from PB: OK
I20251025 07:52:27.213064 523 catalog_manager.cc:6719] ReplaceTablet: tablet 02e4c4d6ffb34cecaf17beb5ba197e81 deleted and replaced by tablet 3abfd52327a04ba88625f391a8658fbb
I20251025 07:52:27.213250 1169 tablet_bootstrap.cc:492] T 576c51d9a22d45f6a3642a7296b5bf23 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:27.213397 1169 ts_tablet_manager.cc:1403] T 576c51d9a22d45f6a3642a7296b5bf23 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20251025 07:52:27.214205 1169 raft_consensus.cc:359] T 576c51d9a22d45f6a3642a7296b5bf23 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.214316 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 02e4c4d6ffb34cecaf17beb5ba197e81 with delete_type TABLET_DATA_DELETED (replaced by tablet 3abfd52327a04ba88625f391a8658fbb) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:27.214401 1169 raft_consensus.cc:385] T 576c51d9a22d45f6a3642a7296b5bf23 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:27.214471 1169 raft_consensus.cc:740] T 576c51d9a22d45f6a3642a7296b5bf23 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:27.214726 1169 consensus_queue.cc:260] T 576c51d9a22d45f6a3642a7296b5bf23 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.215309 1169 ts_tablet_manager.cc:1434] T 576c51d9a22d45f6a3642a7296b5bf23 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:27.215644 1169 tablet_bootstrap.cc:492] T 299572b2dbe244918dcebc5ea5f3c45a P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:27.218308 1169 tablet_bootstrap.cc:654] T 299572b2dbe244918dcebc5ea5f3c45a P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:27.218348 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_3cf5eecb57c54611bdcc3dfca5fcdca1" } new_table_name: "default.Table_f84af906af804c7a8e98ad602783d747" modify_external_catalogs: true
I20251025 07:52:27.221418 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 02e4c4d6ffb34cecaf17beb5ba197e81 with delete_type TABLET_DATA_DELETED (replaced by tablet 3abfd52327a04ba88625f391a8658fbb) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:27.221518 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 20e4178c16bd4f18b1c6bc969f6a0c6c (table default.Table_b42187d20a7e4f8783a8d25acb249350 [id=a5c73cb1e5a349ca8675f4ba4cb8a82d]) successfully deleted
W20251025 07:52:27.221760 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet d37758f606344127b3db74c87babe69c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d37758f606344127b3db74c87babe69c already in progress: creating tablet
I20251025 07:52:27.223480 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5e5e5bbcf9f43118eea502b69880af5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.223026 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 105) took 1061 ms. Trace:
I20251025 07:52:27.223961 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 33c608beb4ef4a859a135b11e07c81a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:27.225538 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c3eb43209f24592820638fc25486b78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:56590
W20251025 07:52:27.226267 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 21e52928580c4762b0ae7152490ef955 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21e52928580c4762b0ae7152490ef955 already in progress: creating tablet
I20251025 07:52:27.226372 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 02e4c4d6ffb34cecaf17beb5ba197e81 with delete_type TABLET_DATA_DELETED (replaced by tablet 3abfd52327a04ba88625f391a8658fbb) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:27.226488 1172 rpcz_store.cc:276] 1025 07:52:26.097406 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:26.104248 (+ 6842us) service_pool.cc:225] Handling call
1025 07:52:27.158288 (+1054040us) ts_tablet_manager.cc:1151] Deleting tablet 1b31177a062b467187d68ed2034f5690
1025 07:52:27.158297 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:27.158822 (+ 525us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1035649}
W20251025 07:52:27.227177 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 1b31177a062b467187d68ed2034f5690 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b31177a062b467187d68ed2034f5690 already in progress: creating tablet
I20251025 07:52:27.227957 1170 tablet_replica.cc:333] T a02b627401a64e1583f9e6123ef1be6f P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
W20251025 07:52:27.233170 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bef0ef23d90d480d9c99aef21d7271fd consensus state has no leader
I20251025 07:52:27.228163 1170 raft_consensus.cc:2243] T a02b627401a64e1583f9e6123ef1be6f P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:27.236809 1170 raft_consensus.cc:2272] T a02b627401a64e1583f9e6123ef1be6f P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:27.238909 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 50433881c0eb4aa5a389b6f5b4a4337a
I20251025 07:52:27.239532 517 catalog_manager.cc:6719] ReplaceTablet: tablet 50433881c0eb4aa5a389b6f5b4a4337a deleted and replaced by tablet 50ae9bca6bed4f749d54191ac1543410
I20251025 07:52:27.240391 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 50433881c0eb4aa5a389b6f5b4a4337a with delete_type TABLET_DATA_DELETED (replaced by tablet 50ae9bca6bed4f749d54191ac1543410) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:27.242893 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet 50433881c0eb4aa5a389b6f5b4a4337a with delete_type TABLET_DATA_DELETED (replaced by tablet 50ae9bca6bed4f749d54191ac1543410) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:27.244504 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 50433881c0eb4aa5a389b6f5b4a4337a with delete_type TABLET_DATA_DELETED (replaced by tablet 50ae9bca6bed4f749d54191ac1543410) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:27.249096 1171 ts_tablet_manager.cc:1929] T 4829ec41391145cc84327d60048d9b0c P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:27.249276 1171 log.cc:1199] T 4829ec41391145cc84327d60048d9b0c P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/4829ec41391145cc84327d60048d9b0c
I20251025 07:52:27.249626 1171 ts_tablet_manager.cc:1950] T 4829ec41391145cc84327d60048d9b0c P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:27.250988 424 master_runner.cc:186] Error getting master registration for 127.0.104.60:37611: Network error: Client connection negotiation failed: client connection to 127.0.104.60:37611: connect: Connection refused (error 111),
I20251025 07:52:27.251144 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 07:52:27.251835 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 4829ec41391145cc84327d60048d9b0c (table default.Table_45b1ec3be7e9460296fe5b819beaa6ea [id=94a9a4e752b547afb3d4d0b6806bdb54]) successfully deleted
I20251025 07:52:27.253268 1169 tablet_bootstrap.cc:492] T 299572b2dbe244918dcebc5ea5f3c45a P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:27.253412 1169 ts_tablet_manager.cc:1403] T 299572b2dbe244918dcebc5ea5f3c45a P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.002s
I20251025 07:52:27.254293 1170 ts_tablet_manager.cc:1916] T a02b627401a64e1583f9e6123ef1be6f P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:27.254328 1169 raft_consensus.cc:359] T 299572b2dbe244918dcebc5ea5f3c45a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.254504 1169 raft_consensus.cc:385] T 299572b2dbe244918dcebc5ea5f3c45a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:27.254590 1169 raft_consensus.cc:740] T 299572b2dbe244918dcebc5ea5f3c45a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:27.254819 1169 consensus_queue.cc:260] T 299572b2dbe244918dcebc5ea5f3c45a P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.255470 1169 ts_tablet_manager.cc:1434] T 299572b2dbe244918dcebc5ea5f3c45a P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251025 07:52:27.256073 521 master_service.cc:954] ReplaceTablet: received request to replace tablet a13bf4d7eb1440f7a28b85cb1bf99667 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:27.256461 1169 tablet_bootstrap.cc:492] T 4829ec41391145cc84327d60048d9b0c P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:27.258220 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 108) took 1119 ms. Trace:
I20251025 07:52:27.258325 1172 rpcz_store.cc:276] 1025 07:52:26.137708 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:26.145298 (+ 7590us) service_pool.cc:225] Handling call
1025 07:52:27.226848 (+1081550us) ts_tablet_manager.cc:1151] Deleting tablet b00153c97104470883f6089d71d23708
1025 07:52:27.256069 (+ 29221us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:27.257077 (+ 1008us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1081331}
I20251025 07:52:27.258899 1169 tablet_bootstrap.cc:654] T 4829ec41391145cc84327d60048d9b0c P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:27.258898 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet b00153c97104470883f6089d71d23708 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b00153c97104470883f6089d71d23708 already in progress: creating tablet
W20251025 07:52:27.259699 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 02e4c4d6ffb34cecaf17beb5ba197e81 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02e4c4d6ffb34cecaf17beb5ba197e81 already in progress: creating tablet
W20251025 07:52:27.260360 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 50433881c0eb4aa5a389b6f5b4a4337a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50433881c0eb4aa5a389b6f5b4a4337a already in progress: creating tablet
I20251025 07:52:27.261937 1166 tablet_bootstrap.cc:654] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:27.263486 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 107) took 1125 ms. Trace:
I20251025 07:52:27.263592 1172 rpcz_store.cc:276] 1025 07:52:26.137605 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:26.145555 (+ 7950us) service_pool.cc:225] Handling call
1025 07:52:27.258595 (+1113040us) ts_tablet_manager.cc:1151] Deleting tablet ac0ea298bc114c56a8093e12def21ce1
1025 07:52:27.262255 (+ 3660us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:27.263332 (+ 1077us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1112882}
W20251025 07:52:27.264230 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet ac0ea298bc114c56a8093e12def21ce1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac0ea298bc114c56a8093e12def21ce1 already in progress: creating tablet
I20251025 07:52:27.264511 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 113) took 1018 ms. Trace:
I20251025 07:52:27.265751 1172 rpcz_store.cc:276] 1025 07:52:26.245390 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:26.245443 (+ 53us) service_pool.cc:225] Handling call
1025 07:52:27.263849 (+1018406us) ts_tablet_manager.cc:1151] Deleting tablet f3c87a83082b421cb4be546ac4ce7ad8
1025 07:52:27.263857 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:27.263940 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1018191}
W20251025 07:52:27.266556 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet f3c87a83082b421cb4be546ac4ce7ad8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3c87a83082b421cb4be546ac4ce7ad8 already in progress: creating tablet
I20251025 07:52:27.267252 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 112) took 1023 ms. Trace:
I20251025 07:52:27.267359 1172 rpcz_store.cc:276] 1025 07:52:26.243806 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:26.248205 (+ 4399us) service_pool.cc:225] Handling call
1025 07:52:27.266038 (+1017833us) ts_tablet_manager.cc:1151] Deleting tablet 5caac9c943bf42a5b2ae522d87fdfe8e
1025 07:52:27.266049 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:27.267198 (+ 1149us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1017611}
I20251025 07:52:27.267766 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 111) took 1024 ms. Trace:
I20251025 07:52:27.267853 1172 rpcz_store.cc:276] 1025 07:52:26.243618 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:26.248459 (+ 4841us) service_pool.cc:225] Handling call
1025 07:52:27.267613 (+1019154us) ts_tablet_manager.cc:1151] Deleting tablet 94408a3a90224645ba3cacccf5de33f1
1025 07:52:27.267624 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:27.267713 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1018966}
I20251025 07:52:27.268179 1172 tablet_replica.cc:333] T a02b627401a64e1583f9e6123ef1be6f P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:27.268395 1172 raft_consensus.cc:2243] T a02b627401a64e1583f9e6123ef1be6f P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:27.268684 1172 pending_rounds.cc:70] T a02b627401a64e1583f9e6123ef1be6f P 3aea2869ff814cc7b142012faa274382: Trying to abort 1 pending ops.
I20251025 07:52:27.268747 1172 pending_rounds.cc:77] T a02b627401a64e1583f9e6123ef1be6f P 3aea2869ff814cc7b142012faa274382: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214607347573067776 op_type: NO_OP noop_request { }
I20251025 07:52:27.268895 1172 raft_consensus.cc:2889] T a02b627401a64e1583f9e6123ef1be6f P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 07:52:27.269004 1172 raft_consensus.cc:2272] T a02b627401a64e1583f9e6123ef1be6f P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:27.269807 1172 ts_tablet_manager.cc:1916] T a02b627401a64e1583f9e6123ef1be6f P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:27.269878 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 94408a3a90224645ba3cacccf5de33f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94408a3a90224645ba3cacccf5de33f1 already in progress: creating tablet
I20251025 07:52:27.271742 1169 tablet_bootstrap.cc:492] T 4829ec41391145cc84327d60048d9b0c P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:27.271939 1169 ts_tablet_manager.cc:1403] T 4829ec41391145cc84327d60048d9b0c P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20251025 07:52:27.272867 1169 raft_consensus.cc:359] T 4829ec41391145cc84327d60048d9b0c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.273082 1169 raft_consensus.cc:385] T 4829ec41391145cc84327d60048d9b0c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:27.273156 1169 raft_consensus.cc:740] T 4829ec41391145cc84327d60048d9b0c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:27.273415 1169 consensus_queue.cc:260] T 4829ec41391145cc84327d60048d9b0c P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.274240 1169 ts_tablet_manager.cc:1434] T 4829ec41391145cc84327d60048d9b0c P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20251025 07:52:27.274562 1169 tablet_bootstrap.cc:492] T 713ae1dd444544bdabdd4a771e109d90 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:27.276458 1169 tablet_bootstrap.cc:654] T 713ae1dd444544bdabdd4a771e109d90 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:27.285737 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8956ad28b35d45f28c683ac055bba441 consensus state has no leader
I20251025 07:52:27.288318 1218 raft_consensus.cc:493] T 713ae1dd444544bdabdd4a771e109d90 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:27.288519 1218 raft_consensus.cc:515] T 713ae1dd444544bdabdd4a771e109d90 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.288585 523 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 2fac7b819e2642378e0d3f9230d0259c (table default.Table_33b6fa5be1ab40fba35b3d5ef9ad9668 [id=6605e11e11804d408a6a4a9d403dde95]). Expected version 1 got 0
I20251025 07:52:27.289258 1218 leader_election.cc:290] T 713ae1dd444544bdabdd4a771e109d90 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:27.289927 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "713ae1dd444544bdabdd4a771e109d90" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:27.290064 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "713ae1dd444544bdabdd4a771e109d90" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
W20251025 07:52:27.290580 850 leader_election.cc:343] T 713ae1dd444544bdabdd4a771e109d90 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:27.290786 847 leader_election.cc:343] T 713ae1dd444544bdabdd4a771e109d90 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:27.290921 847 leader_election.cc:304] T 713ae1dd444544bdabdd4a771e109d90 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:27.291410 1218 raft_consensus.cc:2749] T 713ae1dd444544bdabdd4a771e109d90 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:27.293452 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 70fe9aa2f58d4abea3b3b71066a29653 (DEFAULT_TABLE table=default.Table_517752d265854c7cbd706e2742536620 [id=aa0642dd1fa34cf8bd1fe2070037a55a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.294023 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70fe9aa2f58d4abea3b3b71066a29653. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.298455 1021 tablet_service.cc:1505] Processing CreateTablet for tablet 509c94f2a422493ea4390ecee576db72 (DEFAULT_TABLE table=default.Table_3f081b7bb4e1420c8121ea4115439f3d [id=696a1285fbfe4038a8b1b38708c8ea49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.298883 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 509c94f2a422493ea4390ecee576db72. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.300415 755 tablet_service.cc:1505] Processing CreateTablet for tablet 70fe9aa2f58d4abea3b3b71066a29653 (DEFAULT_TABLE table=default.Table_517752d265854c7cbd706e2742536620 [id=aa0642dd1fa34cf8bd1fe2070037a55a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.300604 522 master_service.cc:954] ReplaceTablet: received request to replace tablet 85fba8b37f2c4540aa81701052bf222a from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:27.300894 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70fe9aa2f58d4abea3b3b71066a29653. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.301021 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 2c50ec95e48e49b59a76fac4fc862d3d (DEFAULT_TABLE table=default.Table_490b6bd20f694746afabf7be7f74fbc0 [id=c28b0b6acd61490caecfdaab803a9d92]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.301586 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c50ec95e48e49b59a76fac4fc862d3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.306046 741 tablet_service.cc:1552] Processing DeleteTablet for tablet 4829ec41391145cc84327d60048d9b0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:27.306560 743 tablet_service.cc:1505] Processing CreateTablet for tablet 509c94f2a422493ea4390ecee576db72 (DEFAULT_TABLE table=default.Table_3f081b7bb4e1420c8121ea4115439f3d [id=696a1285fbfe4038a8b1b38708c8ea49]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:27.306762 510 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 4829ec41391145cc84327d60048d9b0c because the tablet was not found. No further retry: Not found: Tablet not found: 4829ec41391145cc84327d60048d9b0c
I20251025 07:52:27.310518 1166 tablet_bootstrap.cc:492] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:27.310693 1166 ts_tablet_manager.cc:1403] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.141s user 0.002s sys 0.002s
I20251025 07:52:27.311738 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 509c94f2a422493ea4390ecee576db72. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.311957 1166 raft_consensus.cc:359] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.312153 1166 raft_consensus.cc:385] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:27.312222 1166 raft_consensus.cc:740] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:27.312463 1166 consensus_queue.cc:260] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.312924 886 tablet_service.cc:1505] Processing CreateTablet for tablet 2c50ec95e48e49b59a76fac4fc862d3d (DEFAULT_TABLE table=default.Table_490b6bd20f694746afabf7be7f74fbc0 [id=c28b0b6acd61490caecfdaab803a9d92]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.313706 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_692ab786006945128bee1c8578c2a0cd" } modify_external_catalogs: true
I20251025 07:52:27.312677 883 tablet_service.cc:1505] Processing CreateTablet for tablet 509c94f2a422493ea4390ecee576db72 (DEFAULT_TABLE table=default.Table_3f081b7bb4e1420c8121ea4115439f3d [id=696a1285fbfe4038a8b1b38708c8ea49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.313891 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_692ab786006945128bee1c8578c2a0cd" } modify_external_catalogs: true
I20251025 07:52:27.314116 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 509c94f2a422493ea4390ecee576db72. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.314471 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_7c2f0e6e0edf4ae6920557e0d5130468" } modify_external_catalogs: true
I20251025 07:52:27.314689 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_7c2f0e6e0edf4ae6920557e0d5130468" } modify_external_catalogs: true
I20251025 07:52:27.315234 1166 ts_tablet_manager.cc:1434] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251025 07:52:27.315547 1166 tablet_bootstrap.cc:492] T 2947502815d742e89cacb474b7fbb249 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:27.316550 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c50ec95e48e49b59a76fac4fc862d3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.320418 753 tablet_service.cc:1505] Processing CreateTablet for tablet 2c50ec95e48e49b59a76fac4fc862d3d (DEFAULT_TABLE table=default.Table_490b6bd20f694746afabf7be7f74fbc0 [id=c28b0b6acd61490caecfdaab803a9d92]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.320928 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c50ec95e48e49b59a76fac4fc862d3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.323968 887 tablet_service.cc:1505] Processing CreateTablet for tablet 70fe9aa2f58d4abea3b3b71066a29653 (DEFAULT_TABLE table=default.Table_517752d265854c7cbd706e2742536620 [id=aa0642dd1fa34cf8bd1fe2070037a55a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.324445 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70fe9aa2f58d4abea3b3b71066a29653. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.324697 1166 tablet_bootstrap.cc:654] T 2947502815d742e89cacb474b7fbb249 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:27.334782 1172 ts_tablet_manager.cc:1929] T a02b627401a64e1583f9e6123ef1be6f P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:27.334911 1166 tablet_bootstrap.cc:492] T 2947502815d742e89cacb474b7fbb249 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:27.335062 1166 ts_tablet_manager.cc:1403] T 2947502815d742e89cacb474b7fbb249 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.002s
I20251025 07:52:27.336120 1166 raft_consensus.cc:359] T 2947502815d742e89cacb474b7fbb249 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.336323 1166 raft_consensus.cc:385] T 2947502815d742e89cacb474b7fbb249 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:27.336337 518 master_service.cc:954] ReplaceTablet: received request to replace tablet 576c51d9a22d45f6a3642a7296b5bf23 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:27.336416 1166 raft_consensus.cc:740] T 2947502815d742e89cacb474b7fbb249 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:27.336637 1166 consensus_queue.cc:260] T 2947502815d742e89cacb474b7fbb249 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.337177 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet a13bf4d7eb1440f7a28b85cb1bf99667
I20251025 07:52:27.337641 521 catalog_manager.cc:6719] ReplaceTablet: tablet a13bf4d7eb1440f7a28b85cb1bf99667 deleted and replaced by tablet 74e2d78c024849b0af5c0bde1f48c36f
I20251025 07:52:27.337792 1166 ts_tablet_manager.cc:1434] T 2947502815d742e89cacb474b7fbb249 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251025 07:52:27.339597 1166 tablet_bootstrap.cc:492] T 3c3eb43209f24592820638fc25486b78 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:27.339862 753 tablet_service.cc:1552] Processing DeleteTablet for tablet a13bf4d7eb1440f7a28b85cb1bf99667 with delete_type TABLET_DATA_DELETED (replaced by tablet 74e2d78c024849b0af5c0bde1f48c36f) from {username='slave'} at 127.0.0.1:48830
W20251025 07:52:27.340461 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet a13bf4d7eb1440f7a28b85cb1bf99667 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a13bf4d7eb1440f7a28b85cb1bf99667 already in progress: creating tablet
I20251025 07:52:27.335059 1172 log.cc:1199] T a02b627401a64e1583f9e6123ef1be6f P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/a02b627401a64e1583f9e6123ef1be6f
I20251025 07:52:27.342303 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 2947502815d742e89cacb474b7fbb249 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:27.342453 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet a13bf4d7eb1440f7a28b85cb1bf99667 with delete_type TABLET_DATA_DELETED (replaced by tablet 74e2d78c024849b0af5c0bde1f48c36f) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:27.342573 1166 tablet_bootstrap.cc:654] T 3c3eb43209f24592820638fc25486b78 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:27.343703 1167 tablet_bootstrap.cc:492] T 299572b2dbe244918dcebc5ea5f3c45a P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:27.343892 1167 ts_tablet_manager.cc:1403] T 299572b2dbe244918dcebc5ea5f3c45a P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.163s user 0.006s sys 0.022s
I20251025 07:52:27.344362 1170 ts_tablet_manager.cc:1929] T a02b627401a64e1583f9e6123ef1be6f P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:27.344601 1170 log.cc:1199] T a02b627401a64e1583f9e6123ef1be6f P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/a02b627401a64e1583f9e6123ef1be6f
I20251025 07:52:27.344838 1167 raft_consensus.cc:359] T 299572b2dbe244918dcebc5ea5f3c45a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.345069 1167 raft_consensus.cc:385] T 299572b2dbe244918dcebc5ea5f3c45a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:27.345162 1167 raft_consensus.cc:740] T 299572b2dbe244918dcebc5ea5f3c45a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:27.345178 1170 ts_tablet_manager.cc:1950] T a02b627401a64e1583f9e6123ef1be6f P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:27.345379 884 tablet_service.cc:1552] Processing DeleteTablet for tablet a13bf4d7eb1440f7a28b85cb1bf99667 with delete_type TABLET_DATA_DELETED (replaced by tablet 74e2d78c024849b0af5c0bde1f48c36f) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:27.345434 1167 consensus_queue.cc:260] T 299572b2dbe244918dcebc5ea5f3c45a P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.346657 1172 ts_tablet_manager.cc:1950] T a02b627401a64e1583f9e6123ef1be6f P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:27.346763 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet a02b627401a64e1583f9e6123ef1be6f (table default.Table_1437bb6ce9154a15b1817a029151c71a [id=9fd173d153294521a4f14c390b316594]) successfully deleted
I20251025 07:52:27.347169 1170 tablet_replica.cc:333] T 785d0ce85b234826ba8021fbd182bf3c P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:27.347416 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 117) took 1079 ms. Trace:
I20251025 07:52:27.347452 1170 raft_consensus.cc:2243] T 785d0ce85b234826ba8021fbd182bf3c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:27.347797 1170 raft_consensus.cc:2272] T 785d0ce85b234826ba8021fbd182bf3c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:27.348091 1218 raft_consensus.cc:493] T 2947502815d742e89cacb474b7fbb249 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:27.348325 1218 raft_consensus.cc:515] T 2947502815d742e89cacb474b7fbb249 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.348589 1170 ts_tablet_manager.cc:1916] T 785d0ce85b234826ba8021fbd182bf3c P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:27.348955 1218 leader_election.cc:290] T 2947502815d742e89cacb474b7fbb249 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:27.349148 1172 rpcz_store.cc:276] 1025 07:52:26.267555 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:26.268074 (+ 519us) service_pool.cc:225] Handling call
1025 07:52:27.268084 (+1000010us) ts_tablet_manager.cc:1151] Deleting tablet a02b627401a64e1583f9e6123ef1be6f
1025 07:52:27.268095 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:27.284121 (+ 16026us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:27.334675 (+ 50554us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:27.347361 (+ 12686us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":997847}
I20251025 07:52:27.349380 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2947502815d742e89cacb474b7fbb249" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:27.349419 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2947502815d742e89cacb474b7fbb249" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:27.349937 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet a02b627401a64e1583f9e6123ef1be6f (table default.Table_1437bb6ce9154a15b1817a029151c71a [id=9fd173d153294521a4f14c390b316594]) successfully deleted
W20251025 07:52:27.349967 850 leader_election.cc:343] T 2947502815d742e89cacb474b7fbb249 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:27.350224 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 7f29b64704e147aa9b0b5f6cda860f09 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f29b64704e147aa9b0b5f6cda860f09 already in progress: creating tablet
I20251025 07:52:27.350286 1172 tablet_replica.cc:333] T 5680e1adaf3f4b7f8c5e05523dd0950a P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:27.350472 1172 raft_consensus.cc:2243] T 5680e1adaf3f4b7f8c5e05523dd0950a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:27.350628 1172 raft_consensus.cc:2272] T 5680e1adaf3f4b7f8c5e05523dd0950a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:27.351094 1172 ts_tablet_manager.cc:1916] T 5680e1adaf3f4b7f8c5e05523dd0950a P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:27.351094 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 4f25c9caa05b4da4bd56a8a21fde423c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f25c9caa05b4da4bd56a8a21fde423c already in progress: creating tablet
W20251025 07:52:27.351356 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 85e27fc52e9a4d62a8622645594f59c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 85e27fc52e9a4d62a8622645594f59c6 already in progress: creating tablet
W20251025 07:52:27.351594 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 2784bbfb97d5466f9ec856add7d0b29a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2784bbfb97d5466f9ec856add7d0b29a already in progress: creating tablet
W20251025 07:52:27.352196 847 leader_election.cc:343] T 2947502815d742e89cacb474b7fbb249 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:27.352347 847 leader_election.cc:304] T 2947502815d742e89cacb474b7fbb249 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:27.352831 1218 raft_consensus.cc:2749] T 2947502815d742e89cacb474b7fbb249 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:27.362120 523 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 2fac7b819e2642378e0d3f9230d0259c (table default.Table_33b6fa5be1ab40fba35b3d5ef9ad9668 [id=6605e11e11804d408a6a4a9d403dde95]). Expected version 1 got 0
I20251025 07:52:27.362426 523 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 299572b2dbe244918dcebc5ea5f3c45a (table default.Table_bd5089a7892041649dd05b7fb0961255 [id=840b3ffcb44144548843457c48f46380]). Expected version 1 got 0
I20251025 07:52:27.364236 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_936e8f1bd9344704940225361305cf5b" } modify_external_catalogs: true
I20251025 07:52:27.364419 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_936e8f1bd9344704940225361305cf5b" } modify_external_catalogs: true
I20251025 07:52:27.364989 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_37ad28b796a342f3a6e962ead289ec5b" } modify_external_catalogs: true
I20251025 07:52:27.365170 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_37ad28b796a342f3a6e962ead289ec5b" } modify_external_catalogs: true
I20251025 07:52:27.366050 1167 ts_tablet_manager.cc:1434] T 299572b2dbe244918dcebc5ea5f3c45a P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20251025 07:52:27.366547 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_5487882c2d0c4af4ad17df240fa90277" } modify_external_catalogs: true
I20251025 07:52:27.366715 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_5487882c2d0c4af4ad17df240fa90277" } modify_external_catalogs: true
I20251025 07:52:27.367754 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet 576c51d9a22d45f6a3642a7296b5bf23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:27.369213 1167 tablet_bootstrap.cc:492] T 713ae1dd444544bdabdd4a771e109d90 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:27.369428 753 tablet_service.cc:1505] Processing CreateTablet for tablet 551d07a9242740dcb9366bb2d707a4a8 (DEFAULT_TABLE table=default.Table_ac2ef60285674f918695309a38ad40b7 [id=dc6d03745bdc42d0ad8bdb9f81971b81]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.369881 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 551d07a9242740dcb9366bb2d707a4a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.370398 755 tablet_service.cc:1505] Processing CreateTablet for tablet 6ec421eb820b4f40afcff5a5bf80c508 (DEFAULT_TABLE table=default.Table_5487882c2d0c4af4ad17df240fa90277 [id=d73352841fda408baf49087af00b4250]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.370850 884 tablet_service.cc:1505] Processing CreateTablet for tablet 551d07a9242740dcb9366bb2d707a4a8 (DEFAULT_TABLE table=default.Table_ac2ef60285674f918695309a38ad40b7 [id=dc6d03745bdc42d0ad8bdb9f81971b81]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.371214 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 551d07a9242740dcb9366bb2d707a4a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.372582 883 tablet_service.cc:1505] Processing CreateTablet for tablet 6ec421eb820b4f40afcff5a5bf80c508 (DEFAULT_TABLE table=default.Table_5487882c2d0c4af4ad17df240fa90277 [id=d73352841fda408baf49087af00b4250]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.372972 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ec421eb820b4f40afcff5a5bf80c508. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.374814 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 551d07a9242740dcb9366bb2d707a4a8 (DEFAULT_TABLE table=default.Table_ac2ef60285674f918695309a38ad40b7 [id=dc6d03745bdc42d0ad8bdb9f81971b81]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.375236 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 551d07a9242740dcb9366bb2d707a4a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.377997 1172 ts_tablet_manager.cc:1929] T 5680e1adaf3f4b7f8c5e05523dd0950a P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:27.378146 1172 log.cc:1199] T 5680e1adaf3f4b7f8c5e05523dd0950a P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/5680e1adaf3f4b7f8c5e05523dd0950a
I20251025 07:52:27.378197 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ec421eb820b4f40afcff5a5bf80c508. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.378511 1172 ts_tablet_manager.cc:1950] T 5680e1adaf3f4b7f8c5e05523dd0950a P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
W20251025 07:52:27.378484 511 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet d5e5e5bbcf9f43118eea502b69880af5 (table default.Table_c82c2dad12af41788d8959f7bd9f92b0 [id=4d9e99d6474b4d00bc66e26a33694e69], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d5e5e5bbcf9f43118eea502b69880af5 consensus state has no leader
I20251025 07:52:27.379612 1172 tablet_replica.cc:333] T 713ae1dd444544bdabdd4a771e109d90 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:27.379824 1172 raft_consensus.cc:2243] T 713ae1dd444544bdabdd4a771e109d90 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:27.379979 1172 raft_consensus.cc:2272] T 713ae1dd444544bdabdd4a771e109d90 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:27.380510 1172 ts_tablet_manager.cc:1916] T 713ae1dd444544bdabdd4a771e109d90 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:27.380673 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 6ec421eb820b4f40afcff5a5bf80c508 (DEFAULT_TABLE table=default.Table_5487882c2d0c4af4ad17df240fa90277 [id=d73352841fda408baf49087af00b4250]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.380892 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 5680e1adaf3f4b7f8c5e05523dd0950a (table default.Table_31d7f6f73d6e4123b1ff1df86ff62205 [id=cb037f61ee0b4b6cb8fb613263a278fa]) successfully deleted
I20251025 07:52:27.381071 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ec421eb820b4f40afcff5a5bf80c508. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:27.381148 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 198db9a45257435f86df2605992ccb34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 198db9a45257435f86df2605992ccb34 already in progress: creating tablet
I20251025 07:52:27.381690 492 master_runner.cc:186] Error getting master registration for 127.0.104.60:37611: Network error: Client connection negotiation failed: client connection to 127.0.104.60:37611: connect: Connection refused (error 111),
I20251025 07:52:27.381851 492 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251025 07:52:27.385061 1169 tablet_bootstrap.cc:492] T 713ae1dd444544bdabdd4a771e109d90 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:27.385215 1169 ts_tablet_manager.cc:1403] T 713ae1dd444544bdabdd4a771e109d90 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.111s user 0.007s sys 0.000s
I20251025 07:52:27.386029 1169 raft_consensus.cc:359] T 713ae1dd444544bdabdd4a771e109d90 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.386240 1169 raft_consensus.cc:385] T 713ae1dd444544bdabdd4a771e109d90 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:27.386322 1169 raft_consensus.cc:740] T 713ae1dd444544bdabdd4a771e109d90 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:27.386546 1169 consensus_queue.cc:260] T 713ae1dd444544bdabdd4a771e109d90 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.387599 1167 tablet_bootstrap.cc:654] T 713ae1dd444544bdabdd4a771e109d90 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:27.390098 1169 ts_tablet_manager.cc:1434] T 713ae1dd444544bdabdd4a771e109d90 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251025 07:52:27.392918 1169 tablet_bootstrap.cc:492] T 3d5e0dd14e474fe09993a03851f8b1f0 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:27.393988 1166 tablet_bootstrap.cc:492] T 3c3eb43209f24592820638fc25486b78 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:27.394074 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 713ae1dd444544bdabdd4a771e109d90 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:27.394207 1166 ts_tablet_manager.cc:1403] T 3c3eb43209f24592820638fc25486b78 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.009s
I20251025 07:52:27.395119 1166 raft_consensus.cc:359] T 3c3eb43209f24592820638fc25486b78 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.395323 1166 raft_consensus.cc:385] T 3c3eb43209f24592820638fc25486b78 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:27.395303 753 tablet_service.cc:1505] Processing CreateTablet for tablet ef1854d35a3047e3a9fbe0166ab1295a (DEFAULT_TABLE table=default.Table_97c3fbaf37cd4533bb3f4d8f55a0f945 [id=834669c54dae449790818684fd8659bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.395392 1166 raft_consensus.cc:740] T 3c3eb43209f24592820638fc25486b78 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:27.395730 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef1854d35a3047e3a9fbe0166ab1295a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.395644 1166 consensus_queue.cc:260] T 3c3eb43209f24592820638fc25486b78 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.395924 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 2209089667dd4937abd96615f6929449
I20251025 07:52:27.396154 1166 ts_tablet_manager.cc:1434] T 3c3eb43209f24592820638fc25486b78 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 07:52:27.396515 1166 tablet_bootstrap.cc:492] T 50433881c0eb4aa5a389b6f5b4a4337a P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:27.398294 1166 tablet_bootstrap.cc:654] T 50433881c0eb4aa5a389b6f5b4a4337a P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:27.400457 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 2209089667dd4937abd96615f6929449 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:27.400646 1024 tablet_service.cc:1505] Processing CreateTablet for tablet ef1854d35a3047e3a9fbe0166ab1295a (DEFAULT_TABLE table=default.Table_97c3fbaf37cd4533bb3f4d8f55a0f945 [id=834669c54dae449790818684fd8659bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.401051 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef1854d35a3047e3a9fbe0166ab1295a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.402112 1178 raft_consensus.cc:493] T 2fac7b819e2642378e0d3f9230d0259c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:27.402307 1178 raft_consensus.cc:515] T 2fac7b819e2642378e0d3f9230d0259c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.402952 1178 leader_election.cc:290] T 2fac7b819e2642378e0d3f9230d0259c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:27.403628 523 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 299572b2dbe244918dcebc5ea5f3c45a (table default.Table_bd5089a7892041649dd05b7fb0961255 [id=840b3ffcb44144548843457c48f46380]). Expected version 1 got 0
I20251025 07:52:27.403807 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fac7b819e2642378e0d3f9230d0259c" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:27.404038 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fac7b819e2642378e0d3f9230d0259c" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:27.404078 914 raft_consensus.cc:2468] T 2fac7b819e2642378e0d3f9230d0259c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:27.404232 1048 raft_consensus.cc:2468] T 2fac7b819e2642378e0d3f9230d0259c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:27.404843 710 leader_election.cc:304] T 2fac7b819e2642378e0d3f9230d0259c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:27.406651 1178 raft_consensus.cc:2804] T 2fac7b819e2642378e0d3f9230d0259c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:27.406773 1178 raft_consensus.cc:493] T 2fac7b819e2642378e0d3f9230d0259c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:27.406873 1178 raft_consensus.cc:3060] T 2fac7b819e2642378e0d3f9230d0259c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:27.407600 523 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 299572b2dbe244918dcebc5ea5f3c45a (table default.Table_bd5089a7892041649dd05b7fb0961255 [id=840b3ffcb44144548843457c48f46380]). Expected version 1 got 0
I20251025 07:52:27.407616 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 2209089667dd4937abd96615f6929449 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:27.407871 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 83084be89b2646b0bda208b3f6f60d77
I20251025 07:52:27.408023 883 tablet_service.cc:1505] Processing CreateTablet for tablet ef1854d35a3047e3a9fbe0166ab1295a (DEFAULT_TABLE table=default.Table_97c3fbaf37cd4533bb3f4d8f55a0f945 [id=834669c54dae449790818684fd8659bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.408416 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef1854d35a3047e3a9fbe0166ab1295a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.408845 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 83084be89b2646b0bda208b3f6f60d77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:27.409158 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 85fba8b37f2c4540aa81701052bf222a
W20251025 07:52:27.409624 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 83084be89b2646b0bda208b3f6f60d77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83084be89b2646b0bda208b3f6f60d77 already in progress: creating tablet
I20251025 07:52:27.409631 1178 raft_consensus.cc:515] T 2fac7b819e2642378e0d3f9230d0259c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.410072 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 85fba8b37f2c4540aa81701052bf222a with delete_type TABLET_DATA_DELETED (replaced by tablet 6a58931e7cd7454fb583f1c70cd2ad15) from {username='slave'} at 127.0.0.1:48830
W20251025 07:52:27.410701 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 85fba8b37f2c4540aa81701052bf222a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 85fba8b37f2c4540aa81701052bf222a already in progress: creating tablet
I20251025 07:52:27.411075 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_98454703d7234750b3cb2aab947ef63e" } new_table_name: "default.Table_e4cd0fd928264afea9fbdf342da2a781" modify_external_catalogs: true
I20251025 07:52:27.411551 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 6ec421eb820b4f40afcff5a5bf80c508
I20251025 07:52:27.411679 1178 leader_election.cc:290] T 2fac7b819e2642378e0d3f9230d0259c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:27.411859 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 2209089667dd4937abd96615f6929449 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:27.411986 1167 tablet_bootstrap.cc:492] T 713ae1dd444544bdabdd4a771e109d90 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:27.412113 1167 ts_tablet_manager.cc:1403] T 713ae1dd444544bdabdd4a771e109d90 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.001s
I20251025 07:52:27.411096 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_c45e312143074d47be4fac483ee16136"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:27.412798 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c45e312143074d47be4fac483ee16136 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:27.412936 522 catalog_manager.cc:6719] ReplaceTablet: tablet 85fba8b37f2c4540aa81701052bf222a deleted and replaced by tablet 6a58931e7cd7454fb583f1c70cd2ad15
I20251025 07:52:27.413331 1167 raft_consensus.cc:359] T 713ae1dd444544bdabdd4a771e109d90 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.413494 1167 raft_consensus.cc:385] T 713ae1dd444544bdabdd4a771e109d90 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:27.413568 1167 raft_consensus.cc:740] T 713ae1dd444544bdabdd4a771e109d90 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:27.413672 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_618540173e8a45828c270476360228f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:27.413771 1167 consensus_queue.cc:260] T 713ae1dd444544bdabdd4a771e109d90 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:27.414177 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_618540173e8a45828c270476360228f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:27.414285 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fac7b819e2642378e0d3f9230d0259c" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:27.414475 914 raft_consensus.cc:3060] T 2fac7b819e2642378e0d3f9230d0259c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:27.415402 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 3775a2e08d4e40bd9f57e4cda759d535
I20251025 07:52:27.416380 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 3775a2e08d4e40bd9f57e4cda759d535 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:27.416544 1172 ts_tablet_manager.cc:1929] T 713ae1dd444544bdabdd4a771e109d90 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:27.416680 1172 log.cc:1199] T 713ae1dd444544bdabdd4a771e109d90 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/713ae1dd444544bdabdd4a771e109d90
I20251025 07:52:27.417052 1172 ts_tablet_manager.cc:1950] T 713ae1dd444544bdabdd4a771e109d90 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:27.417500 914 raft_consensus.cc:2468] T 2fac7b819e2642378e0d3f9230d0259c P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:27.418099 710 leader_election.cc:304] T 2fac7b819e2642378e0d3f9230d0259c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:27.418252 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 713ae1dd444544bdabdd4a771e109d90 (table default.Table_70998787ec5746b9bb56f6e5e642898e [id=034108fa919f469bb53041fd03a1b5d3]) successfully deleted
I20251025 07:52:27.418576 1178 raft_consensus.cc:2804] T 2fac7b819e2642378e0d3f9230d0259c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:27.418102 1167 ts_tablet_manager.cc:1434] T 713ae1dd444544bdabdd4a771e109d90 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20251025 07:52:27.419198 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 83084be89b2646b0bda208b3f6f60d77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:27.419340 1167 tablet_bootstrap.cc:492] T a231f3d31b204f7b82f018892ccc8435 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
W20251025 07:52:27.420480 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 2209089667dd4937abd96615f6929449 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2209089667dd4937abd96615f6929449 already in progress: creating tablet
I20251025 07:52:27.421636 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_a002bf5ad8ed49e4991ba95c2a739554"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:27.422227 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a002bf5ad8ed49e4991ba95c2a739554 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:27.423580 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_c361b0312923449da58cd2d6a012be15" } new_table_name: "default.Table_d3207dcbfb93438e9f51b211e8b1ebdc" modify_external_catalogs: true
I20251025 07:52:27.425119 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ec421eb820b4f40afcff5a5bf80c508 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:27.425388 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 85fba8b37f2c4540aa81701052bf222a with delete_type TABLET_DATA_DELETED (replaced by tablet 6a58931e7cd7454fb583f1c70cd2ad15) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:27.425422 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 3775a2e08d4e40bd9f57e4cda759d535 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:27.425460 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fac7b819e2642378e0d3f9230d0259c" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:27.425664 1048 raft_consensus.cc:3060] T 2fac7b819e2642378e0d3f9230d0259c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:27.426398 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_6e28f51cd66f471a8d55f19e6f5becff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:27.426870 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e28f51cd66f471a8d55f19e6f5becff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:27.428756 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_028b24c9775b49ea8adf1a62fe9df71d" } new_table_name: "default.Table_20b987167f0e4863824e7bb7c17e5c03" modify_external_catalogs: true
I20251025 07:52:27.428988 1048 raft_consensus.cc:2468] T 2fac7b819e2642378e0d3f9230d0259c P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:27.429461 741 tablet_service.cc:1552] Processing DeleteTablet for tablet 713ae1dd444544bdabdd4a771e109d90 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:27.429685 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 3775a2e08d4e40bd9f57e4cda759d535 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:27.429915 1171 tablet_replica.cc:333] T 713ae1dd444544bdabdd4a771e109d90 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:27.430083 1171 raft_consensus.cc:2243] T 713ae1dd444544bdabdd4a771e109d90 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:27.430307 1171 raft_consensus.cc:2272] T 713ae1dd444544bdabdd4a771e109d90 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:27.431049 1171 ts_tablet_manager.cc:1916] T 713ae1dd444544bdabdd4a771e109d90 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:27.431788 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ec421eb820b4f40afcff5a5bf80c508 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:27.432466 1169 tablet_bootstrap.cc:654] T 3d5e0dd14e474fe09993a03851f8b1f0 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:27.434329 755 tablet_service.cc:1505] Processing CreateTablet for tablet 5fca357d7e24494fb6af4bd73d98833c (DEFAULT_TABLE table=default.Table_400f06d166054eeb97890ad56453afb2 [id=5f20de141a6f452a82ed736f41c29d9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.434762 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fca357d7e24494fb6af4bd73d98833c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:27.436231 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 1483c40eee834f20a92e6b4cd89775b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1483c40eee834f20a92e6b4cd89775b6 already in progress: creating tablet
I20251025 07:52:27.430055 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_919e690ef3d041c2bcf9b0c9e59d9bc8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:27.436743 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_919e690ef3d041c2bcf9b0c9e59d9bc8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:27.437160 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 1aeaad5c5d01478bb997254ac372d9a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1aeaad5c5d01478bb997254ac372d9a6 already in progress: creating tablet
W20251025 07:52:27.437929 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 8c76d1b509f04ab1a10b83cff675c11e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c76d1b509f04ab1a10b83cff675c11e already in progress: creating tablet
W20251025 07:52:27.438681 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet cb824bd780524832b2b60f5618a39da5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb824bd780524832b2b60f5618a39da5 already in progress: creating tablet
W20251025 07:52:27.439402 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet fab67b56eaa44b54886c17181d8c52b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fab67b56eaa44b54886c17181d8c52b2 already in progress: creating tablet
I20251025 07:52:27.439911 1172 tablet_replica.cc:333] T 576c51d9a22d45f6a3642a7296b5bf23 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:27.440464 1172 raft_consensus.cc:2243] T 576c51d9a22d45f6a3642a7296b5bf23 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:27.440523 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 5fca357d7e24494fb6af4bd73d98833c (DEFAULT_TABLE table=default.Table_400f06d166054eeb97890ad56453afb2 [id=5f20de141a6f452a82ed736f41c29d9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.440950 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fca357d7e24494fb6af4bd73d98833c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.441011 1172 raft_consensus.cc:2272] T 576c51d9a22d45f6a3642a7296b5bf23 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:27.441577 1172 ts_tablet_manager.cc:1916] T 576c51d9a22d45f6a3642a7296b5bf23 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:27.448731 1178 raft_consensus.cc:697] T 2fac7b819e2642378e0d3f9230d0259c P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:27.449169 1178 consensus_queue.cc:237] T 2fac7b819e2642378e0d3f9230d0259c P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.450572 886 tablet_service.cc:1552] Processing DeleteTablet for tablet 85fba8b37f2c4540aa81701052bf222a with delete_type TABLET_DATA_DELETED (replaced by tablet 6a58931e7cd7454fb583f1c70cd2ad15) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:27.450666 1171 ts_tablet_manager.cc:1929] T 713ae1dd444544bdabdd4a771e109d90 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:27.450713 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 83084be89b2646b0bda208b3f6f60d77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:27.450970 1171 log.cc:1199] T 713ae1dd444544bdabdd4a771e109d90 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/713ae1dd444544bdabdd4a771e109d90
I20251025 07:52:27.451330 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_6921ba3d8feb4bd49739be0ba51dab61" } new_table_name: "default.Table_7f531c5ddd174b49b0117a226cd0d91f" modify_external_catalogs: true
I20251025 07:52:27.451424 1171 ts_tablet_manager.cc:1950] T 713ae1dd444544bdabdd4a771e109d90 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:27.452473 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 713ae1dd444544bdabdd4a771e109d90 (table default.Table_70998787ec5746b9bb56f6e5e642898e [id=034108fa919f469bb53041fd03a1b5d3]) successfully deleted
W20251025 07:52:27.452965 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 3775a2e08d4e40bd9f57e4cda759d535 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3775a2e08d4e40bd9f57e4cda759d535 already in progress: creating tablet
W20251025 07:52:27.453220 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 6ec421eb820b4f40afcff5a5bf80c508 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ec421eb820b4f40afcff5a5bf80c508 already in progress: creating tablet
I20251025 07:52:27.453821 524 master_service.cc:954] ReplaceTablet: received request to replace tablet 1d261e432cf5403ea162129706bc4d4d from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:27.453860 1167 tablet_bootstrap.cc:654] T a231f3d31b204f7b82f018892ccc8435 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:27.454728 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f3001314b56345f5a70314777cae0125 consensus state has no leader
I20251025 07:52:27.455148 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_c0432016895e4094b7ff582abe612bbd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:27.455663 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c0432016895e4094b7ff582abe612bbd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:27.457132 743 tablet_service.cc:1505] Processing CreateTablet for tablet d66ad049d5484228a268b0facb00ecf1 (DEFAULT_TABLE table=default.Table_56d474131b4047f49432e55467b20cdd [id=55440328803042a28df1b6869d279f59]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.457484 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d66ad049d5484228a268b0facb00ecf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.458485 741 tablet_service.cc:1505] Processing CreateTablet for tablet 5361f08f50b745eb98bf02e979d7c71e (DEFAULT_TABLE table=default.Table_4bb5ba285e1c46f8b698e4915e37fab9 [id=572bc5595be947468e24f2a9144f12a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.458868 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5361f08f50b745eb98bf02e979d7c71e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.458873 883 tablet_service.cc:1505] Processing CreateTablet for tablet d66ad049d5484228a268b0facb00ecf1 (DEFAULT_TABLE table=default.Table_56d474131b4047f49432e55467b20cdd [id=55440328803042a28df1b6869d279f59]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.459271 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d66ad049d5484228a268b0facb00ecf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.459678 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet fa77085e81c7440d9e255c44d1e80c26
I20251025 07:52:27.460367 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_45f85cb4514d4ec3b06cff95dd4a70f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:27.460877 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45f85cb4514d4ec3b06cff95dd4a70f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:27.462929 1166 tablet_bootstrap.cc:492] T 50433881c0eb4aa5a389b6f5b4a4337a P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:27.463081 1166 ts_tablet_manager.cc:1403] T 50433881c0eb4aa5a389b6f5b4a4337a P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.067s user 0.001s sys 0.004s
I20251025 07:52:27.463862 1166 raft_consensus.cc:359] T 50433881c0eb4aa5a389b6f5b4a4337a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:27.464037 1166 raft_consensus.cc:385] T 50433881c0eb4aa5a389b6f5b4a4337a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:27.464118 743 tablet_service.cc:1552] Processing DeleteTablet for tablet fa77085e81c7440d9e255c44d1e80c26 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:27.464131 1166 raft_consensus.cc:740] T 50433881c0eb4aa5a389b6f5b4a4337a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:27.464406 1166 consensus_queue.cc:260] T 50433881c0eb4aa5a389b6f5b4a4337a P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
W20251025 07:52:27.464900 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet fa77085e81c7440d9e255c44d1e80c26 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fa77085e81c7440d9e255c44d1e80c26 already in progress: creating tablet
I20251025 07:52:27.466843 1166 ts_tablet_manager.cc:1434] T 50433881c0eb4aa5a389b6f5b4a4337a P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251025 07:52:27.467146 1166 tablet_bootstrap.cc:492] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
W20251025 07:52:27.468349 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6897405bbc1c47a4ab0aa3af80e60e3d consensus state has no leader
I20251025 07:52:27.471544 525 catalog_manager.cc:5649] T 2fac7b819e2642378e0d3f9230d0259c P 23eeffd3de254ce18db5febfde971c6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 23eeffd3de254ce18db5febfde971c6b (127.0.104.1). New cstate: current_term: 1 leader_uuid: "23eeffd3de254ce18db5febfde971c6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: UNKNOWN } } }
I20251025 07:52:27.471820 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_503b86ee2ab545afa9829bebcc9ea11d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:27.471947 525 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 2fac7b819e2642378e0d3f9230d0259c (table default.Table_33b6fa5be1ab40fba35b3d5ef9ad9668 [id=6605e11e11804d408a6a4a9d403dde95]). Expected version 1 got 0
W20251025 07:52:27.472457 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_503b86ee2ab545afa9829bebcc9ea11d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:27.475358 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_0b0bd6b71b92454fb8cdf967ffd8349e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:27.476001 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b0bd6b71b92454fb8cdf967ffd8349e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:27.477996 755 tablet_service.cc:1505] Processing CreateTablet for tablet 8f2c8582fed34c848d9903b8ad8b4057 (DEFAULT_TABLE table=default.Table_aa90c6fe1e744173a6174066750dc583 [id=7c7fcd265e624d7fab74661db505d9c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.479647 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f2c8582fed34c848d9903b8ad8b4057. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.480695 741 tablet_service.cc:1505] Processing CreateTablet for tablet 50ae9bca6bed4f749d54191ac1543410 (DEFAULT_TABLE table=default.Table_b42187d20a7e4f8783a8d25acb249350 [id=a5c73cb1e5a349ca8675f4ba4cb8a82d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.480774 1167 tablet_bootstrap.cc:492] T a231f3d31b204f7b82f018892ccc8435 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:27.481603 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50ae9bca6bed4f749d54191ac1543410. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.481796 754 tablet_service.cc:1505] Processing CreateTablet for tablet fa998b9ca6724c4cb1eb68a3eecb0b11 (DEFAULT_TABLE table=default.Table_4976d4f5d05540258d144751d129bdb8 [id=8cde66661cf749919839da894ed4dc05]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.483177 1167 ts_tablet_manager.cc:1403] T a231f3d31b204f7b82f018892ccc8435 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.062s user 0.000s sys 0.004s
I20251025 07:52:27.485502 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_192549a3e67840f3aefc31029cf2ff52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:27.485971 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_192549a3e67840f3aefc31029cf2ff52 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:27.486295 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa998b9ca6724c4cb1eb68a3eecb0b11. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:27.489280 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3d5e0dd14e474fe09993a03851f8b1f0 consensus state has no leader
I20251025 07:52:27.490664 753 tablet_service.cc:1505] Processing CreateTablet for tablet 410ef150a7014c6c867a021b544a2f4e (DEFAULT_TABLE table=default.Table_39bd22f18d6540d3be94520671683032 [id=baf017a8539143ad9c8a443a9e1b6339]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.491258 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 410ef150a7014c6c867a021b544a2f4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.491179 739 tablet_service.cc:1505] Processing CreateTablet for tablet df1248f6dbd94af68c587da7d149e279 (DEFAULT_TABLE table=default.Table_73279bdee234454e9def34441da9579a [id=6430d82b82c042469a77ea9f4dad15fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.491521 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df1248f6dbd94af68c587da7d149e279. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.488673 743 tablet_service.cc:1505] Processing CreateTablet for tablet 3abfd52327a04ba88625f391a8658fbb (DEFAULT_TABLE table=default.Table_cfa54e1a04154e2eb8747fd3a4812883 [id=bf3a3f8dae184d999f2159dfcc7803b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.494238 523 master_service.cc:954] ReplaceTablet: received request to replace tablet b772025852794a49b072f37cccbd3291 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:27.496295 890 tablet_service.cc:1505] Processing CreateTablet for tablet 50ae9bca6bed4f749d54191ac1543410 (DEFAULT_TABLE table=default.Table_b42187d20a7e4f8783a8d25acb249350 [id=a5c73cb1e5a349ca8675f4ba4cb8a82d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.496568 885 tablet_service.cc:1505] Processing CreateTablet for tablet 410ef150a7014c6c867a021b544a2f4e (DEFAULT_TABLE table=default.Table_39bd22f18d6540d3be94520671683032 [id=baf017a8539143ad9c8a443a9e1b6339]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.497294 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3abfd52327a04ba88625f391a8658fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.497241 886 tablet_service.cc:1552] Processing DeleteTablet for tablet fa77085e81c7440d9e255c44d1e80c26 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:27.499783 891 tablet_service.cc:1505] Processing CreateTablet for tablet fa998b9ca6724c4cb1eb68a3eecb0b11 (DEFAULT_TABLE table=default.Table_4976d4f5d05540258d144751d129bdb8 [id=8cde66661cf749919839da894ed4dc05]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.500967 1166 tablet_bootstrap.cc:654] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:27.500931 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ec421eb820b4f40afcff5a5bf80c508 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:27.502693 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_e4cd0fd928264afea9fbdf342da2a781" } new_table_name: "default.Table_d771d85f5f6a4016bfa426dd9dbf586a" modify_external_catalogs: true
I20251025 07:52:27.504879 889 tablet_service.cc:1505] Processing CreateTablet for tablet 5fca357d7e24494fb6af4bd73d98833c (DEFAULT_TABLE table=default.Table_400f06d166054eeb97890ad56453afb2 [id=5f20de141a6f452a82ed736f41c29d9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.505800 894 tablet_service.cc:1505] Processing CreateTablet for tablet 8f2c8582fed34c848d9903b8ad8b4057 (DEFAULT_TABLE table=default.Table_aa90c6fe1e744173a6174066750dc583 [id=7c7fcd265e624d7fab74661db505d9c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.506228 892 tablet_service.cc:1505] Processing CreateTablet for tablet 3abfd52327a04ba88625f391a8658fbb (DEFAULT_TABLE table=default.Table_cfa54e1a04154e2eb8747fd3a4812883 [id=bf3a3f8dae184d999f2159dfcc7803b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.506577 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50ae9bca6bed4f749d54191ac1543410. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.506816 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa998b9ca6724c4cb1eb68a3eecb0b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.507014 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f2c8582fed34c848d9903b8ad8b4057. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.507308 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 410ef150a7014c6c867a021b544a2f4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.510262 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3abfd52327a04ba88625f391a8658fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.512056 1167 raft_consensus.cc:359] T a231f3d31b204f7b82f018892ccc8435 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.512300 1167 raft_consensus.cc:385] T a231f3d31b204f7b82f018892ccc8435 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:27.512373 1167 raft_consensus.cc:740] T a231f3d31b204f7b82f018892ccc8435 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:27.518020 1167 consensus_queue.cc:260] T a231f3d31b204f7b82f018892ccc8435 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.518710 1218 raft_consensus.cc:493] T 33c608beb4ef4a859a135b11e07c81a4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:27.518888 1218 raft_consensus.cc:515] T 33c608beb4ef4a859a135b11e07c81a4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.519629 1218 leader_election.cc:290] T 33c608beb4ef4a859a135b11e07c81a4 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:27.521076 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "33c608beb4ef4a859a135b11e07c81a4" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
W20251025 07:52:27.521896 847 leader_election.cc:343] T 33c608beb4ef4a859a135b11e07c81a4 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:27.522460 1167 ts_tablet_manager.cc:1434] T a231f3d31b204f7b82f018892ccc8435 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.039s user 0.002s sys 0.000s
I20251025 07:52:27.522377 886 tablet_service.cc:1505] Processing CreateTablet for tablet df1248f6dbd94af68c587da7d149e279 (DEFAULT_TABLE table=default.Table_73279bdee234454e9def34441da9579a [id=6430d82b82c042469a77ea9f4dad15fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.507330 884 tablet_service.cc:1505] Processing CreateTablet for tablet 5361f08f50b745eb98bf02e979d7c71e (DEFAULT_TABLE table=default.Table_4bb5ba285e1c46f8b698e4915e37fab9 [id=572bc5595be947468e24f2a9144f12a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.522993 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fca357d7e24494fb6af4bd73d98833c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.527541 1167 tablet_bootstrap.cc:492] T 3d5e0dd14e474fe09993a03851f8b1f0 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:27.529484 1028 tablet_service.cc:1505] Processing CreateTablet for tablet fa998b9ca6724c4cb1eb68a3eecb0b11 (DEFAULT_TABLE table=default.Table_4976d4f5d05540258d144751d129bdb8 [id=8cde66661cf749919839da894ed4dc05]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.529721 1167 tablet_bootstrap.cc:654] T 3d5e0dd14e474fe09993a03851f8b1f0 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:27.533070 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet fa77085e81c7440d9e255c44d1e80c26 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:27.533239 1167 tablet_bootstrap.cc:492] T 3d5e0dd14e474fe09993a03851f8b1f0 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:27.533380 1167 ts_tablet_manager.cc:1403] T 3d5e0dd14e474fe09993a03851f8b1f0 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20251025 07:52:27.533428 1020 tablet_service.cc:1505] Processing CreateTablet for tablet 5361f08f50b745eb98bf02e979d7c71e (DEFAULT_TABLE table=default.Table_4bb5ba285e1c46f8b698e4915e37fab9 [id=572bc5595be947468e24f2a9144f12a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.533952 1023 tablet_service.cc:1505] Processing CreateTablet for tablet 410ef150a7014c6c867a021b544a2f4e (DEFAULT_TABLE table=default.Table_39bd22f18d6540d3be94520671683032 [id=baf017a8539143ad9c8a443a9e1b6339]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.534258 1026 tablet_service.cc:1505] Processing CreateTablet for tablet d66ad049d5484228a268b0facb00ecf1 (DEFAULT_TABLE table=default.Table_56d474131b4047f49432e55467b20cdd [id=55440328803042a28df1b6869d279f59]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.534265 1167 raft_consensus.cc:359] T 3d5e0dd14e474fe09993a03851f8b1f0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.534435 1167 raft_consensus.cc:385] T 3d5e0dd14e474fe09993a03851f8b1f0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:27.534525 1167 raft_consensus.cc:740] T 3d5e0dd14e474fe09993a03851f8b1f0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:27.534536 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "33c608beb4ef4a859a135b11e07c81a4" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:27.534772 1167 consensus_queue.cc:260] T 3d5e0dd14e474fe09993a03851f8b1f0 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.534876 1019 tablet_service.cc:1505] Processing CreateTablet for tablet df1248f6dbd94af68c587da7d149e279 (DEFAULT_TABLE table=default.Table_73279bdee234454e9def34441da9579a [id=6430d82b82c042469a77ea9f4dad15fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.536183 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 3abfd52327a04ba88625f391a8658fbb (DEFAULT_TABLE table=default.Table_cfa54e1a04154e2eb8747fd3a4812883 [id=bf3a3f8dae184d999f2159dfcc7803b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.536449 1018 tablet_service.cc:1505] Processing CreateTablet for tablet 50ae9bca6bed4f749d54191ac1543410 (DEFAULT_TABLE table=default.Table_b42187d20a7e4f8783a8d25acb249350 [id=a5c73cb1e5a349ca8675f4ba4cb8a82d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.536770 1169 tablet_bootstrap.cc:492] T 3d5e0dd14e474fe09993a03851f8b1f0 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:27.538726 1167 ts_tablet_manager.cc:1434] T 3d5e0dd14e474fe09993a03851f8b1f0 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20251025 07:52:27.538915 1169 ts_tablet_manager.cc:1403] T 3d5e0dd14e474fe09993a03851f8b1f0 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.144s user 0.004s sys 0.000s
I20251025 07:52:27.539726 1169 raft_consensus.cc:359] T 3d5e0dd14e474fe09993a03851f8b1f0 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.540398 1169 raft_consensus.cc:385] T 3d5e0dd14e474fe09993a03851f8b1f0 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:27.540490 1169 raft_consensus.cc:740] T 3d5e0dd14e474fe09993a03851f8b1f0 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:27.541714 1167 tablet_bootstrap.cc:492] T 33c608beb4ef4a859a135b11e07c81a4 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:27.543147 1169 consensus_queue.cc:260] T 3d5e0dd14e474fe09993a03851f8b1f0 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.543669 1169 ts_tablet_manager.cc:1434] T 3d5e0dd14e474fe09993a03851f8b1f0 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251025 07:52:27.544166 1167 tablet_bootstrap.cc:654] T 33c608beb4ef4a859a135b11e07c81a4 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:27.549166 1169 tablet_bootstrap.cc:492] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:27.551410 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa998b9ca6724c4cb1eb68a3eecb0b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.554229 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 8f2c8582fed34c848d9903b8ad8b4057 (DEFAULT_TABLE table=default.Table_aa90c6fe1e744173a6174066750dc583 [id=7c7fcd265e624d7fab74661db505d9c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:27.555821 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 410ef150a7014c6c867a021b544a2f4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.570228 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d66ad049d5484228a268b0facb00ecf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.583108 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3abfd52327a04ba88625f391a8658fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.584717 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df1248f6dbd94af68c587da7d149e279. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.588970 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5361f08f50b745eb98bf02e979d7c71e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.589752 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5361f08f50b745eb98bf02e979d7c71e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.590545 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50ae9bca6bed4f749d54191ac1543410. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.591841 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f2c8582fed34c848d9903b8ad8b4057. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:27.597716 850 leader_election.cc:343] T 33c608beb4ef4a859a135b11e07c81a4 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:27.597900 850 leader_election.cc:304] T 33c608beb4ef4a859a135b11e07c81a4 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:27.598412 1218 raft_consensus.cc:2749] T 33c608beb4ef4a859a135b11e07c81a4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:27.599225 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df1248f6dbd94af68c587da7d149e279. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:27.611606 1247 raft_consensus.cc:493] T 576c51d9a22d45f6a3642a7296b5bf23 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:27.611795 1247 raft_consensus.cc:515] T 576c51d9a22d45f6a3642a7296b5bf23 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.612510 1247 leader_election.cc:290] T 576c51d9a22d45f6a3642a7296b5bf23 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:27.613668 982 leader_election.cc:343] T 576c51d9a22d45f6a3642a7296b5bf23 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: 576c51d9a22d45f6a3642a7296b5bf23
I20251025 07:52:27.615912 1172 ts_tablet_manager.cc:1929] T 576c51d9a22d45f6a3642a7296b5bf23 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:27.616044 1172 log.cc:1199] T 576c51d9a22d45f6a3642a7296b5bf23 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/576c51d9a22d45f6a3642a7296b5bf23
I20251025 07:52:27.616422 1172 ts_tablet_manager.cc:1950] T 576c51d9a22d45f6a3642a7296b5bf23 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:27.617419 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 576c51d9a22d45f6a3642a7296b5bf23 (table default.Table_14cbf20f7ff449499612663b54f0acd9 [id=e9343178deb142339649b0cc6d26a9af]) successfully deleted
W20251025 07:52:27.618443 983 leader_election.cc:343] T 576c51d9a22d45f6a3642a7296b5bf23 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 576c51d9a22d45f6a3642a7296b5bf23
I20251025 07:52:27.618589 983 leader_election.cc:304] T 576c51d9a22d45f6a3642a7296b5bf23 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:27.619251 1247 raft_consensus.cc:2749] T 576c51d9a22d45f6a3642a7296b5bf23 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:27.619869 1224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 07:52:27.620337 1224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 07:52:27.620497 1224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251025 07:52:27.634454 1224 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 07:52:27.634684 1224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 07:52:27.634840 1224 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 07:52:27.634986 1224 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 07:52:27.650575 1172 tablet_replica.cc:333] T 4829ec41391145cc84327d60048d9b0c P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:27.650815 1172 raft_consensus.cc:2243] T 4829ec41391145cc84327d60048d9b0c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:27.651017 1172 raft_consensus.cc:2272] T 4829ec41391145cc84327d60048d9b0c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:27.651585 1172 ts_tablet_manager.cc:1916] T 4829ec41391145cc84327d60048d9b0c P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:27.652004 1224 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:40091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:37421,127.0.104.61:42203,127.0.104.60:37611
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:37611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=38531
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 25 Oct 2025 07:43:22 UTC on 24a791456cd2
build id 8708
ASAN enabled
I20251025 07:52:27.654446 1224 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 07:52:27.655520 1166 tablet_bootstrap.cc:492] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:27.655519 1169 tablet_bootstrap.cc:654] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:27.655697 1166 ts_tablet_manager.cc:1403] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.189s user 0.007s sys 0.018s
I20251025 07:52:27.656616 1178 raft_consensus.cc:493] T a231f3d31b204f7b82f018892ccc8435 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:27.656558 1166 raft_consensus.cc:359] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.656826 1166 raft_consensus.cc:385] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:27.656904 1166 raft_consensus.cc:740] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:27.656808 1178 raft_consensus.cc:515] T a231f3d31b204f7b82f018892ccc8435 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.657194 1166 consensus_queue.cc:260] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.657287 1224 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251025 07:52:27.657699 1178 leader_election.cc:290] T a231f3d31b204f7b82f018892ccc8435 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:27.658528 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a231f3d31b204f7b82f018892ccc8435" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:27.659528 1166 ts_tablet_manager.cc:1434] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
W20251025 07:52:27.659533 712 leader_election.cc:343] T a231f3d31b204f7b82f018892ccc8435 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:27.660014 1166 tablet_bootstrap.cc:492] T b00153c97104470883f6089d71d23708 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:27.660491 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a231f3d31b204f7b82f018892ccc8435" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:27.661549 710 leader_election.cc:343] T a231f3d31b204f7b82f018892ccc8435 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:27.661674 710 leader_election.cc:304] T a231f3d31b204f7b82f018892ccc8435 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:27.662070 1178 raft_consensus.cc:2749] T a231f3d31b204f7b82f018892ccc8435 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:27.668979 1166 tablet_bootstrap.cc:654] T b00153c97104470883f6089d71d23708 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:27.673458 1247 raft_consensus.cc:493] T 713ae1dd444544bdabdd4a771e109d90 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:27.673632 1247 raft_consensus.cc:515] T 713ae1dd444544bdabdd4a771e109d90 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.674324 1247 leader_election.cc:290] T 713ae1dd444544bdabdd4a771e109d90 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:27.675616 982 leader_election.cc:343] T 713ae1dd444544bdabdd4a771e109d90 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: 713ae1dd444544bdabdd4a771e109d90
W20251025 07:52:27.676587 983 leader_election.cc:343] T 713ae1dd444544bdabdd4a771e109d90 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 713ae1dd444544bdabdd4a771e109d90
I20251025 07:52:27.676774 983 leader_election.cc:304] T 713ae1dd444544bdabdd4a771e109d90 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:27.677251 1247 raft_consensus.cc:2749] T 713ae1dd444544bdabdd4a771e109d90 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:27.687359 1166 tablet_bootstrap.cc:492] T b00153c97104470883f6089d71d23708 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:27.687592 1166 ts_tablet_manager.cc:1403] T b00153c97104470883f6089d71d23708 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.002s
I20251025 07:52:27.688683 1166 raft_consensus.cc:359] T b00153c97104470883f6089d71d23708 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.688943 1166 raft_consensus.cc:385] T b00153c97104470883f6089d71d23708 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:27.689061 1166 raft_consensus.cc:740] T b00153c97104470883f6089d71d23708 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:27.689409 1166 consensus_queue.cc:260] T b00153c97104470883f6089d71d23708 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.696147 1166 ts_tablet_manager.cc:1434] T b00153c97104470883f6089d71d23708 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.008s user 0.002s sys 0.002s
I20251025 07:52:27.696595 1166 tablet_bootstrap.cc:492] T 83084be89b2646b0bda208b3f6f60d77 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:27.707116 1172 ts_tablet_manager.cc:1929] T 4829ec41391145cc84327d60048d9b0c P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:27.707346 1172 log.cc:1199] T 4829ec41391145cc84327d60048d9b0c P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/4829ec41391145cc84327d60048d9b0c
W20251025 07:52:27.707329 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7fbae8be73bf4e18b5a5c4ec4cdebc12 consensus state has no leader
I20251025 07:52:27.707814 1172 ts_tablet_manager.cc:1950] T 4829ec41391145cc84327d60048d9b0c P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:27.708678 519 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 3d5e0dd14e474fe09993a03851f8b1f0 (table default.Table_20b987167f0e4863824e7bb7c17e5c03 [id=1b5db2f5053c4cb3a3b40eb109d5d5d9]). Expected version 2 got 0
I20251025 07:52:27.710713 1166 tablet_bootstrap.cc:654] T 83084be89b2646b0bda208b3f6f60d77 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:27.713398 519 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 50ffbcdd4a9247bf99b8e2361bc02ef5 (table default.Table_9f0ae2442a334cf8ab13f2eba387ebf3 [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 3 got 0
I20251025 07:52:27.715932 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_a002bf5ad8ed49e4991ba95c2a739554" } modify_external_catalogs: true
I20251025 07:52:27.716130 1172 tablet_replica.cc:333] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:27.716269 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_a002bf5ad8ed49e4991ba95c2a739554" } modify_external_catalogs: true
I20251025 07:52:27.716356 1172 raft_consensus.cc:2243] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:27.718410 1172 raft_consensus.cc:2272] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:27.718729 1315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 07:52:27.719298 1172 ts_tablet_manager.cc:1916] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:27.723330 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:43563, user_credentials={real_user=slave}} blocked reactor thread for 54768.3us
I20251025 07:52:27.725160 1170 ts_tablet_manager.cc:1929] T 785d0ce85b234826ba8021fbd182bf3c P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:27.727469 1166 tablet_bootstrap.cc:492] T 83084be89b2646b0bda208b3f6f60d77 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:27.727685 1166 ts_tablet_manager.cc:1403] T 83084be89b2646b0bda208b3f6f60d77 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.001s
I20251025 07:52:27.728756 1166 raft_consensus.cc:359] T 83084be89b2646b0bda208b3f6f60d77 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.729024 1166 raft_consensus.cc:385] T 83084be89b2646b0bda208b3f6f60d77 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:27.729136 1166 raft_consensus.cc:740] T 83084be89b2646b0bda208b3f6f60d77 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:27.729449 1166 consensus_queue.cc:260] T 83084be89b2646b0bda208b3f6f60d77 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:27.730437 511 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:37611, user_credentials={real_user=slave}} blocked reactor thread for 100906us
I20251025 07:52:27.731123 1172 ts_tablet_manager.cc:1929] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:27.731256 1172 log.cc:1199] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/19918a3c1f1d4ed6a78d5b4f0f4e38b3
I20251025 07:52:27.731505 1166 ts_tablet_manager.cc:1434] T 83084be89b2646b0bda208b3f6f60d77 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20251025 07:52:27.731698 1172 ts_tablet_manager.cc:1950] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:27.731846 1166 tablet_bootstrap.cc:492] T 94408a3a90224645ba3cacccf5de33f1 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
W20251025 07:52:27.732188 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 21e52928580c4762b0ae7152490ef955 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21e52928580c4762b0ae7152490ef955 already in progress: creating tablet
I20251025 07:52:27.732470 1218 raft_consensus.cc:493] T 3d5e0dd14e474fe09993a03851f8b1f0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 07:52:27.732503 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet d37758f606344127b3db74c87babe69c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d37758f606344127b3db74c87babe69c already in progress: creating tablet
I20251025 07:52:27.732600 1172 tablet_replica.cc:333] T 33c608beb4ef4a859a135b11e07c81a4 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:27.732685 1218 raft_consensus.cc:515] T 3d5e0dd14e474fe09993a03851f8b1f0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.732867 1172 raft_consensus.cc:2243] T 33c608beb4ef4a859a135b11e07c81a4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251025 07:52:27.733284 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 0b21b7da1e2f47829c3f6cdaee689e8a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b21b7da1e2f47829c3f6cdaee689e8a already in progress: creating tablet
I20251025 07:52:27.733381 1172 raft_consensus.cc:2272] T 33c608beb4ef4a859a135b11e07c81a4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:27.733413 1218 leader_election.cc:290] T 3d5e0dd14e474fe09993a03851f8b1f0 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
W20251025 07:52:27.733505 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet f170424be2f643e7af499834955472b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f170424be2f643e7af499834955472b0 already in progress: creating tablet
I20251025 07:52:27.733901 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 4829ec41391145cc84327d60048d9b0c (table default.Table_45b1ec3be7e9460296fe5b819beaa6ea [id=94a9a4e752b547afb3d4d0b6806bdb54]) successfully deleted
I20251025 07:52:27.733932 1172 ts_tablet_manager.cc:1916] T 33c608beb4ef4a859a135b11e07c81a4 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:27.734089 1166 tablet_bootstrap.cc:654] T 94408a3a90224645ba3cacccf5de33f1 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:27.734223 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 19918a3c1f1d4ed6a78d5b4f0f4e38b3 (table default.Table_b1c43f4e0ede4adcba8b010f04bdb72a [id=ee3cc33c49494767b860c9f1b7fa0d4d]) successfully deleted
W20251025 07:52:27.734465 511 consensus_peers.cc:597] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 -> Peer b4a999d754c941ec850c143e4e24c407 (127.0.104.60:37611): Couldn't send request to peer b4a999d754c941ec850c143e4e24c407. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:37611: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20251025 07:52:27.728749 1247 raft_consensus.cc:493] T 299572b2dbe244918dcebc5ea5f3c45a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:27.734783 1247 raft_consensus.cc:515] T 299572b2dbe244918dcebc5ea5f3c45a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.734547 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d5e0dd14e474fe09993a03851f8b1f0" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:27.735116 1048 raft_consensus.cc:2468] T 3d5e0dd14e474fe09993a03851f8b1f0 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:27.735474 1247 leader_election.cc:290] T 299572b2dbe244918dcebc5ea5f3c45a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:27.735586 850 leader_election.cc:304] T 3d5e0dd14e474fe09993a03851f8b1f0 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:27.735802 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d5e0dd14e474fe09993a03851f8b1f0" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
W20251025 07:52:27.735883 1317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 07:52:27.735967 1218 raft_consensus.cc:2804] T 3d5e0dd14e474fe09993a03851f8b1f0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:27.736037 775 raft_consensus.cc:2468] T 3d5e0dd14e474fe09993a03851f8b1f0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:27.736060 1218 raft_consensus.cc:493] T 3d5e0dd14e474fe09993a03851f8b1f0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:27.736167 1218 raft_consensus.cc:3060] T 3d5e0dd14e474fe09993a03851f8b1f0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:27.737006 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "299572b2dbe244918dcebc5ea5f3c45a" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:27.737214 914 raft_consensus.cc:2468] T 299572b2dbe244918dcebc5ea5f3c45a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:27.737680 983 leader_election.cc:304] T 299572b2dbe244918dcebc5ea5f3c45a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:27.738446 1247 raft_consensus.cc:2804] T 299572b2dbe244918dcebc5ea5f3c45a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:27.738554 1247 raft_consensus.cc:493] T 299572b2dbe244918dcebc5ea5f3c45a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:27.738624 1247 raft_consensus.cc:3060] T 299572b2dbe244918dcebc5ea5f3c45a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:27.738994 1218 raft_consensus.cc:515] T 3d5e0dd14e474fe09993a03851f8b1f0 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.740685 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d5e0dd14e474fe09993a03851f8b1f0" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:27.740895 1048 raft_consensus.cc:3060] T 3d5e0dd14e474fe09993a03851f8b1f0 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:27.741308 1247 raft_consensus.cc:515] T 299572b2dbe244918dcebc5ea5f3c45a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.742384 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "299572b2dbe244918dcebc5ea5f3c45a" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:27.742579 775 raft_consensus.cc:2468] T 299572b2dbe244918dcebc5ea5f3c45a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:27.743568 1247 leader_election.cc:290] T 299572b2dbe244918dcebc5ea5f3c45a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:27.744376 1218 leader_election.cc:290] T 3d5e0dd14e474fe09993a03851f8b1f0 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:27.744482 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d5e0dd14e474fe09993a03851f8b1f0" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:27.744717 776 raft_consensus.cc:3060] T 3d5e0dd14e474fe09993a03851f8b1f0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:27.743722 1048 raft_consensus.cc:2468] T 3d5e0dd14e474fe09993a03851f8b1f0 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:27.745373 850 leader_election.cc:304] T 3d5e0dd14e474fe09993a03851f8b1f0 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:27.745720 1218 raft_consensus.cc:2804] T 3d5e0dd14e474fe09993a03851f8b1f0 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:27.746469 1172 ts_tablet_manager.cc:1929] T 33c608beb4ef4a859a135b11e07c81a4 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:27.746542 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "299572b2dbe244918dcebc5ea5f3c45a" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:27.746771 914 raft_consensus.cc:3060] T 299572b2dbe244918dcebc5ea5f3c45a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:27.748343 776 raft_consensus.cc:2468] T 3d5e0dd14e474fe09993a03851f8b1f0 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:27.750329 914 raft_consensus.cc:2468] T 299572b2dbe244918dcebc5ea5f3c45a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:27.750905 983 leader_election.cc:304] T 299572b2dbe244918dcebc5ea5f3c45a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:27.744504 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "299572b2dbe244918dcebc5ea5f3c45a" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:27.751111 775 raft_consensus.cc:3060] T 299572b2dbe244918dcebc5ea5f3c45a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:27.752496 1247 raft_consensus.cc:2804] T 299572b2dbe244918dcebc5ea5f3c45a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:27.753262 1170 log.cc:1199] T 785d0ce85b234826ba8021fbd182bf3c P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/785d0ce85b234826ba8021fbd182bf3c
I20251025 07:52:27.746625 1172 log.cc:1199] T 33c608beb4ef4a859a135b11e07c81a4 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/33c608beb4ef4a859a135b11e07c81a4
I20251025 07:52:27.753937 1172 ts_tablet_manager.cc:1950] T 33c608beb4ef4a859a135b11e07c81a4 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:27.755069 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 33c608beb4ef4a859a135b11e07c81a4 (table default.Table_7a19acf479d64317bd910c04b08a8b4f [id=8bbb3b9204634ee6a04f5a4753739476]) successfully deleted
I20251025 07:52:27.755196 775 raft_consensus.cc:2468] T 299572b2dbe244918dcebc5ea5f3c45a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:27.756150 1172 tablet_replica.cc:333] T 3c3eb43209f24592820638fc25486b78 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:27.756336 1172 raft_consensus.cc:2243] T 3c3eb43209f24592820638fc25486b78 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:27.756536 1172 raft_consensus.cc:2272] T 3c3eb43209f24592820638fc25486b78 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:27.757329 1169 tablet_bootstrap.cc:492] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:27.757548 1169 ts_tablet_manager.cc:1403] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.213s user 0.004s sys 0.001s
I20251025 07:52:27.757998 1172 ts_tablet_manager.cc:1916] T 3c3eb43209f24592820638fc25486b78 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:27.758013 517 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 50ffbcdd4a9247bf99b8e2361bc02ef5 (table default.Table_9f0ae2442a334cf8ab13f2eba387ebf3 [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 3 got 0
I20251025 07:52:27.758755 1169 raft_consensus.cc:359] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.758937 1169 raft_consensus.cc:385] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:27.759025 1169 raft_consensus.cc:740] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:27.759114 517 master_service.cc:954] ReplaceTablet: received request to replace tablet 2784bbfb97d5466f9ec856add7d0b29a from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:27.759275 1169 consensus_queue.cc:260] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.760931 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_e7d32e91b9c64ce1b3fcff3d806e959f" } modify_external_catalogs: true
I20251025 07:52:27.761152 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_e7d32e91b9c64ce1b3fcff3d806e959f" } modify_external_catalogs: true
I20251025 07:52:27.762990 884 tablet_service.cc:1552] Processing DeleteTablet for tablet b00153c97104470883f6089d71d23708 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:27.763240 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 1d261e432cf5403ea162129706bc4d4d
I20251025 07:52:27.763747 524 catalog_manager.cc:6719] ReplaceTablet: tablet 1d261e432cf5403ea162129706bc4d4d deleted and replaced by tablet e54fe7cbcffa40c4b04144b47bee2b48
I20251025 07:52:27.765491 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_21a2393beb5d49a8ba839143c8328569" } modify_external_catalogs: true
I20251025 07:52:27.765683 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_21a2393beb5d49a8ba839143c8328569" } modify_external_catalogs: true
I20251025 07:52:27.766886 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d261e432cf5403ea162129706bc4d4d with delete_type TABLET_DATA_DELETED (replaced by tablet e54fe7cbcffa40c4b04144b47bee2b48) from {username='slave'} at 127.0.0.1:48830
W20251025 07:52:27.767784 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 1d261e432cf5403ea162129706bc4d4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d261e432cf5403ea162129706bc4d4d already in progress: creating tablet
I20251025 07:52:27.774016 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d261e432cf5403ea162129706bc4d4d with delete_type TABLET_DATA_DELETED (replaced by tablet e54fe7cbcffa40c4b04144b47bee2b48) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:27.775467 1166 tablet_bootstrap.cc:492] T 94408a3a90224645ba3cacccf5de33f1 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:27.775637 1166 ts_tablet_manager.cc:1403] T 94408a3a90224645ba3cacccf5de33f1 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.044s user 0.000s sys 0.003s
I20251025 07:52:27.776193 1172 ts_tablet_manager.cc:1929] T 3c3eb43209f24592820638fc25486b78 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:27.776341 1172 log.cc:1199] T 3c3eb43209f24592820638fc25486b78 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/3c3eb43209f24592820638fc25486b78
I20251025 07:52:27.776602 1166 raft_consensus.cc:359] T 94408a3a90224645ba3cacccf5de33f1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.776722 1172 ts_tablet_manager.cc:1950] T 3c3eb43209f24592820638fc25486b78 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:27.776818 1166 raft_consensus.cc:385] T 94408a3a90224645ba3cacccf5de33f1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:27.776922 1166 raft_consensus.cc:740] T 94408a3a90224645ba3cacccf5de33f1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:27.777176 1166 consensus_queue.cc:260] T 94408a3a90224645ba3cacccf5de33f1 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.778106 1172 tablet_replica.cc:333] T 50433881c0eb4aa5a389b6f5b4a4337a P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:27.778332 1172 raft_consensus.cc:2243] T 50433881c0eb4aa5a389b6f5b4a4337a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:27.778419 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d261e432cf5403ea162129706bc4d4d with delete_type TABLET_DATA_DELETED (replaced by tablet e54fe7cbcffa40c4b04144b47bee2b48) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:27.778488 1172 raft_consensus.cc:2272] T 50433881c0eb4aa5a389b6f5b4a4337a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:27.779625 1172 ts_tablet_manager.cc:1916] T 50433881c0eb4aa5a389b6f5b4a4337a P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:27.787169 1166 ts_tablet_manager.cc:1434] T 94408a3a90224645ba3cacccf5de33f1 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.011s user 0.002s sys 0.002s
I20251025 07:52:27.787458 1172 ts_tablet_manager.cc:1929] T 50433881c0eb4aa5a389b6f5b4a4337a P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:27.787612 1172 log.cc:1199] T 50433881c0eb4aa5a389b6f5b4a4337a P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/50433881c0eb4aa5a389b6f5b4a4337a
I20251025 07:52:27.788026 1172 ts_tablet_manager.cc:1950] T 50433881c0eb4aa5a389b6f5b4a4337a P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:27.788256 1166 tablet_bootstrap.cc:492] T 9bf486d82b984d2c89ee32e84a85894f P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:27.789072 1172 tablet_replica.cc:333] T 2947502815d742e89cacb474b7fbb249 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:27.789311 1172 raft_consensus.cc:2243] T 2947502815d742e89cacb474b7fbb249 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:27.789510 1172 raft_consensus.cc:2272] T 2947502815d742e89cacb474b7fbb249 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:27.786723 1324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 07:52:27.790227 1166 tablet_bootstrap.cc:654] T 9bf486d82b984d2c89ee32e84a85894f P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:27.790453 1224 server_base.cc:1047] running on GCE node
I20251025 07:52:27.790679 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 3c3eb43209f24592820638fc25486b78 (table default.Table_9b8e648894e14f8595796e4698407495 [id=89d0b4e609a74dde8540eb740446be5a]) successfully deleted
W20251025 07:52:27.791090 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 02e4c4d6ffb34cecaf17beb5ba197e81 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02e4c4d6ffb34cecaf17beb5ba197e81 already in progress: creating tablet
I20251025 07:52:27.791111 1172 ts_tablet_manager.cc:1916] T 2947502815d742e89cacb474b7fbb249 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:27.791805 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 50433881c0eb4aa5a389b6f5b4a4337a (table default.Table_b42187d20a7e4f8783a8d25acb249350 [id=a5c73cb1e5a349ca8675f4ba4cb8a82d]) successfully deleted
I20251025 07:52:27.797034 1170 ts_tablet_manager.cc:1950] T 785d0ce85b234826ba8021fbd182bf3c P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:27.798557 1172 ts_tablet_manager.cc:1929] T 2947502815d742e89cacb474b7fbb249 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:27.798705 1172 log.cc:1199] T 2947502815d742e89cacb474b7fbb249 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/2947502815d742e89cacb474b7fbb249
I20251025 07:52:27.799121 1172 ts_tablet_manager.cc:1950] T 2947502815d742e89cacb474b7fbb249 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:27.801433 1172 tablet_replica.cc:333] T 83084be89b2646b0bda208b3f6f60d77 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:27.801717 1172 raft_consensus.cc:2243] T 83084be89b2646b0bda208b3f6f60d77 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:27.801909 1172 raft_consensus.cc:2272] T 83084be89b2646b0bda208b3f6f60d77 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:27.803117 1172 ts_tablet_manager.cc:1916] T 83084be89b2646b0bda208b3f6f60d77 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:27.804675 1224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 07:52:27.798394 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 785d0ce85b234826ba8021fbd182bf3c (table default.Table_e0e89ced2ea94c139ed635849e78cb35 [id=807c428924b74fde847c2becad78d4ef]) successfully deleted
I20251025 07:52:27.811065 1224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 07:52:27.811827 1172 ts_tablet_manager.cc:1929] T 83084be89b2646b0bda208b3f6f60d77 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:27.811970 1172 log.cc:1199] T 83084be89b2646b0bda208b3f6f60d77 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/83084be89b2646b0bda208b3f6f60d77
I20251025 07:52:27.808247 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 2947502815d742e89cacb474b7fbb249 (table default.Table_c96e62972b3c4ee48b6c3ea90831a34f [id=e7a5b463914840c980a306b7d46ea871]) successfully deleted
I20251025 07:52:27.812288 1172 ts_tablet_manager.cc:1950] T 83084be89b2646b0bda208b3f6f60d77 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
W20251025 07:52:27.812484 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet a13bf4d7eb1440f7a28b85cb1bf99667 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a13bf4d7eb1440f7a28b85cb1bf99667 already in progress: creating tablet
W20251025 07:52:27.812816 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 2209089667dd4937abd96615f6929449 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2209089667dd4937abd96615f6929449 already in progress: creating tablet
W20251025 07:52:27.813136 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 3775a2e08d4e40bd9f57e4cda759d535 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3775a2e08d4e40bd9f57e4cda759d535 already in progress: creating tablet
W20251025 07:52:27.813400 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 85fba8b37f2c4540aa81701052bf222a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 85fba8b37f2c4540aa81701052bf222a already in progress: creating tablet
I20251025 07:52:27.813696 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 83084be89b2646b0bda208b3f6f60d77 (table default.Table_7c2f0e6e0edf4ae6920557e0d5130468 [id=31c55cf000ed4e4dbefe4a407fa1828c]) successfully deleted
I20251025 07:52:27.813791 1172 tablet_replica.cc:333] T b00153c97104470883f6089d71d23708 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
W20251025 07:52:27.814024 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet fa77085e81c7440d9e255c44d1e80c26 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fa77085e81c7440d9e255c44d1e80c26 already in progress: creating tablet
I20251025 07:52:27.814072 1172 raft_consensus.cc:2243] T b00153c97104470883f6089d71d23708 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:27.814409 1224 hybrid_clock.cc:648] HybridClock initialized: now 1761378747814245 us; error 152 us; skew 500 ppm
W20251025 07:52:27.814410 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 6ec421eb820b4f40afcff5a5bf80c508 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ec421eb820b4f40afcff5a5bf80c508 already in progress: creating tablet
I20251025 07:52:27.815032 1172 raft_consensus.cc:2272] T b00153c97104470883f6089d71d23708 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:27.815035 1224 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 07:52:27.817066 1170 tablet_replica.cc:333] T 9cf88862e0934634b3dd172e6b40a0ba P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:27.817407 1170 raft_consensus.cc:2243] T 9cf88862e0934634b3dd172e6b40a0ba P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:27.817667 1170 raft_consensus.cc:2272] T 9cf88862e0934634b3dd172e6b40a0ba P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:27.818723 1170 ts_tablet_manager.cc:1916] T 9cf88862e0934634b3dd172e6b40a0ba P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:27.820546 1331 raft_consensus.cc:493] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:27.820992 1331 raft_consensus.cc:515] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.823314 1331 leader_election.cc:290] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:27.824841 1172 ts_tablet_manager.cc:1916] T b00153c97104470883f6089d71d23708 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:27.824868 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_b42187d20a7e4f8783a8d25acb249350" } modify_external_catalogs: true
I20251025 07:52:27.825116 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_b42187d20a7e4f8783a8d25acb249350" } modify_external_catalogs: true
I20251025 07:52:27.825326 1331 raft_consensus.cc:493] T 4829ec41391145cc84327d60048d9b0c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:27.825470 1331 raft_consensus.cc:515] T 4829ec41391145cc84327d60048d9b0c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.825996 1331 leader_election.cc:290] T 4829ec41391145cc84327d60048d9b0c P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:27.826521 1224 webserver.cc:492] Webserver started at http://127.0.104.60:38531/ using document root <none> and password file <none>
I20251025 07:52:27.827668 1224 fs_manager.cc:362] Metadata directory not provided
I20251025 07:52:27.829530 1224 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 07:52:27.831938 525 master_service.cc:954] ReplaceTablet: received request to replace tablet cf2cab1552bd4e7394797c93b149f450 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:27.832499 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_054db0ca31bd42ebb5d61ec81db8146b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:27.833073 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_054db0ca31bd42ebb5d61ec81db8146b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:27.836290 1172 ts_tablet_manager.cc:1929] T b00153c97104470883f6089d71d23708 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:27.836416 1172 log.cc:1199] T b00153c97104470883f6089d71d23708 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/b00153c97104470883f6089d71d23708
I20251025 07:52:27.836802 1172 ts_tablet_manager.cc:1950] T b00153c97104470883f6089d71d23708 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
W20251025 07:52:27.839018 983 leader_election.cc:343] T 4829ec41391145cc84327d60048d9b0c P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 4829ec41391145cc84327d60048d9b0c
W20251025 07:52:27.839694 983 leader_election.cc:343] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 19918a3c1f1d4ed6a78d5b4f0f4e38b3
I20251025 07:52:27.842654 1247 raft_consensus.cc:697] T 299572b2dbe244918dcebc5ea5f3c45a P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:27.843108 1247 consensus_queue.cc:237] T 299572b2dbe244918dcebc5ea5f3c45a P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.844666 1169 ts_tablet_manager.cc:1434] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.087s user 0.001s sys 0.004s
I20251025 07:52:27.845599 1169 tablet_bootstrap.cc:492] T a231f3d31b204f7b82f018892ccc8435 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:27.847714 1169 tablet_bootstrap.cc:654] T a231f3d31b204f7b82f018892ccc8435 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:27.849449 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19918a3c1f1d4ed6a78d5b4f0f4e38b3" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
W20251025 07:52:27.850893 982 leader_election.cc:343] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:27.851125 982 leader_election.cc:304] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
W20251025 07:52:27.851680 982 leader_election.cc:343] T 4829ec41391145cc84327d60048d9b0c P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: 4829ec41391145cc84327d60048d9b0c
I20251025 07:52:27.851884 982 leader_election.cc:304] T 4829ec41391145cc84327d60048d9b0c P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:27.862828 1321 raft_consensus.cc:2749] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:27.863188 1321 raft_consensus.cc:2749] T 4829ec41391145cc84327d60048d9b0c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:27.865231 1224 fs_manager.cc:714] Time spent opening directory manager: real 0.027s user 0.004s sys 0.004s
I20251025 07:52:27.868006 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_c82c2dad12af41788d8959f7bd9f92b0" } new_table_name: "default.Table_3bd421885a1b4e06b11d12780c34cad4" modify_external_catalogs: true
I20251025 07:52:27.878942 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet b00153c97104470883f6089d71d23708 (table default.Table_b330f1a41da2427995e699545d5fc49b [id=7ec5f704dc254c11b865c660ffabc4fe]) successfully deleted
W20251025 07:52:27.879323 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 1d261e432cf5403ea162129706bc4d4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d261e432cf5403ea162129706bc4d4d already in progress: creating tablet
I20251025 07:52:27.881106 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_db98182d07a54725823623925f51a0cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:27.881824 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db98182d07a54725823623925f51a0cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:27.884296 1166 tablet_bootstrap.cc:492] T 9bf486d82b984d2c89ee32e84a85894f P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:27.884931 1166 ts_tablet_manager.cc:1403] T 9bf486d82b984d2c89ee32e84a85894f P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.097s user 0.021s sys 0.025s
I20251025 07:52:27.886709 1166 raft_consensus.cc:359] T 9bf486d82b984d2c89ee32e84a85894f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.887248 1166 raft_consensus.cc:385] T 9bf486d82b984d2c89ee32e84a85894f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:27.887387 1166 raft_consensus.cc:740] T 9bf486d82b984d2c89ee32e84a85894f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:27.887717 1166 consensus_queue.cc:260] T 9bf486d82b984d2c89ee32e84a85894f P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.888747 1166 ts_tablet_manager.cc:1434] T 9bf486d82b984d2c89ee32e84a85894f P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251025 07:52:27.889144 1166 tablet_bootstrap.cc:492] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:27.891230 1166 tablet_bootstrap.cc:654] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:27.892910 1336 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 07:52:27.896600 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet b772025852794a49b072f37cccbd3291
I20251025 07:52:27.897112 523 catalog_manager.cc:6719] ReplaceTablet: tablet b772025852794a49b072f37cccbd3291 deleted and replaced by tablet f97acc9a638b4d5a968dcc3412a384bc
I20251025 07:52:27.897320 523 master_service.cc:954] ReplaceTablet: received request to replace tablet d5e5e5bbcf9f43118eea502b69880af5 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:27.898759 884 tablet_service.cc:1552] Processing DeleteTablet for tablet b772025852794a49b072f37cccbd3291 with delete_type TABLET_DATA_DELETED (replaced by tablet f97acc9a638b4d5a968dcc3412a384bc) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:27.898841 1224 fs_manager.cc:730] Time spent opening block manager: real 0.025s user 0.003s sys 0.003s
I20251025 07:52:27.899327 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet b772025852794a49b072f37cccbd3291 with delete_type TABLET_DATA_DELETED (replaced by tablet f97acc9a638b4d5a968dcc3412a384bc) from {username='slave'} at 127.0.0.1:47408
W20251025 07:52:27.899730 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet b772025852794a49b072f37cccbd3291 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b772025852794a49b072f37cccbd3291 already in progress: creating tablet
I20251025 07:52:27.899075 1224 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-2/data,/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-2/wal
uuid: "b4a999d754c941ec850c143e4e24c407"
format_stamp: "Formatted at 2025-10-25 07:52:21 on dist-test-slave-tm48"
I20251025 07:52:27.903651 753 tablet_service.cc:1552] Processing DeleteTablet for tablet b772025852794a49b072f37cccbd3291 with delete_type TABLET_DATA_DELETED (replaced by tablet f97acc9a638b4d5a968dcc3412a384bc) from {username='slave'} at 127.0.0.1:48830
W20251025 07:52:27.904356 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet b772025852794a49b072f37cccbd3291 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b772025852794a49b072f37cccbd3291 already in progress: creating tablet
I20251025 07:52:27.904879 1224 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 07:52:27.905596 1169 tablet_bootstrap.cc:492] T a231f3d31b204f7b82f018892ccc8435 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:27.905771 1169 ts_tablet_manager.cc:1403] T a231f3d31b204f7b82f018892ccc8435 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.061s user 0.000s sys 0.003s
I20251025 07:52:27.906584 1169 raft_consensus.cc:359] T a231f3d31b204f7b82f018892ccc8435 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.906800 1169 raft_consensus.cc:385] T a231f3d31b204f7b82f018892ccc8435 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:27.906870 1169 raft_consensus.cc:740] T a231f3d31b204f7b82f018892ccc8435 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:27.907248 1218 raft_consensus.cc:697] T 3d5e0dd14e474fe09993a03851f8b1f0 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:27.910692 1218 consensus_queue.cc:237] T 3d5e0dd14e474fe09993a03851f8b1f0 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.913674 1169 consensus_queue.cc:260] T a231f3d31b204f7b82f018892ccc8435 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.914036 1167 tablet_bootstrap.cc:492] T 33c608beb4ef4a859a135b11e07c81a4 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:27.914290 1167 ts_tablet_manager.cc:1403] T 33c608beb4ef4a859a135b11e07c81a4 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.373s user 0.055s sys 0.109s
I20251025 07:52:27.915464 1167 raft_consensus.cc:359] T 33c608beb4ef4a859a135b11e07c81a4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.915719 1167 raft_consensus.cc:385] T 33c608beb4ef4a859a135b11e07c81a4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:27.915803 1167 raft_consensus.cc:740] T 33c608beb4ef4a859a135b11e07c81a4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:27.916173 1167 consensus_queue.cc:260] T 33c608beb4ef4a859a135b11e07c81a4 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:27.917644 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f3001314b56345f5a70314777cae0125 consensus state has no leader
I20251025 07:52:27.923244 1167 ts_tablet_manager.cc:1434] T 33c608beb4ef4a859a135b11e07c81a4 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251025 07:52:27.922104 1169 ts_tablet_manager.cc:1434] T a231f3d31b204f7b82f018892ccc8435 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20251025 07:52:27.924321 519 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 47d0abb3b7f74a9ea14e7f86b6c6a963
I20251025 07:52:27.924623 1169 tablet_bootstrap.cc:492] T 3c3eb43209f24592820638fc25486b78 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:27.926517 1169 tablet_bootstrap.cc:654] T 3c3eb43209f24592820638fc25486b78 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:27.928970 1170 ts_tablet_manager.cc:1929] T 9cf88862e0934634b3dd172e6b40a0ba P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:27.929124 1170 log.cc:1199] T 9cf88862e0934634b3dd172e6b40a0ba P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/9cf88862e0934634b3dd172e6b40a0ba
I20251025 07:52:27.929515 1170 ts_tablet_manager.cc:1950] T 9cf88862e0934634b3dd172e6b40a0ba P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:27.930922 1170 tablet_replica.cc:333] T 4829ec41391145cc84327d60048d9b0c P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:27.929389 1167 tablet_bootstrap.cc:492] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:27.931356 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 9cf88862e0934634b3dd172e6b40a0ba (table default.Table_1eb640b2196849a8b0136b390deea1c2 [id=ec85197b425f466ba7cd69e27caf6c93]) successfully deleted
W20251025 07:52:27.931640 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 0b21b7da1e2f47829c3f6cdaee689e8a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b21b7da1e2f47829c3f6cdaee689e8a already in progress: creating tablet
I20251025 07:52:27.931660 1170 raft_consensus.cc:2243] T 4829ec41391145cc84327d60048d9b0c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:27.931849 1170 raft_consensus.cc:2272] T 4829ec41391145cc84327d60048d9b0c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:27.931901 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet f170424be2f643e7af499834955472b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f170424be2f643e7af499834955472b0 already in progress: creating tablet
I20251025 07:52:27.932384 1170 ts_tablet_manager.cc:1916] T 4829ec41391145cc84327d60048d9b0c P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:27.933425 1166 tablet_bootstrap.cc:492] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:27.933607 1166 ts_tablet_manager.cc:1403] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.045s user 0.023s sys 0.019s
I20251025 07:52:27.934244 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_ac4beca0c3bd4cf29565faf5323d9ff7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:27.934541 1167 tablet_bootstrap.cc:654] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:27.934520 1166 raft_consensus.cc:359] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.934676 1166 raft_consensus.cc:385] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:27.934757 1166 raft_consensus.cc:740] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
W20251025 07:52:27.934876 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac4beca0c3bd4cf29565faf5323d9ff7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:27.935003 1166 consensus_queue.cc:260] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.942757 1166 ts_tablet_manager.cc:1434] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20251025 07:52:27.943087 1166 tablet_bootstrap.cc:492] T cf2cab1552bd4e7394797c93b149f450 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:27.944279 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 9d6a9937d10a4cc2813671dd1bfa1664
I20251025 07:52:27.944851 1166 tablet_bootstrap.cc:654] T cf2cab1552bd4e7394797c93b149f450 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:27.947710 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d6a9937d10a4cc2813671dd1bfa1664 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:27.948113 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d6a9937d10a4cc2813671dd1bfa1664 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:56590
W20251025 07:52:27.948729 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 9d6a9937d10a4cc2813671dd1bfa1664 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d6a9937d10a4cc2813671dd1bfa1664 already in progress: creating tablet
W20251025 07:52:27.948908 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 9d6a9937d10a4cc2813671dd1bfa1664 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d6a9937d10a4cc2813671dd1bfa1664 already in progress: creating tablet
I20251025 07:52:27.950654 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d6a9937d10a4cc2813671dd1bfa1664 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:27.951907 1170 ts_tablet_manager.cc:1929] T 4829ec41391145cc84327d60048d9b0c P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:27.952039 1170 log.cc:1199] T 4829ec41391145cc84327d60048d9b0c P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/4829ec41391145cc84327d60048d9b0c
I20251025 07:52:27.952414 1170 ts_tablet_manager.cc:1950] T 4829ec41391145cc84327d60048d9b0c P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:27.954186 1167 tablet_bootstrap.cc:492] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:27.954340 1167 ts_tablet_manager.cc:1403] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.008s
I20251025 07:52:27.954986 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 9bf486d82b984d2c89ee32e84a85894f
I20251025 07:52:27.955981 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 9bf486d82b984d2c89ee32e84a85894f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:27.956559 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 9bf486d82b984d2c89ee32e84a85894f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:27.956612 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 9bf486d82b984d2c89ee32e84a85894f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:27.956868 1172 tablet_replica.cc:333] T 9bf486d82b984d2c89ee32e84a85894f P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:27.957099 1172 raft_consensus.cc:2243] T 9bf486d82b984d2c89ee32e84a85894f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:27.957305 1172 raft_consensus.cc:2272] T 9bf486d82b984d2c89ee32e84a85894f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:27.957284 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 9bf486d82b984d2c89ee32e84a85894f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9bf486d82b984d2c89ee32e84a85894f already in progress: creating tablet
I20251025 07:52:27.958153 1172 ts_tablet_manager.cc:1916] T 9bf486d82b984d2c89ee32e84a85894f P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:27.959710 1170 tablet_replica.cc:333] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:27.959961 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 4829ec41391145cc84327d60048d9b0c (table default.Table_45b1ec3be7e9460296fe5b819beaa6ea [id=94a9a4e752b547afb3d4d0b6806bdb54]) successfully deleted
W20251025 07:52:27.960237 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 3c3eb43209f24592820638fc25486b78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c3eb43209f24592820638fc25486b78 already in progress: creating tablet
I20251025 07:52:27.962175 1170 raft_consensus.cc:2243] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:27.962363 1170 raft_consensus.cc:2272] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:27.962375 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 50ae9bca6bed4f749d54191ac1543410
I20251025 07:52:27.963847 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 50ae9bca6bed4f749d54191ac1543410 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:48830
W20251025 07:52:27.964552 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 50ae9bca6bed4f749d54191ac1543410 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50ae9bca6bed4f749d54191ac1543410 already in progress: creating tablet
I20251025 07:52:27.965533 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_4b63840ca3ac4296b23104bf46030bb4" } modify_external_catalogs: true
I20251025 07:52:27.965730 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_4b63840ca3ac4296b23104bf46030bb4" } modify_external_catalogs: true
I20251025 07:52:27.965909 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 50ae9bca6bed4f749d54191ac1543410 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:27.966632 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_aa90c6fe1e744173a6174066750dc583" } new_table_name: "default.Table_438f67f3dc62465999a917986922b3ed" modify_external_catalogs: true
I20251025 07:52:27.967952 1169 tablet_bootstrap.cc:492] T 3c3eb43209f24592820638fc25486b78 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:27.968539 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 50ae9bca6bed4f749d54191ac1543410 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:27.969408 1169 ts_tablet_manager.cc:1403] T 3c3eb43209f24592820638fc25486b78 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.045s user 0.008s sys 0.002s
I20251025 07:52:27.972994 1169 raft_consensus.cc:359] T 3c3eb43209f24592820638fc25486b78 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.973331 1169 raft_consensus.cc:385] T 3c3eb43209f24592820638fc25486b78 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:27.973464 1169 raft_consensus.cc:740] T 3c3eb43209f24592820638fc25486b78 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:27.973788 1169 consensus_queue.cc:260] T 3c3eb43209f24592820638fc25486b78 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.974498 1169 ts_tablet_manager.cc:1434] T 3c3eb43209f24592820638fc25486b78 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251025 07:52:27.974829 1169 tablet_bootstrap.cc:492] T 198db9a45257435f86df2605992ccb34 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
W20251025 07:52:27.975667 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d5e5e5bbcf9f43118eea502b69880af5 consensus state has no leader
I20251025 07:52:27.976586 1169 tablet_bootstrap.cc:654] T 198db9a45257435f86df2605992ccb34 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:27.980774 1167 raft_consensus.cc:359] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.981036 1167 raft_consensus.cc:385] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:27.981143 1167 raft_consensus.cc:740] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:27.981431 1167 consensus_queue.cc:260] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.982686 1167 ts_tablet_manager.cc:1434] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.028s user 0.000s sys 0.002s
I20251025 07:52:27.982993 1167 tablet_bootstrap.cc:492] T 796413da731245459cc5dffacc3deb2f P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:27.984800 1169 tablet_bootstrap.cc:492] T 198db9a45257435f86df2605992ccb34 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:27.985049 1169 ts_tablet_manager.cc:1403] T 198db9a45257435f86df2605992ccb34 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20251025 07:52:27.985924 1169 raft_consensus.cc:359] T 198db9a45257435f86df2605992ccb34 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.986227 1169 raft_consensus.cc:385] T 198db9a45257435f86df2605992ccb34 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:27.987039 1169 raft_consensus.cc:740] T 198db9a45257435f86df2605992ccb34 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:27.987363 1169 consensus_queue.cc:260] T 198db9a45257435f86df2605992ccb34 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:27.988826 1169 ts_tablet_manager.cc:1434] T 198db9a45257435f86df2605992ccb34 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251025 07:52:27.989179 1169 tablet_bootstrap.cc:492] T 50433881c0eb4aa5a389b6f5b4a4337a P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:27.991094 1169 tablet_bootstrap.cc:654] T 50433881c0eb4aa5a389b6f5b4a4337a P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:28.000105 1169 tablet_bootstrap.cc:492] T 50433881c0eb4aa5a389b6f5b4a4337a P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:28.000427 1169 ts_tablet_manager.cc:1403] T 50433881c0eb4aa5a389b6f5b4a4337a P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20251025 07:52:28.001446 1169 raft_consensus.cc:359] T 50433881c0eb4aa5a389b6f5b4a4337a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:28.001645 1170 ts_tablet_manager.cc:1916] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:28.001729 1169 raft_consensus.cc:385] T 50433881c0eb4aa5a389b6f5b4a4337a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:28.001930 1169 raft_consensus.cc:740] T 50433881c0eb4aa5a389b6f5b4a4337a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:28.002323 1169 consensus_queue.cc:260] T 50433881c0eb4aa5a389b6f5b4a4337a P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:28.003885 1169 ts_tablet_manager.cc:1434] T 50433881c0eb4aa5a389b6f5b4a4337a P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251025 07:52:28.004567 1169 tablet_bootstrap.cc:492] T b00153c97104470883f6089d71d23708 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:27.984651 1172 ts_tablet_manager.cc:1929] T 9bf486d82b984d2c89ee32e84a85894f P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:28.005450 1172 log.cc:1199] T 9bf486d82b984d2c89ee32e84a85894f P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/9bf486d82b984d2c89ee32e84a85894f
I20251025 07:52:28.006670 1169 tablet_bootstrap.cc:654] T b00153c97104470883f6089d71d23708 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:28.007660 1172 ts_tablet_manager.cc:1950] T 9bf486d82b984d2c89ee32e84a85894f P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:28.009913 1166 tablet_bootstrap.cc:492] T cf2cab1552bd4e7394797c93b149f450 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:28.010104 1166 ts_tablet_manager.cc:1403] T cf2cab1552bd4e7394797c93b149f450 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.067s user 0.010s sys 0.008s
I20251025 07:52:28.010152 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 9bf486d82b984d2c89ee32e84a85894f (table default.Table_21a2393beb5d49a8ba839143c8328569 [id=5fcd7d82bf9443edbf595c28fcab99ae]) successfully deleted
I20251025 07:52:28.010504 1167 tablet_bootstrap.cc:654] T 796413da731245459cc5dffacc3deb2f P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:28.011976 1166 raft_consensus.cc:359] T cf2cab1552bd4e7394797c93b149f450 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.012223 1166 raft_consensus.cc:385] T cf2cab1552bd4e7394797c93b149f450 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:28.012318 1166 raft_consensus.cc:740] T cf2cab1552bd4e7394797c93b149f450 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
W20251025 07:52:28.012418 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 50ae9bca6bed4f749d54191ac1543410 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50ae9bca6bed4f749d54191ac1543410 already in progress: creating tablet
I20251025 07:52:28.012607 1166 consensus_queue.cc:260] T cf2cab1552bd4e7394797c93b149f450 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.015177 1169 tablet_bootstrap.cc:492] T b00153c97104470883f6089d71d23708 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:28.015398 1169 ts_tablet_manager.cc:1403] T b00153c97104470883f6089d71d23708 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.001s
I20251025 07:52:28.015856 1166 ts_tablet_manager.cc:1434] T cf2cab1552bd4e7394797c93b149f450 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20251025 07:52:28.016337 1169 raft_consensus.cc:359] T b00153c97104470883f6089d71d23708 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.016604 1169 raft_consensus.cc:385] T b00153c97104470883f6089d71d23708 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:28.016737 1169 raft_consensus.cc:740] T b00153c97104470883f6089d71d23708 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:28.017061 1169 consensus_queue.cc:260] T b00153c97104470883f6089d71d23708 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.018914 1169 ts_tablet_manager.cc:1434] T b00153c97104470883f6089d71d23708 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251025 07:52:28.019279 1169 tablet_bootstrap.cc:492] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:28.021071 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet d5e5e5bbcf9f43118eea502b69880af5
I20251025 07:52:28.021327 1169 tablet_bootstrap.cc:654] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:28.021585 523 catalog_manager.cc:6719] ReplaceTablet: tablet d5e5e5bbcf9f43118eea502b69880af5 deleted and replaced by tablet f4578ce69a0d4322944e5ce7c8a59831
I20251025 07:52:28.022172 1170 ts_tablet_manager.cc:1929] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:28.022315 1166 tablet_bootstrap.cc:492] T d0fd00baa02d48278afaa5b8c19182f0 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:28.022356 1170 log.cc:1199] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/19918a3c1f1d4ed6a78d5b4f0f4e38b3
I20251025 07:52:28.022809 1170 ts_tablet_manager.cc:1950] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:28.023950 1170 tablet_replica.cc:333] T 50433881c0eb4aa5a389b6f5b4a4337a P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:28.024106 1166 tablet_bootstrap.cc:654] T d0fd00baa02d48278afaa5b8c19182f0 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:28.024361 1169 tablet_bootstrap.cc:492] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:28.024480 1169 ts_tablet_manager.cc:1403] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20251025 07:52:28.025401 1169 raft_consensus.cc:359] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.025621 1169 raft_consensus.cc:385] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:28.025722 1169 raft_consensus.cc:740] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:28.025970 1169 consensus_queue.cc:260] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.026567 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 19918a3c1f1d4ed6a78d5b4f0f4e38b3 (table default.Table_b1c43f4e0ede4adcba8b010f04bdb72a [id=ee3cc33c49494767b860c9f1b7fa0d4d]) successfully deleted
W20251025 07:52:28.026916 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 02e4c4d6ffb34cecaf17beb5ba197e81 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02e4c4d6ffb34cecaf17beb5ba197e81 already in progress: creating tablet
I20251025 07:52:28.027590 884 tablet_service.cc:1552] Processing DeleteTablet for tablet d5e5e5bbcf9f43118eea502b69880af5 with delete_type TABLET_DATA_DELETED (replaced by tablet f4578ce69a0d4322944e5ce7c8a59831) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:28.027920 753 tablet_service.cc:1552] Processing DeleteTablet for tablet d5e5e5bbcf9f43118eea502b69880af5 with delete_type TABLET_DATA_DELETED (replaced by tablet f4578ce69a0d4322944e5ce7c8a59831) from {username='slave'} at 127.0.0.1:48830
W20251025 07:52:28.028635 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet d5e5e5bbcf9f43118eea502b69880af5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d5e5e5bbcf9f43118eea502b69880af5 already in progress: creating tablet
W20251025 07:52:28.028991 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet d5e5e5bbcf9f43118eea502b69880af5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d5e5e5bbcf9f43118eea502b69880af5 already in progress: creating tablet
I20251025 07:52:28.029145 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet d5e5e5bbcf9f43118eea502b69880af5 with delete_type TABLET_DATA_DELETED (replaced by tablet f4578ce69a0d4322944e5ce7c8a59831) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:28.029496 1169 ts_tablet_manager.cc:1434] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251025 07:52:28.029764 1170 raft_consensus.cc:2243] T 50433881c0eb4aa5a389b6f5b4a4337a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:28.030455 1170 raft_consensus.cc:2272] T 50433881c0eb4aa5a389b6f5b4a4337a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:28.031595 1170 ts_tablet_manager.cc:1916] T 50433881c0eb4aa5a389b6f5b4a4337a P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:28.032606 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_a0015f25f2554a0caeac8b446759378b" } new_table_name: "default.Table_6b02014a8bfe42b78c2856f646f0c75b" modify_external_catalogs: true
W20251025 07:52:28.033819 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8f2c8582fed34c848d9903b8ad8b4057 consensus state has no leader
I20251025 07:52:28.034193 1166 tablet_bootstrap.cc:492] T d0fd00baa02d48278afaa5b8c19182f0 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:28.034162 524 master_service.cc:954] ReplaceTablet: received request to replace tablet 2f26667f053a4f16b5da330c0b5e40f9 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:28.034333 1166 ts_tablet_manager.cc:1403] T d0fd00baa02d48278afaa5b8c19182f0 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.010s
I20251025 07:52:28.034263 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_5ecc1936fc8447669d0602cfd6133c6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:28.035023 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ecc1936fc8447669d0602cfd6133c6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:28.035197 1166 raft_consensus.cc:359] T d0fd00baa02d48278afaa5b8c19182f0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.035374 1166 raft_consensus.cc:385] T d0fd00baa02d48278afaa5b8c19182f0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:28.035461 1166 raft_consensus.cc:740] T d0fd00baa02d48278afaa5b8c19182f0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:28.032835 1169 tablet_bootstrap.cc:492] T 83084be89b2646b0bda208b3f6f60d77 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:28.035708 1166 consensus_queue.cc:260] T d0fd00baa02d48278afaa5b8c19182f0 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.036768 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_d2a7e16c6f0d4b2886c16bfe6b685c00"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:28.037357 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2a7e16c6f0d4b2886c16bfe6b685c00 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:28.037595 1166 ts_tablet_manager.cc:1434] T d0fd00baa02d48278afaa5b8c19182f0 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251025 07:52:28.037770 1169 tablet_bootstrap.cc:654] T 83084be89b2646b0bda208b3f6f60d77 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:28.037919 1166 tablet_bootstrap.cc:492] T 796413da731245459cc5dffacc3deb2f P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:28.034946 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_2d41b478c59b488bb7aadc3ce393c24c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:28.039058 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d41b478c59b488bb7aadc3ce393c24c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:28.039595 1166 tablet_bootstrap.cc:654] T 796413da731245459cc5dffacc3deb2f P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:28.039539 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_33b6fa5be1ab40fba35b3d5ef9ad9668" } new_table_name: "default.Table_dda83c899ebe4be999ac2ab537d3d323" modify_external_catalogs: true
W20251025 07:52:28.047992 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5cba9c05dd424cc788febc614a34819a consensus state has no leader
I20251025 07:52:28.048287 525 master_service.cc:954] ReplaceTablet: received request to replace tablet 299572b2dbe244918dcebc5ea5f3c45a from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:28.048684 1167 tablet_bootstrap.cc:492] T 796413da731245459cc5dffacc3deb2f P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:28.048879 1167 ts_tablet_manager.cc:1403] T 796413da731245459cc5dffacc3deb2f P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.066s user 0.000s sys 0.009s
I20251025 07:52:28.049647 1167 raft_consensus.cc:359] T 796413da731245459cc5dffacc3deb2f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.049849 1167 raft_consensus.cc:385] T 796413da731245459cc5dffacc3deb2f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:28.049933 1167 raft_consensus.cc:740] T 796413da731245459cc5dffacc3deb2f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:28.050199 1167 consensus_queue.cc:260] T 796413da731245459cc5dffacc3deb2f P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.050715 1167 ts_tablet_manager.cc:1434] T 796413da731245459cc5dffacc3deb2f P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:28.053157 526 catalog_manager.cc:5649] T 299572b2dbe244918dcebc5ea5f3c45a P db42be7ad3c741a0aa427f27e245993b reported cstate change: term changed from 0 to 1, leader changed from <none> to db42be7ad3c741a0aa427f27e245993b (127.0.104.3). New cstate: current_term: 1 leader_uuid: "db42be7ad3c741a0aa427f27e245993b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: UNKNOWN } } }
I20251025 07:52:28.053570 526 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 299572b2dbe244918dcebc5ea5f3c45a (table default.Table_bd5089a7892041649dd05b7fb0961255 [id=840b3ffcb44144548843457c48f46380]). Expected version 1 got 0
I20251025 07:52:28.054987 520 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 3d5e0dd14e474fe09993a03851f8b1f0 (table default.Table_20b987167f0e4863824e7bb7c17e5c03 [id=1b5db2f5053c4cb3a3b40eb109d5d5d9]). Expected version 2 got 0
I20251025 07:52:28.056795 517 catalog_manager.cc:5649] T 3d5e0dd14e474fe09993a03851f8b1f0 P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } }
I20251025 07:52:28.057384 517 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 3d5e0dd14e474fe09993a03851f8b1f0 (table default.Table_20b987167f0e4863824e7bb7c17e5c03 [id=1b5db2f5053c4cb3a3b40eb109d5d5d9]). Expected version 2 got 0
I20251025 07:52:28.057696 884 tablet_service.cc:1505] Processing CreateTablet for tablet 22c97b3ee92140c68a7e1b93033cb299 (DEFAULT_TABLE table=default.Table_c45e312143074d47be4fac483ee16136 [id=a003a6f6574b4cde88d30928eaa7f4ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.058488 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22c97b3ee92140c68a7e1b93033cb299. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.058704 753 tablet_service.cc:1552] Processing DeleteTablet for tablet a231f3d31b204f7b82f018892ccc8435 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:28.058893 1170 ts_tablet_manager.cc:1929] T 50433881c0eb4aa5a389b6f5b4a4337a P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:28.058979 1171 tablet_replica.cc:333] T a231f3d31b204f7b82f018892ccc8435 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:28.058902 755 tablet_service.cc:1505] Processing CreateTablet for tablet 22c97b3ee92140c68a7e1b93033cb299 (DEFAULT_TABLE table=default.Table_c45e312143074d47be4fac483ee16136 [id=a003a6f6574b4cde88d30928eaa7f4ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.059019 1170 log.cc:1199] T 50433881c0eb4aa5a389b6f5b4a4337a P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/50433881c0eb4aa5a389b6f5b4a4337a
I20251025 07:52:28.059221 1171 raft_consensus.cc:2243] T a231f3d31b204f7b82f018892ccc8435 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:28.059311 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22c97b3ee92140c68a7e1b93033cb299. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.059408 1170 ts_tablet_manager.cc:1950] T 50433881c0eb4aa5a389b6f5b4a4337a P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:28.059399 1171 raft_consensus.cc:2272] T a231f3d31b204f7b82f018892ccc8435 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:28.059985 1171 ts_tablet_manager.cc:1916] T a231f3d31b204f7b82f018892ccc8435 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:28.060500 1170 tablet_replica.cc:333] T 576c51d9a22d45f6a3642a7296b5bf23 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:28.060716 1170 raft_consensus.cc:2243] T 576c51d9a22d45f6a3642a7296b5bf23 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:28.060855 1170 raft_consensus.cc:2272] T 576c51d9a22d45f6a3642a7296b5bf23 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:28.061362 1170 ts_tablet_manager.cc:1916] T 576c51d9a22d45f6a3642a7296b5bf23 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:28.062927 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet a13bf4d7eb1440f7a28b85cb1bf99667 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a13bf4d7eb1440f7a28b85cb1bf99667 already in progress: creating tablet
I20251025 07:52:28.063766 1247 raft_consensus.cc:493] T a231f3d31b204f7b82f018892ccc8435 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:28.063952 1247 raft_consensus.cc:515] T a231f3d31b204f7b82f018892ccc8435 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.064649 1247 leader_election.cc:290] T a231f3d31b204f7b82f018892ccc8435 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:28.065426 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a231f3d31b204f7b82f018892ccc8435" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:28.065852 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a231f3d31b204f7b82f018892ccc8435" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
W20251025 07:52:28.066388 983 leader_election.cc:343] T a231f3d31b204f7b82f018892ccc8435 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:28.070025 1171 ts_tablet_manager.cc:1929] T a231f3d31b204f7b82f018892ccc8435 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:28.070541 889 tablet_service.cc:1505] Processing CreateTablet for tablet 80ee75935d7e461d9d4066fac9819bef (DEFAULT_TABLE table=default.Table_6e28f51cd66f471a8d55f19e6f5becff [id=c80bab0ed06a459eb78b1d94df13f337]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.071182 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80ee75935d7e461d9d4066fac9819bef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.071337 1167 tablet_bootstrap.cc:492] T 50433881c0eb4aa5a389b6f5b4a4337a P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:28.073751 1167 tablet_bootstrap.cc:654] T 50433881c0eb4aa5a389b6f5b4a4337a P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:28.074965 982 leader_election.cc:343] T a231f3d31b204f7b82f018892ccc8435 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: Tablet not RUNNING: STOPPED
I20251025 07:52:28.074944 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_bf5cb8a733934f4ca29ecc9d473e4f2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:28.075158 982 leader_election.cc:304] T a231f3d31b204f7b82f018892ccc8435 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
W20251025 07:52:28.075556 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf5cb8a733934f4ca29ecc9d473e4f2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:28.075528 1018 tablet_service.cc:1505] Processing CreateTablet for tablet 22c97b3ee92140c68a7e1b93033cb299 (DEFAULT_TABLE table=default.Table_c45e312143074d47be4fac483ee16136 [id=a003a6f6574b4cde88d30928eaa7f4ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.075726 1247 raft_consensus.cc:2749] T a231f3d31b204f7b82f018892ccc8435 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:28.076038 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22c97b3ee92140c68a7e1b93033cb299. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.071484 1171 log.cc:1199] T a231f3d31b204f7b82f018892ccc8435 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/a231f3d31b204f7b82f018892ccc8435
I20251025 07:52:28.077050 1171 ts_tablet_manager.cc:1950] T a231f3d31b204f7b82f018892ccc8435 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:28.078238 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet a231f3d31b204f7b82f018892ccc8435 (table default.Table_1c458711d56d4ddeba1d0bc09392a734 [id=7cebd72e604e497db80a8bc823471e97]) successfully deleted
I20251025 07:52:28.080623 1170 ts_tablet_manager.cc:1929] T 576c51d9a22d45f6a3642a7296b5bf23 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:28.080786 1170 log.cc:1199] T 576c51d9a22d45f6a3642a7296b5bf23 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/576c51d9a22d45f6a3642a7296b5bf23
I20251025 07:52:28.081247 1170 ts_tablet_manager.cc:1950] T 576c51d9a22d45f6a3642a7296b5bf23 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:28.083518 1218 raft_consensus.cc:493] T cf2cab1552bd4e7394797c93b149f450 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:28.083715 1218 raft_consensus.cc:515] T cf2cab1552bd4e7394797c93b149f450 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.083838 522 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 796413da731245459cc5dffacc3deb2f (table default.Table_293596c64cca4768bc5869fd9924f856 [id=d1b9d89b22bc4431a7d59ceeb9d75050]). Expected version 2 got 0
I20251025 07:52:28.084474 1218 leader_election.cc:290] T cf2cab1552bd4e7394797c93b149f450 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:28.085163 522 master_service.cc:954] ReplaceTablet: received request to replace tablet 5cba9c05dd424cc788febc614a34819a from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:28.085425 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf2cab1552bd4e7394797c93b149f450" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:28.085824 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf2cab1552bd4e7394797c93b149f450" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:28.082374 1170 tablet_replica.cc:333] T 713ae1dd444544bdabdd4a771e109d90 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:28.086323 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 19918a3c1f1d4ed6a78d5b4f0f4e38b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:28.086589 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 33c608beb4ef4a859a135b11e07c81a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:28.086585 1170 raft_consensus.cc:2243] T 713ae1dd444544bdabdd4a771e109d90 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:28.086807 1170 raft_consensus.cc:2272] T 713ae1dd444544bdabdd4a771e109d90 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:28.086903 847 leader_election.cc:343] T cf2cab1552bd4e7394797c93b149f450 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:28.087517 850 leader_election.cc:343] T cf2cab1552bd4e7394797c93b149f450 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:28.087653 850 leader_election.cc:304] T cf2cab1552bd4e7394797c93b149f450 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:28.088044 1218 raft_consensus.cc:2749] T cf2cab1552bd4e7394797c93b149f450 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:28.088554 1170 ts_tablet_manager.cc:1916] T 713ae1dd444544bdabdd4a771e109d90 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:28.090611 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 576c51d9a22d45f6a3642a7296b5bf23 (table default.Table_14cbf20f7ff449499612663b54f0acd9 [id=e9343178deb142339649b0cc6d26a9af]) successfully deleted
I20251025 07:52:28.096760 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_4f9a077b76bf42979239aeb6710131ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:28.097329 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f9a077b76bf42979239aeb6710131ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:28.097970 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_6588890a31184cc2ab6ccb9b72e60429"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:28.098498 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6588890a31184cc2ab6ccb9b72e60429 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:28.099241 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_580b09b75790418e8dcf68fc88bbbf47"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:28.099727 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_580b09b75790418e8dcf68fc88bbbf47 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:28.099977 1247 raft_consensus.cc:493] T 3c3eb43209f24592820638fc25486b78 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:28.100159 1247 raft_consensus.cc:515] T 3c3eb43209f24592820638fc25486b78 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.100864 1247 leader_election.cc:290] T 3c3eb43209f24592820638fc25486b78 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:28.102835 983 leader_election.cc:343] T 3c3eb43209f24592820638fc25486b78 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 3c3eb43209f24592820638fc25486b78
I20251025 07:52:28.103221 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c3eb43209f24592820638fc25486b78" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:28.105302 1170 ts_tablet_manager.cc:1929] T 713ae1dd444544bdabdd4a771e109d90 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:28.106410 1169 tablet_bootstrap.cc:492] T 83084be89b2646b0bda208b3f6f60d77 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:28.106635 1169 ts_tablet_manager.cc:1403] T 83084be89b2646b0bda208b3f6f60d77 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.074s user 0.017s sys 0.015s
W20251025 07:52:28.106945 982 leader_election.cc:343] T 3c3eb43209f24592820638fc25486b78 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:28.107084 982 leader_election.cc:304] T 3c3eb43209f24592820638fc25486b78 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:28.107753 1169 raft_consensus.cc:359] T 83084be89b2646b0bda208b3f6f60d77 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.107964 1169 raft_consensus.cc:385] T 83084be89b2646b0bda208b3f6f60d77 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:28.108039 1169 raft_consensus.cc:740] T 83084be89b2646b0bda208b3f6f60d77 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:28.108292 1169 consensus_queue.cc:260] T 83084be89b2646b0bda208b3f6f60d77 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.109148 1322 raft_consensus.cc:2749] T 3c3eb43209f24592820638fc25486b78 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:28.112969 1169 ts_tablet_manager.cc:1434] T 83084be89b2646b0bda208b3f6f60d77 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251025 07:52:28.113729 1169 tablet_bootstrap.cc:492] T 796413da731245459cc5dffacc3deb2f P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:28.115106 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 2f26667f053a4f16b5da330c0b5e40f9
I20251025 07:52:28.115546 1169 tablet_bootstrap.cc:654] T 796413da731245459cc5dffacc3deb2f P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:28.115597 524 catalog_manager.cc:6719] ReplaceTablet: tablet 2f26667f053a4f16b5da330c0b5e40f9 deleted and replaced by tablet 34315f64b89148efbf1f20a43d5c5efb
I20251025 07:52:28.116261 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet abf94c32ff2945119b710905c075d63c
I20251025 07:52:28.116484 1170 log.cc:1199] T 713ae1dd444544bdabdd4a771e109d90 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/713ae1dd444544bdabdd4a771e109d90
I20251025 07:52:28.116716 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f26667f053a4f16b5da330c0b5e40f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 34315f64b89148efbf1f20a43d5c5efb) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:28.116911 1170 ts_tablet_manager.cc:1950] T 713ae1dd444544bdabdd4a771e109d90 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:28.117588 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f26667f053a4f16b5da330c0b5e40f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 34315f64b89148efbf1f20a43d5c5efb) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:28.117878 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet abf94c32ff2945119b710905c075d63c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:28.117986 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 713ae1dd444544bdabdd4a771e109d90 (table default.Table_70998787ec5746b9bb56f6e5e642898e [id=034108fa919f469bb53041fd03a1b5d3]) successfully deleted
W20251025 07:52:28.118741 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 2209089667dd4937abd96615f6929449 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2209089667dd4937abd96615f6929449 already in progress: creating tablet
I20251025 07:52:28.119549 1170 tablet_replica.cc:333] T 83084be89b2646b0bda208b3f6f60d77 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:28.119792 1170 raft_consensus.cc:2243] T 83084be89b2646b0bda208b3f6f60d77 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:28.119870 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_6e28f51cd66f471a8d55f19e6f5becff" } new_table_name: "default.Table_adb643980c8844de93515aee9326e2df" modify_external_catalogs: true
I20251025 07:52:28.119963 1170 raft_consensus.cc:2272] T 83084be89b2646b0bda208b3f6f60d77 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:28.122243 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 80ee75935d7e461d9d4066fac9819bef (DEFAULT_TABLE table=default.Table_6e28f51cd66f471a8d55f19e6f5becff [id=c80bab0ed06a459eb78b1d94df13f337]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.122673 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80ee75935d7e461d9d4066fac9819bef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.125072 886 tablet_service.cc:1552] Processing DeleteTablet for tablet abf94c32ff2945119b710905c075d63c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:56590
W20251025 07:52:28.125838 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet abf94c32ff2945119b710905c075d63c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abf94c32ff2945119b710905c075d63c already in progress: creating tablet
I20251025 07:52:28.126456 1170 ts_tablet_manager.cc:1916] T 83084be89b2646b0bda208b3f6f60d77 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:28.126443 753 tablet_service.cc:1552] Processing DeleteTablet for tablet abf94c32ff2945119b710905c075d63c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:28.130462 526 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 50ffbcdd4a9247bf99b8e2361bc02ef5 (table default.Table_9f0ae2442a334cf8ab13f2eba387ebf3 [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 3 got 0
I20251025 07:52:28.131899 524 master_service.cc:954] ReplaceTablet: received request to replace tablet 35ba9d23262248cda6fdf03c51be11a4 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:28.133373 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_f6e21e14e6fb4649af45c7d10a5f993e" } modify_external_catalogs: true
I20251025 07:52:28.133440 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f26667f053a4f16b5da330c0b5e40f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 34315f64b89148efbf1f20a43d5c5efb) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:28.133553 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_f6e21e14e6fb4649af45c7d10a5f993e" } modify_external_catalogs: true
W20251025 07:52:28.134097 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 2f26667f053a4f16b5da330c0b5e40f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f26667f053a4f16b5da330c0b5e40f9 already in progress: creating tablet
I20251025 07:52:28.134924 886 tablet_service.cc:1505] Processing CreateTablet for tablet 92aa605fabbc4257a6a204c087e7cc7f (DEFAULT_TABLE table=default.Table_618540173e8a45828c270476360228f0 [id=187bde879b044237ada58158eb000123]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.135365 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92aa605fabbc4257a6a204c087e7cc7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.136708 1171 tablet_replica.cc:333] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:28.136929 1171 raft_consensus.cc:2243] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:28.137126 1171 raft_consensus.cc:2272] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:28.139417 891 tablet_service.cc:1505] Processing CreateTablet for tablet 74e2d78c024849b0af5c0bde1f48c36f (DEFAULT_TABLE table=default.Table_6b9cad0ecb4a4d74941d9904282d676d [id=3a3af35e11944a6f97fc0144cb9d08d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.139497 741 tablet_service.cc:1505] Processing CreateTablet for tablet 6a58931e7cd7454fb583f1c70cd2ad15 (DEFAULT_TABLE table=default.Table_5237c35dbfa7429e803d5008a8b3487c [id=0d597ecab6f54d1ba6cb5bd3436a9db8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.139846 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74e2d78c024849b0af5c0bde1f48c36f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.139889 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a58931e7cd7454fb583f1c70cd2ad15. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.141201 755 tablet_service.cc:1505] Processing CreateTablet for tablet 80ee75935d7e461d9d4066fac9819bef (DEFAULT_TABLE table=default.Table_6e28f51cd66f471a8d55f19e6f5becff [id=c80bab0ed06a459eb78b1d94df13f337]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.141547 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80ee75935d7e461d9d4066fac9819bef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.142444 739 tablet_service.cc:1505] Processing CreateTablet for tablet 74e2d78c024849b0af5c0bde1f48c36f (DEFAULT_TABLE table=default.Table_6b9cad0ecb4a4d74941d9904282d676d [id=3a3af35e11944a6f97fc0144cb9d08d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.142791 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74e2d78c024849b0af5c0bde1f48c36f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.144044 884 tablet_service.cc:1505] Processing CreateTablet for tablet 6a58931e7cd7454fb583f1c70cd2ad15 (DEFAULT_TABLE table=default.Table_5237c35dbfa7429e803d5008a8b3487c [id=0d597ecab6f54d1ba6cb5bd3436a9db8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.144428 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a58931e7cd7454fb583f1c70cd2ad15. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.146683 753 tablet_service.cc:1505] Processing CreateTablet for tablet 92aa605fabbc4257a6a204c087e7cc7f (DEFAULT_TABLE table=default.Table_618540173e8a45828c270476360228f0 [id=187bde879b044237ada58158eb000123]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.147024 1171 ts_tablet_manager.cc:1916] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:28.147169 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92aa605fabbc4257a6a204c087e7cc7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.152514 1169 tablet_bootstrap.cc:492] T 796413da731245459cc5dffacc3deb2f P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:28.155444 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet a231f3d31b204f7b82f018892ccc8435 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:28.155516 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet 198db9a45257435f86df2605992ccb34 with delete_type TABLET_DATA_DELETED (replaced by tablet 71432a6848944a0da06ef9b81f160dc4) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:28.155593 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c3eb43209f24592820638fc25486b78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:28.155968 1169 ts_tablet_manager.cc:1403] T 796413da731245459cc5dffacc3deb2f P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.000s
I20251025 07:52:28.160696 1166 tablet_bootstrap.cc:492] T 796413da731245459cc5dffacc3deb2f P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:28.160921 1166 ts_tablet_manager.cc:1403] T 796413da731245459cc5dffacc3deb2f P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.123s user 0.011s sys 0.018s
I20251025 07:52:28.162261 1166 raft_consensus.cc:359] T 796413da731245459cc5dffacc3deb2f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.162547 1166 raft_consensus.cc:385] T 796413da731245459cc5dffacc3deb2f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:28.162658 1166 raft_consensus.cc:740] T 796413da731245459cc5dffacc3deb2f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:28.162955 1166 consensus_queue.cc:260] T 796413da731245459cc5dffacc3deb2f P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.164875 1166 ts_tablet_manager.cc:1434] T 796413da731245459cc5dffacc3deb2f P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251025 07:52:28.165246 1166 tablet_bootstrap.cc:492] T a231f3d31b204f7b82f018892ccc8435 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:28.168910 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 94408a3a90224645ba3cacccf5de33f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:28.170624 914 raft_consensus.cc:1275] T 2fac7b819e2642378e0d3f9230d0259c P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Refusing update from remote peer 23eeffd3de254ce18db5febfde971c6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 07:52:28.171363 1172 tablet_replica.cc:333] T 94408a3a90224645ba3cacccf5de33f1 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:28.171558 1172 raft_consensus.cc:2243] T 94408a3a90224645ba3cacccf5de33f1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:28.171746 1172 raft_consensus.cc:2272] T 94408a3a90224645ba3cacccf5de33f1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:28.172247 1178 consensus_queue.cc:1048] T 2fac7b819e2642378e0d3f9230d0259c P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:28.172439 1172 ts_tablet_manager.cc:1916] T 94408a3a90224645ba3cacccf5de33f1 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:28.173331 1020 tablet_service.cc:1505] Processing CreateTablet for tablet 92aa605fabbc4257a6a204c087e7cc7f (DEFAULT_TABLE table=default.Table_618540173e8a45828c270476360228f0 [id=187bde879b044237ada58158eb000123]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.173786 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92aa605fabbc4257a6a204c087e7cc7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.174105 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_c9bc68f9b5c54a33b854f9ee8a1ad311" } modify_external_catalogs: true
I20251025 07:52:28.174338 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_c9bc68f9b5c54a33b854f9ee8a1ad311" } modify_external_catalogs: true
I20251025 07:52:28.176601 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_dda83c899ebe4be999ac2ab537d3d323" } modify_external_catalogs: true
I20251025 07:52:28.176818 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_dda83c899ebe4be999ac2ab537d3d323" } modify_external_catalogs: true
I20251025 07:52:28.179821 1172 ts_tablet_manager.cc:1929] T 94408a3a90224645ba3cacccf5de33f1 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:28.179958 1172 log.cc:1199] T 94408a3a90224645ba3cacccf5de33f1 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/94408a3a90224645ba3cacccf5de33f1
I20251025 07:52:28.180317 1172 ts_tablet_manager.cc:1950] T 94408a3a90224645ba3cacccf5de33f1 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:28.181356 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 94408a3a90224645ba3cacccf5de33f1 (table default.Table_819aedd5270f4d6cbaaaee7c2ff5f676 [id=1e4728f474814b36992fd99de8451b95]) successfully deleted
I20251025 07:52:28.181628 1166 tablet_bootstrap.cc:654] T a231f3d31b204f7b82f018892ccc8435 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:28.182070 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 6a58931e7cd7454fb583f1c70cd2ad15 (DEFAULT_TABLE table=default.Table_5237c35dbfa7429e803d5008a8b3487c [id=0d597ecab6f54d1ba6cb5bd3436a9db8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.182703 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet b00153c97104470883f6089d71d23708 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:28.183050 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a58931e7cd7454fb583f1c70cd2ad15. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.184288 1170 ts_tablet_manager.cc:1929] T 83084be89b2646b0bda208b3f6f60d77 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:28.185128 1218 raft_consensus.cc:493] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:28.185324 1170 log.cc:1199] T 83084be89b2646b0bda208b3f6f60d77 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/83084be89b2646b0bda208b3f6f60d77
I20251025 07:52:28.185319 1218 raft_consensus.cc:515] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.185952 1170 ts_tablet_manager.cc:1950] T 83084be89b2646b0bda208b3f6f60d77 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:28.186013 1218 leader_election.cc:290] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:28.186239 1048 raft_consensus.cc:1275] T 2fac7b819e2642378e0d3f9230d0259c P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Refusing update from remote peer 23eeffd3de254ce18db5febfde971c6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 07:52:28.186764 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "198db9a45257435f86df2605992ccb34" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
W20251025 07:52:28.187392 847 leader_election.cc:343] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:28.187813 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "198db9a45257435f86df2605992ccb34" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:28.188014 1049 raft_consensus.cc:2468] T 198db9a45257435f86df2605992ccb34 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:28.188386 850 leader_election.cc:304] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b
I20251025 07:52:28.188578 1169 raft_consensus.cc:359] T 796413da731245459cc5dffacc3deb2f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.188741 1169 raft_consensus.cc:385] T 796413da731245459cc5dffacc3deb2f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:28.188951 1218 raft_consensus.cc:2804] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:28.188815 1178 consensus_queue.cc:1048] T 2fac7b819e2642378e0d3f9230d0259c P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:28.189056 1218 raft_consensus.cc:493] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:28.189591 1169 raft_consensus.cc:740] T 796413da731245459cc5dffacc3deb2f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:28.189693 1218 raft_consensus.cc:3060] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:28.189844 1169 consensus_queue.cc:260] T 796413da731245459cc5dffacc3deb2f P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.190173 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 5cba9c05dd424cc788febc614a34819a
I20251025 07:52:28.190797 522 catalog_manager.cc:6719] ReplaceTablet: tablet 5cba9c05dd424cc788febc614a34819a deleted and replaced by tablet 4b28215f61574cf48c447dd01955ac77
I20251025 07:52:28.192548 1218 raft_consensus.cc:515] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.193069 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 5cba9c05dd424cc788febc614a34819a with delete_type TABLET_DATA_DELETED (replaced by tablet 4b28215f61574cf48c447dd01955ac77) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:28.193710 1218 leader_election.cc:290] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:28.193837 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "198db9a45257435f86df2605992ccb34" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:28.193977 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 5cba9c05dd424cc788febc614a34819a with delete_type TABLET_DATA_DELETED (replaced by tablet 4b28215f61574cf48c447dd01955ac77) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:28.194444 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 5cba9c05dd424cc788febc614a34819a with delete_type TABLET_DATA_DELETED (replaced by tablet 4b28215f61574cf48c447dd01955ac77) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:28.194464 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 83084be89b2646b0bda208b3f6f60d77 (table default.Table_7c2f0e6e0edf4ae6920557e0d5130468 [id=31c55cf000ed4e4dbefe4a407fa1828c]) successfully deleted
W20251025 07:52:28.194445 847 leader_election.cc:343] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:28.195245 1167 tablet_bootstrap.cc:492] T 50433881c0eb4aa5a389b6f5b4a4337a P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:28.195390 1167 ts_tablet_manager.cc:1403] T 50433881c0eb4aa5a389b6f5b4a4337a P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.125s user 0.016s sys 0.015s
I20251025 07:52:28.196220 1167 raft_consensus.cc:359] T 50433881c0eb4aa5a389b6f5b4a4337a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:28.196393 1167 raft_consensus.cc:385] T 50433881c0eb4aa5a389b6f5b4a4337a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:28.196484 1167 raft_consensus.cc:740] T 50433881c0eb4aa5a389b6f5b4a4337a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:28.196692 1167 consensus_queue.cc:260] T 50433881c0eb4aa5a389b6f5b4a4337a P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:28.197216 1167 ts_tablet_manager.cc:1434] T 50433881c0eb4aa5a389b6f5b4a4337a P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20251025 07:52:28.198314 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 6ec421eb820b4f40afcff5a5bf80c508 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ec421eb820b4f40afcff5a5bf80c508 already in progress: creating tablet
I20251025 07:52:28.199039 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 74e2d78c024849b0af5c0bde1f48c36f (DEFAULT_TABLE table=default.Table_6b9cad0ecb4a4d74941d9904282d676d [id=3a3af35e11944a6f97fc0144cb9d08d4]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:28.199748 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 5cba9c05dd424cc788febc614a34819a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cba9c05dd424cc788febc614a34819a already in progress: creating tablet
I20251025 07:52:28.199877 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "198db9a45257435f86df2605992ccb34" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:28.200066 1048 raft_consensus.cc:3060] T 198db9a45257435f86df2605992ccb34 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:28.201267 1169 ts_tablet_manager.cc:1434] T 796413da731245459cc5dffacc3deb2f P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.034s user 0.002s sys 0.000s
I20251025 07:52:28.201629 1169 tablet_bootstrap.cc:492] T d0fd00baa02d48278afaa5b8c19182f0 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
W20251025 07:52:28.202335 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 85fba8b37f2c4540aa81701052bf222a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 85fba8b37f2c4540aa81701052bf222a already in progress: creating tablet
W20251025 07:52:28.202590 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 3775a2e08d4e40bd9f57e4cda759d535 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3775a2e08d4e40bd9f57e4cda759d535 already in progress: creating tablet
W20251025 07:52:28.202818 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet fa77085e81c7440d9e255c44d1e80c26 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fa77085e81c7440d9e255c44d1e80c26 already in progress: creating tablet
I20251025 07:52:28.203068 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 94408a3a90224645ba3cacccf5de33f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:28.204895 1167 tablet_bootstrap.cc:492] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
W20251025 07:52:28.205341 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 1d261e432cf5403ea162129706bc4d4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d261e432cf5403ea162129706bc4d4d already in progress: creating tablet
I20251025 07:52:28.206338 1224 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 07:52:28.206890 1167 tablet_bootstrap.cc:654] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:28.208508 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet b772025852794a49b072f37cccbd3291 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b772025852794a49b072f37cccbd3291 already in progress: creating tablet
I20251025 07:52:28.210427 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74e2d78c024849b0af5c0bde1f48c36f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:28.209652 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 9d6a9937d10a4cc2813671dd1bfa1664 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d6a9937d10a4cc2813671dd1bfa1664 already in progress: creating tablet
W20251025 07:52:28.211544 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 9bf486d82b984d2c89ee32e84a85894f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9bf486d82b984d2c89ee32e84a85894f already in progress: creating tablet
I20251025 07:52:28.213775 1171 ts_tablet_manager.cc:1929] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:28.214589 775 raft_consensus.cc:1275] T 3d5e0dd14e474fe09993a03851f8b1f0 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Refusing update from remote peer 3aea2869ff814cc7b142012faa274382: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 07:52:28.214610 1171 log.cc:1199] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/19918a3c1f1d4ed6a78d5b4f0f4e38b3
I20251025 07:52:28.215154 1171 ts_tablet_manager.cc:1950] T 19918a3c1f1d4ed6a78d5b4f0f4e38b3 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:28.215783 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 198db9a45257435f86df2605992ccb34 with delete_type TABLET_DATA_DELETED (replaced by tablet 71432a6848944a0da06ef9b81f160dc4) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:28.215943 886 tablet_service.cc:1552] Processing DeleteTablet for tablet d0fd00baa02d48278afaa5b8c19182f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:28.216202 1048 raft_consensus.cc:2468] T 198db9a45257435f86df2605992ccb34 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
W20251025 07:52:28.217026 511 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 94408a3a90224645ba3cacccf5de33f1 because the tablet was not found. No further retry: Not found: Tablet not found: 94408a3a90224645ba3cacccf5de33f1
I20251025 07:52:28.218016 1048 raft_consensus.cc:1275] T 3d5e0dd14e474fe09993a03851f8b1f0 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Refusing update from remote peer 3aea2869ff814cc7b142012faa274382: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 07:52:28.218844 884 tablet_service.cc:1552] Processing DeleteTablet for tablet cf2cab1552bd4e7394797c93b149f450 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:28.219025 850 leader_election.cc:304] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b
I20251025 07:52:28.219360 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_d771d85f5f6a4016bfa426dd9dbf586a" } modify_external_catalogs: true
I20251025 07:52:28.219565 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_d771d85f5f6a4016bfa426dd9dbf586a" } modify_external_catalogs: true
I20251025 07:52:28.219647 1326 raft_consensus.cc:2804] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:28.221895 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 19918a3c1f1d4ed6a78d5b4f0f4e38b3 (table default.Table_b1c43f4e0ede4adcba8b010f04bdb72a [id=ee3cc33c49494767b860c9f1b7fa0d4d]) successfully deleted
I20251025 07:52:28.223650 1171 tablet_replica.cc:333] T 33c608beb4ef4a859a135b11e07c81a4 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:28.224125 1171 raft_consensus.cc:2243] T 33c608beb4ef4a859a135b11e07c81a4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:28.224635 1171 raft_consensus.cc:2272] T 33c608beb4ef4a859a135b11e07c81a4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:28.225510 1172 tablet_replica.cc:333] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:28.226202 1171 ts_tablet_manager.cc:1916] T 33c608beb4ef4a859a135b11e07c81a4 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:28.226576 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 50ae9bca6bed4f749d54191ac1543410 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50ae9bca6bed4f749d54191ac1543410 already in progress: creating tablet
W20251025 07:52:28.226869 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet d5e5e5bbcf9f43118eea502b69880af5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d5e5e5bbcf9f43118eea502b69880af5 already in progress: creating tablet
I20251025 07:52:28.229187 1169 tablet_bootstrap.cc:654] T d0fd00baa02d48278afaa5b8c19182f0 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:28.229423 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_6fd078e1af7a41f8a24caebf5e12e2b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:28.229979 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6fd078e1af7a41f8a24caebf5e12e2b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:28.231293 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 80ee75935d7e461d9d4066fac9819bef consensus state has no leader
I20251025 07:52:28.231755 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_f8c38604344c429295d6d0ab9cb85b13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:28.231874 1218 consensus_queue.cc:1048] T 3d5e0dd14e474fe09993a03851f8b1f0 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
W20251025 07:52:28.232290 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8c38604344c429295d6d0ab9cb85b13 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:28.232427 1320 consensus_queue.cc:1048] T 3d5e0dd14e474fe09993a03851f8b1f0 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:28.232916 1326 raft_consensus.cc:697] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:28.233253 1326 consensus_queue.cc:237] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.243192 1170 tablet_replica.cc:333] T a231f3d31b204f7b82f018892ccc8435 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:28.243526 1170 raft_consensus.cc:2243] T a231f3d31b204f7b82f018892ccc8435 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:28.243976 1170 raft_consensus.cc:2272] T a231f3d31b204f7b82f018892ccc8435 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:28.244083 1166 tablet_bootstrap.cc:492] T a231f3d31b204f7b82f018892ccc8435 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
W20251025 07:52:28.244141 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 2f26667f053a4f16b5da330c0b5e40f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f26667f053a4f16b5da330c0b5e40f9 already in progress: creating tablet
I20251025 07:52:28.244256 1166 ts_tablet_manager.cc:1403] T a231f3d31b204f7b82f018892ccc8435 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.079s user 0.004s sys 0.003s
W20251025 07:52:28.244377 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet abf94c32ff2945119b710905c075d63c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abf94c32ff2945119b710905c075d63c already in progress: creating tablet
I20251025 07:52:28.245083 1166 raft_consensus.cc:359] T a231f3d31b204f7b82f018892ccc8435 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.245281 1166 raft_consensus.cc:385] T a231f3d31b204f7b82f018892ccc8435 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:28.245375 1166 raft_consensus.cc:740] T a231f3d31b204f7b82f018892ccc8435 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:28.245602 1166 consensus_queue.cc:260] T a231f3d31b204f7b82f018892ccc8435 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.246088 1166 ts_tablet_manager.cc:1434] T a231f3d31b204f7b82f018892ccc8435 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 07:52:28.246448 1166 tablet_bootstrap.cc:492] T dd898450f2604987b08639cf87d0e402 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:28.248785 1170 ts_tablet_manager.cc:1916] T a231f3d31b204f7b82f018892ccc8435 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:28.254539 1167 tablet_bootstrap.cc:492] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:28.254714 1167 ts_tablet_manager.cc:1403] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.000s
I20251025 07:52:28.249168 1361 mvcc.cc:204] Tried to move back new op lower bound from 7214607352496115712 to 7214607349576753152. Current Snapshot: MvccSnapshot[applied={T|T < 7214607352496115712}]
I20251025 07:52:28.253651 1172 raft_consensus.cc:2243] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:28.256989 1172 pending_rounds.cc:70] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382: Trying to abort 1 pending ops.
I20251025 07:52:28.257164 1172 pending_rounds.cc:77] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214607352787292160 op_type: NO_OP noop_request { }
I20251025 07:52:28.257452 1172 raft_consensus.cc:2889] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 07:52:28.257663 1172 raft_consensus.cc:2272] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:28.257962 1359 mvcc.cc:204] Tried to move back new op lower bound from 7214607352496115712 to 7214607349576753152. Current Snapshot: MvccSnapshot[applied={T|T < 7214607352496115712}]
I20251025 07:52:28.261420 1166 tablet_bootstrap.cc:654] T dd898450f2604987b08639cf87d0e402 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:28.262470 1167 raft_consensus.cc:359] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.264415 1167 raft_consensus.cc:385] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:28.264705 1167 raft_consensus.cc:740] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:28.265383 1167 consensus_queue.cc:260] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.265851 1374 tablet.cc:1722] T 2fac7b819e2642378e0d3f9230d0259c P 3aea2869ff814cc7b142012faa274382: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251025 07:52:28.266521 1167 ts_tablet_manager.cc:1434] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20251025 07:52:28.266570 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 299572b2dbe244918dcebc5ea5f3c45a
I20251025 07:52:28.266355 1373 tablet.cc:1722] T 2fac7b819e2642378e0d3f9230d0259c P 23eeffd3de254ce18db5febfde971c6b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251025 07:52:28.266876 1167 tablet_bootstrap.cc:492] T 2947502815d742e89cacb474b7fbb249 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:28.267047 525 catalog_manager.cc:6719] ReplaceTablet: tablet 299572b2dbe244918dcebc5ea5f3c45a deleted and replaced by tablet aaef24ccdfba45fe8204327723df834e
W20251025 07:52:28.259619 1372 consensus_queue.cc:1175] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "3aea2869ff814cc7b142012faa274382" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20251025 07:52:28.267828 1169 tablet_bootstrap.cc:492] T d0fd00baa02d48278afaa5b8c19182f0 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:28.268215 1169 ts_tablet_manager.cc:1403] T d0fd00baa02d48278afaa5b8c19182f0 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.066s user 0.008s sys 0.000s
I20251025 07:52:28.268847 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 299572b2dbe244918dcebc5ea5f3c45a with delete_type TABLET_DATA_DELETED (replaced by tablet aaef24ccdfba45fe8204327723df834e) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:28.269002 1169 raft_consensus.cc:359] T d0fd00baa02d48278afaa5b8c19182f0 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.269184 1169 raft_consensus.cc:385] T d0fd00baa02d48278afaa5b8c19182f0 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:28.269253 1169 raft_consensus.cc:740] T d0fd00baa02d48278afaa5b8c19182f0 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:28.269589 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 299572b2dbe244918dcebc5ea5f3c45a with delete_type TABLET_DATA_DELETED (replaced by tablet aaef24ccdfba45fe8204327723df834e) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:28.269551 1169 consensus_queue.cc:260] T d0fd00baa02d48278afaa5b8c19182f0 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.270656 1172 ts_tablet_manager.cc:1916] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:28.271080 1169 ts_tablet_manager.cc:1434] T d0fd00baa02d48278afaa5b8c19182f0 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20251025 07:52:28.272117 1169 tablet_bootstrap.cc:492] T 94408a3a90224645ba3cacccf5de33f1 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:28.273422 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 299572b2dbe244918dcebc5ea5f3c45a with delete_type TABLET_DATA_DELETED (replaced by tablet aaef24ccdfba45fe8204327723df834e) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:28.275620 1167 tablet_bootstrap.cc:654] T 2947502815d742e89cacb474b7fbb249 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:28.222568 1224 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 07:52:28.276161 1224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 07:52:28.277629 1172 ts_tablet_manager.cc:1929] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:28.277762 1172 log.cc:1199] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/198db9a45257435f86df2605992ccb34
I20251025 07:52:28.280063 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 35ba9d23262248cda6fdf03c51be11a4
I20251025 07:52:28.280252 1169 tablet_bootstrap.cc:654] T 94408a3a90224645ba3cacccf5de33f1 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:28.280539 524 catalog_manager.cc:6719] ReplaceTablet: tablet 35ba9d23262248cda6fdf03c51be11a4 deleted and replaced by tablet 01f82980559b4662bdc09786ed429ddc
I20251025 07:52:28.281208 1170 ts_tablet_manager.cc:1929] T a231f3d31b204f7b82f018892ccc8435 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:28.283974 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 35ba9d23262248cda6fdf03c51be11a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 01f82980559b4662bdc09786ed429ddc) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:28.284317 1167 tablet_bootstrap.cc:492] T 2947502815d742e89cacb474b7fbb249 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:28.284504 1167 ts_tablet_manager.cc:1403] T 2947502815d742e89cacb474b7fbb249 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20251025 07:52:28.284700 1170 log.cc:1199] T a231f3d31b204f7b82f018892ccc8435 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/a231f3d31b204f7b82f018892ccc8435
I20251025 07:52:28.285135 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 35ba9d23262248cda6fdf03c51be11a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 01f82980559b4662bdc09786ed429ddc) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:28.285451 1167 raft_consensus.cc:359] T 2947502815d742e89cacb474b7fbb249 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.285665 1167 raft_consensus.cc:385] T 2947502815d742e89cacb474b7fbb249 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:28.285773 1167 raft_consensus.cc:740] T 2947502815d742e89cacb474b7fbb249 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:28.286255 1171 ts_tablet_manager.cc:1929] T 33c608beb4ef4a859a135b11e07c81a4 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:28.286449 1171 log.cc:1199] T 33c608beb4ef4a859a135b11e07c81a4 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/33c608beb4ef4a859a135b11e07c81a4
I20251025 07:52:28.286909 1171 ts_tablet_manager.cc:1950] T 33c608beb4ef4a859a135b11e07c81a4 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:28.287299 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 35ba9d23262248cda6fdf03c51be11a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 01f82980559b4662bdc09786ed429ddc) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:28.287891 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_acca538f4b584f3ea4a55921a708d5a9" } new_table_name: "default.Table_500d949c884c4644b70943543bd3e0c5" modify_external_catalogs: true
I20251025 07:52:28.290462 1170 ts_tablet_manager.cc:1950] T a231f3d31b204f7b82f018892ccc8435 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:28.291761 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet a231f3d31b204f7b82f018892ccc8435 (table default.Table_1c458711d56d4ddeba1d0bc09392a734 [id=7cebd72e604e497db80a8bc823471e97]) successfully deleted
I20251025 07:52:28.293572 1169 tablet_bootstrap.cc:492] T 94408a3a90224645ba3cacccf5de33f1 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:28.293726 1169 ts_tablet_manager.cc:1403] T 94408a3a90224645ba3cacccf5de33f1 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.001s
I20251025 07:52:28.289381 1167 consensus_queue.cc:260] T 2947502815d742e89cacb474b7fbb249 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.294523 1169 raft_consensus.cc:359] T 94408a3a90224645ba3cacccf5de33f1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.294692 1169 raft_consensus.cc:385] T 94408a3a90224645ba3cacccf5de33f1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:28.294780 1169 raft_consensus.cc:740] T 94408a3a90224645ba3cacccf5de33f1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:28.295015 1167 ts_tablet_manager.cc:1434] T 2947502815d742e89cacb474b7fbb249 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251025 07:52:28.295013 1169 consensus_queue.cc:260] T 94408a3a90224645ba3cacccf5de33f1 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.296687 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet c88c46e022ff41a58b02538f2151e260
I20251025 07:52:28.296993 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 33c608beb4ef4a859a135b11e07c81a4 (table default.Table_7a19acf479d64317bd910c04b08a8b4f [id=8bbb3b9204634ee6a04f5a4753739476]) successfully deleted
I20251025 07:52:28.297487 755 tablet_service.cc:1552] Processing DeleteTablet for tablet c88c46e022ff41a58b02538f2151e260 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:28.297849 1167 tablet_bootstrap.cc:492] T 3c3eb43209f24592820638fc25486b78 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:28.298303 1170 tablet_replica.cc:333] T 3c3eb43209f24592820638fc25486b78 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
W20251025 07:52:28.298343 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 2f26667f053a4f16b5da330c0b5e40f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f26667f053a4f16b5da330c0b5e40f9 already in progress: creating tablet
I20251025 07:52:28.299458 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet c88c46e022ff41a58b02538f2151e260 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:28.299748 1172 ts_tablet_manager.cc:1950] T 198db9a45257435f86df2605992ccb34 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:28.300766 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 198db9a45257435f86df2605992ccb34 (table default.Table_79969e4ace6f4227b2748b59136027d6 [id=a1135611169142ba9ef987e3e9f95ce8]) successfully deleted
W20251025 07:52:28.301378 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet abf94c32ff2945119b710905c075d63c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abf94c32ff2945119b710905c075d63c already in progress: creating tablet
I20251025 07:52:28.301607 1167 tablet_bootstrap.cc:654] T 3c3eb43209f24592820638fc25486b78 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:28.303787 1362 raft_consensus.cc:493] T 2947502815d742e89cacb474b7fbb249 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:28.303987 1362 raft_consensus.cc:515] T 2947502815d742e89cacb474b7fbb249 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.304636 891 tablet_service.cc:1552] Processing DeleteTablet for tablet c88c46e022ff41a58b02538f2151e260 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:56590
W20251025 07:52:28.304795 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 5cba9c05dd424cc788febc614a34819a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cba9c05dd424cc788febc614a34819a already in progress: creating tablet
I20251025 07:52:28.305147 1171 tablet_replica.cc:333] T 299572b2dbe244918dcebc5ea5f3c45a P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:28.305842 1171 raft_consensus.cc:2243] T 299572b2dbe244918dcebc5ea5f3c45a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:28.306118 1172 tablet_replica.cc:333] T d0fd00baa02d48278afaa5b8c19182f0 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:28.306914 1171 raft_consensus.cc:2272] T 299572b2dbe244918dcebc5ea5f3c45a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:28.307178 1172 raft_consensus.cc:2243] T d0fd00baa02d48278afaa5b8c19182f0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:28.307751 1171 ts_tablet_manager.cc:1916] T 299572b2dbe244918dcebc5ea5f3c45a P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:28.303851 1374 tablet.cc:1722] T 3d5e0dd14e474fe09993a03851f8b1f0 P 3aea2869ff814cc7b142012faa274382: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251025 07:52:28.310266 1170 raft_consensus.cc:2243] T 3c3eb43209f24592820638fc25486b78 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:28.307971 1172 raft_consensus.cc:2272] T d0fd00baa02d48278afaa5b8c19182f0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:28.306674 1373 tablet.cc:1722] T 3d5e0dd14e474fe09993a03851f8b1f0 P 23eeffd3de254ce18db5febfde971c6b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251025 07:52:28.313512 1172 ts_tablet_manager.cc:1916] T d0fd00baa02d48278afaa5b8c19182f0 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:28.314519 1170 raft_consensus.cc:2272] T 3c3eb43209f24592820638fc25486b78 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:28.315378 1170 ts_tablet_manager.cc:1916] T 3c3eb43209f24592820638fc25486b78 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:28.315724 1166 tablet_bootstrap.cc:492] T dd898450f2604987b08639cf87d0e402 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:28.315903 1166 ts_tablet_manager.cc:1403] T dd898450f2604987b08639cf87d0e402 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.070s user 0.009s sys 0.003s
I20251025 07:52:28.316789 1384 tablet.cc:1722] T 3d5e0dd14e474fe09993a03851f8b1f0 P db42be7ad3c741a0aa427f27e245993b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251025 07:52:28.317888 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2947502815d742e89cacb474b7fbb249" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:28.318401 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_e317f8d728da4c76886f67d0995ab7f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:28.319080 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e317f8d728da4c76886f67d0995ab7f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:28.319214 1169 ts_tablet_manager.cc:1434] T 94408a3a90224645ba3cacccf5de33f1 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20251025 07:52:28.319181 1362 leader_election.cc:290] T 2947502815d742e89cacb474b7fbb249 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:28.319770 710 leader_election.cc:343] T 2947502815d742e89cacb474b7fbb249 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 2947502815d742e89cacb474b7fbb249
I20251025 07:52:28.320329 1169 tablet_bootstrap.cc:492] T 33c608beb4ef4a859a135b11e07c81a4 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
W20251025 07:52:28.320632 712 leader_election.cc:343] T 2947502815d742e89cacb474b7fbb249 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:28.320775 712 leader_election.cc:304] T 2947502815d742e89cacb474b7fbb249 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:28.321177 1362 raft_consensus.cc:2749] T 2947502815d742e89cacb474b7fbb249 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:28.323364 1166 raft_consensus.cc:359] T dd898450f2604987b08639cf87d0e402 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.323911 1166 raft_consensus.cc:385] T dd898450f2604987b08639cf87d0e402 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:28.324008 1166 raft_consensus.cc:740] T dd898450f2604987b08639cf87d0e402 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:28.325534 1169 tablet_bootstrap.cc:654] T 33c608beb4ef4a859a135b11e07c81a4 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:28.327378 1166 consensus_queue.cc:260] T dd898450f2604987b08639cf87d0e402 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.327931 1166 ts_tablet_manager.cc:1434] T dd898450f2604987b08639cf87d0e402 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20251025 07:52:28.328390 1166 tablet_bootstrap.cc:492] T 87a93778620f4c84b3b3f7a3d7d7e87f P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:28.337311 1166 tablet_bootstrap.cc:654] T 87a93778620f4c84b3b3f7a3d7d7e87f P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:28.339097 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet c4610b0c62e741eca28c21f63cc3de81
I20251025 07:52:28.340458 1172 ts_tablet_manager.cc:1929] T d0fd00baa02d48278afaa5b8c19182f0 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:28.340622 1172 log.cc:1199] T d0fd00baa02d48278afaa5b8c19182f0 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/d0fd00baa02d48278afaa5b8c19182f0
I20251025 07:52:28.341033 1172 ts_tablet_manager.cc:1950] T d0fd00baa02d48278afaa5b8c19182f0 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:28.341620 1172 tablet_replica.cc:333] T cf2cab1552bd4e7394797c93b149f450 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:28.341867 1172 raft_consensus.cc:2243] T cf2cab1552bd4e7394797c93b149f450 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:28.342074 1172 raft_consensus.cc:2272] T cf2cab1552bd4e7394797c93b149f450 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:28.343120 1166 tablet_bootstrap.cc:492] T 87a93778620f4c84b3b3f7a3d7d7e87f P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:28.343304 1166 ts_tablet_manager.cc:1403] T 87a93778620f4c84b3b3f7a3d7d7e87f P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20251025 07:52:28.344350 1166 raft_consensus.cc:359] T 87a93778620f4c84b3b3f7a3d7d7e87f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.344558 1166 raft_consensus.cc:385] T 87a93778620f4c84b3b3f7a3d7d7e87f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:28.344645 1166 raft_consensus.cc:740] T 87a93778620f4c84b3b3f7a3d7d7e87f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:28.344916 1172 ts_tablet_manager.cc:1916] T cf2cab1552bd4e7394797c93b149f450 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:28.344885 1166 consensus_queue.cc:260] T 87a93778620f4c84b3b3f7a3d7d7e87f P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.345382 1166 ts_tablet_manager.cc:1434] T 87a93778620f4c84b3b3f7a3d7d7e87f P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 07:52:28.345707 1166 tablet_bootstrap.cc:492] T 5caac9c943bf42a5b2ae522d87fdfe8e P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:28.347960 891 tablet_service.cc:1552] Processing DeleteTablet for tablet c4610b0c62e741eca28c21f63cc3de81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:28.348253 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet d0fd00baa02d48278afaa5b8c19182f0 (table default.Table_1d4029fc57dd4ecb84b8621c33d27958 [id=ff352eacb0614dbbbb6ab6369b9192e8]) successfully deleted
I20251025 07:52:28.349045 755 tablet_service.cc:1552] Processing DeleteTablet for tablet c4610b0c62e741eca28c21f63cc3de81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:28.351423 1166 tablet_bootstrap.cc:654] T 5caac9c943bf42a5b2ae522d87fdfe8e P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:28.352550 1172 ts_tablet_manager.cc:1929] T cf2cab1552bd4e7394797c93b149f450 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:28.352669 1172 log.cc:1199] T cf2cab1552bd4e7394797c93b149f450 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/cf2cab1552bd4e7394797c93b149f450
I20251025 07:52:28.353087 1172 ts_tablet_manager.cc:1950] T cf2cab1552bd4e7394797c93b149f450 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:28.354226 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet cf2cab1552bd4e7394797c93b149f450 (table default.Table_4c12bad86d7946b9ac019c8e3856dcba [id=7de1e93439f94b2ebd4db3ed27f95872]) successfully deleted
I20251025 07:52:28.358352 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_919e690ef3d041c2bcf9b0c9e59d9bc8" } new_table_name: "default.Table_1ff54f4df7a2444489ae4e4060084156" modify_external_catalogs: true
I20251025 07:52:28.358829 1166 tablet_bootstrap.cc:492] T 5caac9c943bf42a5b2ae522d87fdfe8e P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:28.359016 1166 ts_tablet_manager.cc:1403] T 5caac9c943bf42a5b2ae522d87fdfe8e P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20251025 07:52:28.359999 1166 raft_consensus.cc:359] T 5caac9c943bf42a5b2ae522d87fdfe8e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.360271 1166 raft_consensus.cc:385] T 5caac9c943bf42a5b2ae522d87fdfe8e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:28.360369 1166 raft_consensus.cc:740] T 5caac9c943bf42a5b2ae522d87fdfe8e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:28.364068 1171 ts_tablet_manager.cc:1929] T 299572b2dbe244918dcebc5ea5f3c45a P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:28.363997 1166 consensus_queue.cc:260] T 5caac9c943bf42a5b2ae522d87fdfe8e P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.364305 1172 tablet_replica.cc:333] T 299572b2dbe244918dcebc5ea5f3c45a P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:28.364631 1166 ts_tablet_manager.cc:1434] T 5caac9c943bf42a5b2ae522d87fdfe8e P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20251025 07:52:28.365067 1166 tablet_bootstrap.cc:492] T 1b31177a062b467187d68ed2034f5690 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:28.364226 1171 log.cc:1199] T 299572b2dbe244918dcebc5ea5f3c45a P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/299572b2dbe244918dcebc5ea5f3c45a
I20251025 07:52:28.367501 1171 ts_tablet_manager.cc:1950] T 299572b2dbe244918dcebc5ea5f3c45a P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:28.368822 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 299572b2dbe244918dcebc5ea5f3c45a (table default.Table_bd5089a7892041649dd05b7fb0961255 [id=840b3ffcb44144548843457c48f46380]) successfully deleted
I20251025 07:52:28.368965 1172 raft_consensus.cc:2243] T 299572b2dbe244918dcebc5ea5f3c45a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:28.369218 1172 raft_consensus.cc:2272] T 299572b2dbe244918dcebc5ea5f3c45a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:28.370043 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 35ba9d23262248cda6fdf03c51be11a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35ba9d23262248cda6fdf03c51be11a4 already in progress: creating tablet
W20251025 07:52:28.370723 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet c88c46e022ff41a58b02538f2151e260 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c88c46e022ff41a58b02538f2151e260 already in progress: creating tablet
W20251025 07:52:28.371327 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet c4610b0c62e741eca28c21f63cc3de81 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4610b0c62e741eca28c21f63cc3de81 already in progress: creating tablet
I20251025 07:52:28.371932 1172 ts_tablet_manager.cc:1916] T 299572b2dbe244918dcebc5ea5f3c45a P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:28.372675 1167 tablet_bootstrap.cc:492] T 3c3eb43209f24592820638fc25486b78 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:28.373376 1167 ts_tablet_manager.cc:1403] T 3c3eb43209f24592820638fc25486b78 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.078s user 0.015s sys 0.004s
I20251025 07:52:28.374163 1167 raft_consensus.cc:359] T 3c3eb43209f24592820638fc25486b78 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.374368 1167 raft_consensus.cc:385] T 3c3eb43209f24592820638fc25486b78 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:28.374444 1167 raft_consensus.cc:740] T 3c3eb43209f24592820638fc25486b78 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:28.378882 1166 tablet_bootstrap.cc:654] T 1b31177a062b467187d68ed2034f5690 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:28.380707 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet f3001314b56345f5a70314777cae0125
I20251025 07:52:28.383636 1172 ts_tablet_manager.cc:1929] T 299572b2dbe244918dcebc5ea5f3c45a P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:28.383759 1172 log.cc:1199] T 299572b2dbe244918dcebc5ea5f3c45a P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/299572b2dbe244918dcebc5ea5f3c45a
I20251025 07:52:28.384111 1172 ts_tablet_manager.cc:1950] T 299572b2dbe244918dcebc5ea5f3c45a P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:28.384552 1167 consensus_queue.cc:260] T 3c3eb43209f24592820638fc25486b78 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:28.386545 511 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 8e6d76b1a97945cfb6c3055e1b622f8a (table default.Table_10e57b42f97544208675708b1e028846 [id=f5c1be203e2b4831a025871cf8a2ef17], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8e6d76b1a97945cfb6c3055e1b622f8a consensus state has no leader
I20251025 07:52:28.386696 1166 tablet_bootstrap.cc:492] T 1b31177a062b467187d68ed2034f5690 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:28.386704 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 299572b2dbe244918dcebc5ea5f3c45a (table default.Table_bd5089a7892041649dd05b7fb0961255 [id=840b3ffcb44144548843457c48f46380]) successfully deleted
I20251025 07:52:28.386831 1166 ts_tablet_manager.cc:1403] T 1b31177a062b467187d68ed2034f5690 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.002s
W20251025 07:52:28.386948 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 35ba9d23262248cda6fdf03c51be11a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35ba9d23262248cda6fdf03c51be11a4 already in progress: creating tablet
I20251025 07:52:28.387923 755 tablet_service.cc:1552] Processing DeleteTablet for tablet f3001314b56345f5a70314777cae0125 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:28.388381 1167 ts_tablet_manager.cc:1434] T 3c3eb43209f24592820638fc25486b78 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.015s user 0.000s sys 0.003s
I20251025 07:52:28.388701 891 tablet_service.cc:1552] Processing DeleteTablet for tablet f3001314b56345f5a70314777cae0125 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:56590
W20251025 07:52:28.389250 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet c88c46e022ff41a58b02538f2151e260 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c88c46e022ff41a58b02538f2151e260 already in progress: creating tablet
I20251025 07:52:28.389425 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet c4610b0c62e741eca28c21f63cc3de81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:47408
W20251025 07:52:28.389488 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet c4610b0c62e741eca28c21f63cc3de81 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4610b0c62e741eca28c21f63cc3de81 already in progress: creating tablet
W20251025 07:52:28.389698 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet f3001314b56345f5a70314777cae0125 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3001314b56345f5a70314777cae0125 already in progress: creating tablet
I20251025 07:52:28.390179 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_ed69674fe05b48a184be60a6f44fdbea" } new_table_name: "default.Table_5f916bc95ede499b9f4de53471ae1314" modify_external_catalogs: true
I20251025 07:52:28.390242 1166 raft_consensus.cc:359] T 1b31177a062b467187d68ed2034f5690 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.390416 1166 raft_consensus.cc:385] T 1b31177a062b467187d68ed2034f5690 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:28.390506 1166 raft_consensus.cc:740] T 1b31177a062b467187d68ed2034f5690 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:28.390779 1166 consensus_queue.cc:260] T 1b31177a062b467187d68ed2034f5690 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.391485 1166 ts_tablet_manager.cc:1434] T 1b31177a062b467187d68ed2034f5690 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20251025 07:52:28.391911 1166 tablet_bootstrap.cc:492] T 1483c40eee834f20a92e6b4cd89775b6 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:28.392622 518 master_service.cc:954] ReplaceTablet: received request to replace tablet 7fbae8be73bf4e18b5a5c4ec4cdebc12 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:28.394371 1320 raft_consensus.cc:493] T dd898450f2604987b08639cf87d0e402 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:28.394555 1320 raft_consensus.cc:515] T dd898450f2604987b08639cf87d0e402 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.394770 1365 mvcc.cc:204] Tried to move back new op lower bound from 7214607352496115712 to 7214607349576753152. Current Snapshot: MvccSnapshot[applied={T|T < 7214607352496115712}]
I20251025 07:52:28.395237 1320 leader_election.cc:290] T dd898450f2604987b08639cf87d0e402 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:28.400918 1167 tablet_bootstrap.cc:492] T 94408a3a90224645ba3cacccf5de33f1 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:28.401366 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd898450f2604987b08639cf87d0e402" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
W20251025 07:52:28.402096 847 leader_election.cc:343] T dd898450f2604987b08639cf87d0e402 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:28.402724 1167 tablet_bootstrap.cc:654] T 94408a3a90224645ba3cacccf5de33f1 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:28.408408 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet f3001314b56345f5a70314777cae0125 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:28.408854 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd898450f2604987b08639cf87d0e402" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:28.410357 850 leader_election.cc:343] T dd898450f2604987b08639cf87d0e402 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:28.410508 850 leader_election.cc:304] T dd898450f2604987b08639cf87d0e402 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:28.411199 1320 raft_consensus.cc:2749] T dd898450f2604987b08639cf87d0e402 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:28.412170 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet f3001314b56345f5a70314777cae0125 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3001314b56345f5a70314777cae0125 already in progress: creating tablet
I20251025 07:52:28.419409 1170 ts_tablet_manager.cc:1929] T 3c3eb43209f24592820638fc25486b78 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:28.419629 1170 log.cc:1199] T 3c3eb43209f24592820638fc25486b78 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/3c3eb43209f24592820638fc25486b78
I20251025 07:52:28.420111 1170 ts_tablet_manager.cc:1950] T 3c3eb43209f24592820638fc25486b78 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:28.421305 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 3c3eb43209f24592820638fc25486b78 (table default.Table_9b8e648894e14f8595796e4698407495 [id=89d0b4e609a74dde8540eb740446be5a]) successfully deleted
I20251025 07:52:28.424206 1320 raft_consensus.cc:493] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:28.424404 1320 raft_consensus.cc:515] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.424801 1170 tablet_replica.cc:333] T 198db9a45257435f86df2605992ccb34 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:28.425072 1170 raft_consensus.cc:2243] T 198db9a45257435f86df2605992ccb34 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:28.425113 1320 leader_election.cc:290] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:28.425287 1170 raft_consensus.cc:2272] T 198db9a45257435f86df2605992ccb34 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:28.426594 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50ffbcdd4a9247bf99b8e2361bc02ef5" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:28.426803 1048 raft_consensus.cc:2468] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:28.427635 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50ffbcdd4a9247bf99b8e2361bc02ef5" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:28.427726 850 leader_election.cc:304] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:28.427871 776 raft_consensus.cc:2468] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:28.428489 1320 raft_consensus.cc:2804] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:28.428583 1320 raft_consensus.cc:493] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:28.428682 1320 raft_consensus.cc:3060] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:28.430359 1247 raft_consensus.cc:493] T 94408a3a90224645ba3cacccf5de33f1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:28.430543 1247 raft_consensus.cc:515] T 94408a3a90224645ba3cacccf5de33f1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:28.431035 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c59f6e6eb9594a8e83f109a09b034045 consensus state has no leader
I20251025 07:52:28.431391 1247 leader_election.cc:290] T 94408a3a90224645ba3cacccf5de33f1 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:28.432237 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94408a3a90224645ba3cacccf5de33f1" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
W20251025 07:52:28.432974 982 leader_election.cc:343] T 94408a3a90224645ba3cacccf5de33f1 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:28.433054 1320 raft_consensus.cc:515] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:28.433354 983 leader_election.cc:343] T 94408a3a90224645ba3cacccf5de33f1 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 94408a3a90224645ba3cacccf5de33f1
I20251025 07:52:28.433518 983 leader_election.cc:304] T 94408a3a90224645ba3cacccf5de33f1 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:28.434022 1247 raft_consensus.cc:2749] T 94408a3a90224645ba3cacccf5de33f1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:28.434216 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50ffbcdd4a9247bf99b8e2361bc02ef5" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:28.434407 1048 raft_consensus.cc:3060] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:28.435091 1170 ts_tablet_manager.cc:1916] T 198db9a45257435f86df2605992ccb34 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:28.438458 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b94634ac5077439fadfbbce569f60e8b has no consensus state
I20251025 07:52:28.438452 1048 raft_consensus.cc:2468] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:28.439353 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50ffbcdd4a9247bf99b8e2361bc02ef5" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:28.439581 776 raft_consensus.cc:3060] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:28.439551 850 leader_election.cc:304] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:28.439996 1320 leader_election.cc:290] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:28.440255 1320 raft_consensus.cc:2804] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:28.440428 1320 raft_consensus.cc:697] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:28.440712 1320 consensus_queue.cc:237] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.442751 776 raft_consensus.cc:2468] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:28.443272 1397 tablet.cc:1722] T 2fac7b819e2642378e0d3f9230d0259c P db42be7ad3c741a0aa427f27e245993b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251025 07:52:28.447113 1166 tablet_bootstrap.cc:654] T 1483c40eee834f20a92e6b4cd89775b6 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:28.450666 1247 raft_consensus.cc:493] T b00153c97104470883f6089d71d23708 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:28.450824 1247 raft_consensus.cc:515] T b00153c97104470883f6089d71d23708 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.451661 1247 leader_election.cc:290] T b00153c97104470883f6089d71d23708 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:28.452975 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b00153c97104470883f6089d71d23708" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
W20251025 07:52:28.454366 982 leader_election.cc:343] T b00153c97104470883f6089d71d23708 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:28.454660 983 leader_election.cc:343] T b00153c97104470883f6089d71d23708 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: b00153c97104470883f6089d71d23708
I20251025 07:52:28.454808 983 leader_election.cc:304] T b00153c97104470883f6089d71d23708 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:28.455678 1247 raft_consensus.cc:2749] T b00153c97104470883f6089d71d23708 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:28.466600 1167 tablet_bootstrap.cc:492] T 94408a3a90224645ba3cacccf5de33f1 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:28.466792 1167 ts_tablet_manager.cc:1403] T 94408a3a90224645ba3cacccf5de33f1 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.066s user 0.008s sys 0.008s
I20251025 07:52:28.467438 1166 tablet_bootstrap.cc:492] T 1483c40eee834f20a92e6b4cd89775b6 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:28.470168 1169 tablet_bootstrap.cc:492] T 33c608beb4ef4a859a135b11e07c81a4 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:28.470397 1169 ts_tablet_manager.cc:1403] T 33c608beb4ef4a859a135b11e07c81a4 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.150s user 0.008s sys 0.005s
I20251025 07:52:28.470412 1167 raft_consensus.cc:359] T 94408a3a90224645ba3cacccf5de33f1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.471582 1169 raft_consensus.cc:359] T 33c608beb4ef4a859a135b11e07c81a4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.471845 1169 raft_consensus.cc:385] T 33c608beb4ef4a859a135b11e07c81a4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:28.471930 1169 raft_consensus.cc:740] T 33c608beb4ef4a859a135b11e07c81a4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:28.472223 1169 consensus_queue.cc:260] T 33c608beb4ef4a859a135b11e07c81a4 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.473156 1167 raft_consensus.cc:385] T 94408a3a90224645ba3cacccf5de33f1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:28.473417 1169 ts_tablet_manager.cc:1434] T 33c608beb4ef4a859a135b11e07c81a4 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251025 07:52:28.473277 1167 raft_consensus.cc:740] T 94408a3a90224645ba3cacccf5de33f1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:28.473760 1169 tablet_bootstrap.cc:492] T 2947502815d742e89cacb474b7fbb249 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:28.474751 1167 consensus_queue.cc:260] T 94408a3a90224645ba3cacccf5de33f1 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.475329 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_4b70d9acf0f64bcd992a894b129927c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:28.476006 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b70d9acf0f64bcd992a894b129927c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:28.476490 1169 tablet_bootstrap.cc:654] T 2947502815d742e89cacb474b7fbb249 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:28.477304 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_f71af05e7bf84337ae6f7da79d58bcfb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:28.477823 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f71af05e7bf84337ae6f7da79d58bcfb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:28.478264 1166 ts_tablet_manager.cc:1403] T 1483c40eee834f20a92e6b4cd89775b6 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.086s user 0.003s sys 0.002s
I20251025 07:52:28.479346 1166 raft_consensus.cc:359] T 1483c40eee834f20a92e6b4cd89775b6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.479597 1166 raft_consensus.cc:385] T 1483c40eee834f20a92e6b4cd89775b6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:28.479677 1166 raft_consensus.cc:740] T 1483c40eee834f20a92e6b4cd89775b6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:28.479984 1166 consensus_queue.cc:260] T 1483c40eee834f20a92e6b4cd89775b6 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.480293 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_c57548f494f84e5d92c0e461a7dde7d3" } new_table_name: "default.Table_2c1cdea0079240b889637e5d8d13da95" modify_external_catalogs: true
I20251025 07:52:28.482093 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_f5af41a9cdee45ffa4a44089ba80464b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:28.482613 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f5af41a9cdee45ffa4a44089ba80464b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:28.485298 1166 ts_tablet_manager.cc:1434] T 1483c40eee834f20a92e6b4cd89775b6 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251025 07:52:28.486961 1166 tablet_bootstrap.cc:492] T f51ec427dd9349a48eb221db04cf242b P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:28.489372 1166 tablet_bootstrap.cc:654] T f51ec427dd9349a48eb221db04cf242b P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:28.493888 1167 ts_tablet_manager.cc:1434] T 94408a3a90224645ba3cacccf5de33f1 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.027s user 0.002s sys 0.001s
I20251025 07:52:28.494282 1167 tablet_bootstrap.cc:492] T 83084be89b2646b0bda208b3f6f60d77 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:28.496074 1167 tablet_bootstrap.cc:654] T 83084be89b2646b0bda208b3f6f60d77 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:28.497640 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a6ddefa1738f4ff98586900c344359d2 consensus state has no leader
I20251025 07:52:28.498064 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_0eca32ca64da44198140ae7ec9a77767" } modify_external_catalogs: true
I20251025 07:52:28.498296 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_0eca32ca64da44198140ae7ec9a77767" } modify_external_catalogs: true
I20251025 07:52:28.500385 1167 tablet_bootstrap.cc:492] T 83084be89b2646b0bda208b3f6f60d77 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:28.500551 1167 ts_tablet_manager.cc:1403] T 83084be89b2646b0bda208b3f6f60d77 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20251025 07:52:28.501312 1167 raft_consensus.cc:359] T 83084be89b2646b0bda208b3f6f60d77 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.501513 1167 raft_consensus.cc:385] T 83084be89b2646b0bda208b3f6f60d77 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:28.501597 1167 raft_consensus.cc:740] T 83084be89b2646b0bda208b3f6f60d77 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:28.501860 1167 consensus_queue.cc:260] T 83084be89b2646b0bda208b3f6f60d77 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.502398 1167 ts_tablet_manager.cc:1434] T 83084be89b2646b0bda208b3f6f60d77 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:28.502700 1167 tablet_bootstrap.cc:492] T 9bf486d82b984d2c89ee32e84a85894f P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:28.505352 1167 tablet_bootstrap.cc:654] T 9bf486d82b984d2c89ee32e84a85894f P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:28.506476 1170 ts_tablet_manager.cc:1929] T 198db9a45257435f86df2605992ccb34 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:28.506626 1170 log.cc:1199] T 198db9a45257435f86df2605992ccb34 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/198db9a45257435f86df2605992ccb34
I20251025 07:52:28.507042 1170 ts_tablet_manager.cc:1950] T 198db9a45257435f86df2605992ccb34 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:28.508332 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 198db9a45257435f86df2605992ccb34 (table default.Table_79969e4ace6f4227b2748b59136027d6 [id=a1135611169142ba9ef987e3e9f95ce8]) successfully deleted
I20251025 07:52:28.508652 891 tablet_service.cc:1505] Processing CreateTablet for tablet ef9a2c9f2ccf432ab835396020872208 (DEFAULT_TABLE table=default.Table_2d41b478c59b488bb7aadc3ce393c24c [id=409fab60f9d744798e6f92b868f0f7c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.509089 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef9a2c9f2ccf432ab835396020872208. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.509258 1025 tablet_service.cc:1505] Processing CreateTablet for tablet ef9a2c9f2ccf432ab835396020872208 (DEFAULT_TABLE table=default.Table_2d41b478c59b488bb7aadc3ce393c24c [id=409fab60f9d744798e6f92b868f0f7c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.509672 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef9a2c9f2ccf432ab835396020872208. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.510505 519 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 796413da731245459cc5dffacc3deb2f (table default.Table_293596c64cca4768bc5869fd9924f856 [id=d1b9d89b22bc4431a7d59ceeb9d75050]). Expected version 2 got 0
I20251025 07:52:28.511230 519 master_service.cc:954] ReplaceTablet: received request to replace tablet 299572b2dbe244918dcebc5ea5f3c45a from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:28.511706 525 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 796413da731245459cc5dffacc3deb2f (table default.Table_293596c64cca4768bc5869fd9924f856 [id=d1b9d89b22bc4431a7d59ceeb9d75050]). Expected version 2 got 0
I20251025 07:52:28.511795 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_4a7945d6aa00429b98f77ca4b9f3e3bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:28.512248 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 2fac7b819e2642378e0d3f9230d0259c
W20251025 07:52:28.512364 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a7945d6aa00429b98f77ca4b9f3e3bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:28.512822 517 master_service.cc:954] ReplaceTablet: received request to replace tablet 3abfd52327a04ba88625f391a8658fbb from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:28.514354 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_0f2c837d88c043fda9d4eb94706a5398"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:28.514880 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f2c837d88c043fda9d4eb94706a5398 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:28.515409 1020 tablet_service.cc:1505] Processing CreateTablet for tablet fe5033e5ff404a5ebc6c8443d7548327 (DEFAULT_TABLE table=default.Table_054db0ca31bd42ebb5d61ec81db8146b [id=85f4e4639f224541a0dec677ff5e62d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.515813 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe5033e5ff404a5ebc6c8443d7548327. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.517449 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 2fac7b819e2642378e0d3f9230d0259c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:28.517792 884 tablet_service.cc:1505] Processing CreateTablet for tablet fe5033e5ff404a5ebc6c8443d7548327 (DEFAULT_TABLE table=default.Table_054db0ca31bd42ebb5d61ec81db8146b [id=85f4e4639f224541a0dec677ff5e62d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.518187 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe5033e5ff404a5ebc6c8443d7548327. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.519111 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 943ec703e65f4fd3b0c690e08c0bd226 (DEFAULT_TABLE table=default.Table_db98182d07a54725823623925f51a0cb [id=62cf364c073b47a1939f41dd69222582]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.519536 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 943ec703e65f4fd3b0c690e08c0bd226. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.520666 755 tablet_service.cc:1505] Processing CreateTablet for tablet ef9a2c9f2ccf432ab835396020872208 (DEFAULT_TABLE table=default.Table_2d41b478c59b488bb7aadc3ce393c24c [id=409fab60f9d744798e6f92b868f0f7c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.521034 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef9a2c9f2ccf432ab835396020872208. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.522444 886 tablet_service.cc:1505] Processing CreateTablet for tablet 943ec703e65f4fd3b0c690e08c0bd226 (DEFAULT_TABLE table=default.Table_db98182d07a54725823623925f51a0cb [id=62cf364c073b47a1939f41dd69222582]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.522836 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 943ec703e65f4fd3b0c690e08c0bd226. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.526296 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 2fac7b819e2642378e0d3f9230d0259c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:28.526634 1172 tablet_replica.cc:333] T 2fac7b819e2642378e0d3f9230d0259c P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:28.526839 1172 raft_consensus.cc:2243] T 2fac7b819e2642378e0d3f9230d0259c P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:28.527076 1172 raft_consensus.cc:2272] T 2fac7b819e2642378e0d3f9230d0259c P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:28.530748 1018 tablet_service.cc:1505] Processing CreateTablet for tablet 88bfc23acd274041a9457079c1709d82 (DEFAULT_TABLE table=default.Table_45f85cb4514d4ec3b06cff95dd4a70f6 [id=b7073684e63d4875841df3dd5171bcc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.531117 525 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 50ffbcdd4a9247bf99b8e2361bc02ef5 (table default.Table_9f0ae2442a334cf8ab13f2eba387ebf3 [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 3 got 0
I20251025 07:52:28.531214 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88bfc23acd274041a9457079c1709d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.533450 1172 ts_tablet_manager.cc:1916] T 2fac7b819e2642378e0d3f9230d0259c P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:28.537338 889 tablet_service.cc:1505] Processing CreateTablet for tablet 88bfc23acd274041a9457079c1709d82 (DEFAULT_TABLE table=default.Table_45f85cb4514d4ec3b06cff95dd4a70f6 [id=b7073684e63d4875841df3dd5171bcc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.537782 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88bfc23acd274041a9457079c1709d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.539129 1020 tablet_service.cc:1505] Processing CreateTablet for tablet aa9d9ab8073945d28d52fb02213f201a (DEFAULT_TABLE table=default.Table_c0432016895e4094b7ff582abe612bbd [id=dfd20cf9c3e541c4abe23d59e0aa7f2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.539672 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa9d9ab8073945d28d52fb02213f201a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.542284 892 tablet_service.cc:1505] Processing CreateTablet for tablet aa9d9ab8073945d28d52fb02213f201a (DEFAULT_TABLE table=default.Table_c0432016895e4094b7ff582abe612bbd [id=dfd20cf9c3e541c4abe23d59e0aa7f2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.542632 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa9d9ab8073945d28d52fb02213f201a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.544548 1026 tablet_service.cc:1505] Processing CreateTablet for tablet f4578ce69a0d4322944e5ce7c8a59831 (DEFAULT_TABLE table=default.Table_3bd421885a1b4e06b11d12780c34cad4 [id=4d9e99d6474b4d00bc66e26a33694e69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.544890 883 tablet_service.cc:1505] Processing CreateTablet for tablet ebf3a2a926754305b5bd95fcebb25d29 (DEFAULT_TABLE table=default.Table_ac4beca0c3bd4cf29565faf5323d9ff7 [id=857bcd1dbe2e477bbc0d77f79a58f7f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.544988 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4578ce69a0d4322944e5ce7c8a59831. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.545241 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebf3a2a926754305b5bd95fcebb25d29. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.545837 1023 tablet_service.cc:1505] Processing CreateTablet for tablet e54fe7cbcffa40c4b04144b47bee2b48 (DEFAULT_TABLE table=default.Table_590cc281d8b042b19ef12c7f6436752a [id=b1eee5c0243a48c4bbdecc0b4f38ab2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.546232 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e54fe7cbcffa40c4b04144b47bee2b48. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.547441 1019 tablet_service.cc:1505] Processing CreateTablet for tablet b947ecc352e54f4caa73c6ca97372256 (DEFAULT_TABLE table=default.Table_503b86ee2ab545afa9829bebcc9ea11d [id=395c91146476423dad0455a12eb5a8af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.547827 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b947ecc352e54f4caa73c6ca97372256. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.550338 1170 tablet_replica.cc:333] T b00153c97104470883f6089d71d23708 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:28.550428 890 tablet_service.cc:1505] Processing CreateTablet for tablet f4578ce69a0d4322944e5ce7c8a59831 (DEFAULT_TABLE table=default.Table_3bd421885a1b4e06b11d12780c34cad4 [id=4d9e99d6474b4d00bc66e26a33694e69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.550567 1170 raft_consensus.cc:2243] T b00153c97104470883f6089d71d23708 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:28.550750 1170 raft_consensus.cc:2272] T b00153c97104470883f6089d71d23708 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:28.550799 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4578ce69a0d4322944e5ce7c8a59831. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.551339 1170 ts_tablet_manager.cc:1916] T b00153c97104470883f6089d71d23708 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:28.554224 885 tablet_service.cc:1505] Processing CreateTablet for tablet e54fe7cbcffa40c4b04144b47bee2b48 (DEFAULT_TABLE table=default.Table_590cc281d8b042b19ef12c7f6436752a [id=b1eee5c0243a48c4bbdecc0b4f38ab2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.554548 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e54fe7cbcffa40c4b04144b47bee2b48. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.555015 894 tablet_service.cc:1505] Processing CreateTablet for tablet b947ecc352e54f4caa73c6ca97372256 (DEFAULT_TABLE table=default.Table_503b86ee2ab545afa9829bebcc9ea11d [id=395c91146476423dad0455a12eb5a8af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.555349 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b947ecc352e54f4caa73c6ca97372256. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.555542 753 tablet_service.cc:1505] Processing CreateTablet for tablet fe5033e5ff404a5ebc6c8443d7548327 (DEFAULT_TABLE table=default.Table_054db0ca31bd42ebb5d61ec81db8146b [id=85f4e4639f224541a0dec677ff5e62d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.558432 739 tablet_service.cc:1552] Processing DeleteTablet for tablet 2fac7b819e2642378e0d3f9230d0259c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:28.558683 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe5033e5ff404a5ebc6c8443d7548327. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.558743 1171 tablet_replica.cc:333] T 2fac7b819e2642378e0d3f9230d0259c P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:28.560074 1171 raft_consensus.cc:2243] T 2fac7b819e2642378e0d3f9230d0259c P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:28.560459 1171 raft_consensus.cc:2272] T 2fac7b819e2642378e0d3f9230d0259c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:28.561156 741 tablet_service.cc:1505] Processing CreateTablet for tablet 88bfc23acd274041a9457079c1709d82 (DEFAULT_TABLE table=default.Table_45f85cb4514d4ec3b06cff95dd4a70f6 [id=b7073684e63d4875841df3dd5171bcc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.564700 526 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 796413da731245459cc5dffacc3deb2f (table default.Table_293596c64cca4768bc5869fd9924f856 [id=d1b9d89b22bc4431a7d59ceeb9d75050]). Expected version 2 got 0
W20251025 07:52:28.565282 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 887e7c7a5d0e4aaf88f26246306df19c consensus state has no leader
I20251025 07:52:28.565630 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_af3343b5c3c244deb6f22b9fba10c2b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:28.566251 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af3343b5c3c244deb6f22b9fba10c2b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:28.567589 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_6140b1f390084256848ae1b746059268"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:28.568104 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6140b1f390084256848ae1b746059268 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:28.571059 1170 ts_tablet_manager.cc:1929] T b00153c97104470883f6089d71d23708 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:28.571223 1170 log.cc:1199] T b00153c97104470883f6089d71d23708 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/b00153c97104470883f6089d71d23708
I20251025 07:52:28.571589 1170 ts_tablet_manager.cc:1950] T b00153c97104470883f6089d71d23708 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:28.574401 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 7fbae8be73bf4e18b5a5c4ec4cdebc12
I20251025 07:52:28.574836 518 catalog_manager.cc:6719] ReplaceTablet: tablet 7fbae8be73bf4e18b5a5c4ec4cdebc12 deleted and replaced by tablet 4c155aa7fa5b4d68bbab1918b8432372
I20251025 07:52:28.575138 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_b6f727a336084bbf8d7d9c25a95105ba" } new_table_name: "default.Table_05d7a0bdab8b438591303fb459a61a68" modify_external_catalogs: true
I20251025 07:52:28.576609 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 3abfd52327a04ba88625f391a8658fbb
I20251025 07:52:28.577011 517 catalog_manager.cc:6719] ReplaceTablet: tablet 3abfd52327a04ba88625f391a8658fbb deleted and replaced by tablet 04d27d54fdb7483b869408af65ff85c9
I20251025 07:52:28.577214 517 master_service.cc:954] ReplaceTablet: received request to replace tablet d5e5e5bbcf9f43118eea502b69880af5 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:28.577704 517 master_service.cc:954] ReplaceTablet: received request to replace tablet 27651312cc2e42a8bfcd245f5d301679 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:28.578353 1170 tablet_replica.cc:333] T 299572b2dbe244918dcebc5ea5f3c45a P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:28.578696 1170 raft_consensus.cc:2243] T 299572b2dbe244918dcebc5ea5f3c45a P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:28.579015 1170 pending_rounds.cc:70] T 299572b2dbe244918dcebc5ea5f3c45a P db42be7ad3c741a0aa427f27e245993b: Trying to abort 1 pending ops.
I20251025 07:52:28.579075 1170 pending_rounds.cc:77] T 299572b2dbe244918dcebc5ea5f3c45a P db42be7ad3c741a0aa427f27e245993b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214607351168978944 op_type: NO_OP noop_request { }
I20251025 07:52:28.579088 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 7fbae8be73bf4e18b5a5c4ec4cdebc12 with delete_type TABLET_DATA_DELETED (replaced by tablet 4c155aa7fa5b4d68bbab1918b8432372) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:28.579219 1170 raft_consensus.cc:2889] T 299572b2dbe244918dcebc5ea5f3c45a P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 07:52:28.579319 1170 raft_consensus.cc:2272] T 299572b2dbe244918dcebc5ea5f3c45a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:28.579334 882 tablet_service.cc:1552] Processing DeleteTablet for tablet 3abfd52327a04ba88625f391a8658fbb with delete_type TABLET_DATA_DELETED (replaced by tablet 04d27d54fdb7483b869408af65ff85c9) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:28.579975 521 catalog_manager.cc:5649] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } }
I20251025 07:52:28.580255 1170 ts_tablet_manager.cc:1916] T 299572b2dbe244918dcebc5ea5f3c45a P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:28.580430 521 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 50ffbcdd4a9247bf99b8e2361bc02ef5 (table default.Table_9f0ae2442a334cf8ab13f2eba387ebf3 [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 3 got 0
I20251025 07:52:28.581235 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 87a93778620f4c84b3b3f7a3d7d7e87f
I20251025 07:52:28.581872 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_590cc281d8b042b19ef12c7f6436752a" } modify_external_catalogs: true
I20251025 07:52:28.582064 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_590cc281d8b042b19ef12c7f6436752a" } modify_external_catalogs: true
I20251025 07:52:28.582391 882 tablet_service.cc:1552] Processing DeleteTablet for tablet 87a93778620f4c84b3b3f7a3d7d7e87f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:28.583667 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_11117cabb5ca4540a58aac25d02964b2" } modify_external_catalogs: true
I20251025 07:52:28.583829 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_11117cabb5ca4540a58aac25d02964b2" } modify_external_catalogs: true
I20251025 07:52:28.585306 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet b00153c97104470883f6089d71d23708 (table default.Table_b330f1a41da2427995e699545d5fc49b [id=7ec5f704dc254c11b865c660ffabc4fe]) successfully deleted
W20251025 07:52:28.585747 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 5cba9c05dd424cc788febc614a34819a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cba9c05dd424cc788febc614a34819a already in progress: creating tablet
I20251025 07:52:28.585774 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 3abfd52327a04ba88625f391a8658fbb with delete_type TABLET_DATA_DELETED (replaced by tablet 04d27d54fdb7483b869408af65ff85c9) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:28.586042 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet 7fbae8be73bf4e18b5a5c4ec4cdebc12 with delete_type TABLET_DATA_DELETED (replaced by tablet 4c155aa7fa5b4d68bbab1918b8432372) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:28.590536 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_2c1cdea0079240b889637e5d8d13da95" } modify_external_catalogs: true
I20251025 07:52:28.590731 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_2c1cdea0079240b889637e5d8d13da95" } modify_external_catalogs: true
I20251025 07:52:28.597941 1170 ts_tablet_manager.cc:1929] T 299572b2dbe244918dcebc5ea5f3c45a P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:28.601693 1320 raft_consensus.cc:493] T 796413da731245459cc5dffacc3deb2f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:28.601859 1320 raft_consensus.cc:515] T 796413da731245459cc5dffacc3deb2f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.602679 1320 leader_election.cc:290] T 796413da731245459cc5dffacc3deb2f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:28.602378 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 87a93778620f4c84b3b3f7a3d7d7e87f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:28.602968 1170 log.cc:1199] T 299572b2dbe244918dcebc5ea5f3c45a P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/299572b2dbe244918dcebc5ea5f3c45a
I20251025 07:52:28.603066 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "796413da731245459cc5dffacc3deb2f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:28.603304 1049 raft_consensus.cc:2468] T 796413da731245459cc5dffacc3deb2f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:28.603518 1170 ts_tablet_manager.cc:1950] T 299572b2dbe244918dcebc5ea5f3c45a P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:28.603614 1025 tablet_service.cc:1505] Processing CreateTablet for tablet ebf3a2a926754305b5bd95fcebb25d29 (DEFAULT_TABLE table=default.Table_ac4beca0c3bd4cf29565faf5323d9ff7 [id=857bcd1dbe2e477bbc0d77f79a58f7f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.604055 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebf3a2a926754305b5bd95fcebb25d29. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.604115 850 leader_election.cc:304] T 796413da731245459cc5dffacc3deb2f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:28.604394 1320 raft_consensus.cc:2804] T 796413da731245459cc5dffacc3deb2f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:28.604513 1320 raft_consensus.cc:493] T 796413da731245459cc5dffacc3deb2f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:28.604593 1320 raft_consensus.cc:3060] T 796413da731245459cc5dffacc3deb2f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:28.606078 1170 tablet_replica.cc:333] T 2fac7b819e2642378e0d3f9230d0259c P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:28.606751 1170 raft_consensus.cc:2243] T 2fac7b819e2642378e0d3f9230d0259c P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:28.607065 1170 raft_consensus.cc:2272] T 2fac7b819e2642378e0d3f9230d0259c P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:28.607812 1320 raft_consensus.cc:515] T 796413da731245459cc5dffacc3deb2f P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.608845 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "796413da731245459cc5dffacc3deb2f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:28.609036 1049 raft_consensus.cc:3060] T 796413da731245459cc5dffacc3deb2f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:28.611089 1170 ts_tablet_manager.cc:1916] T 2fac7b819e2642378e0d3f9230d0259c P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:28.612511 1049 raft_consensus.cc:2468] T 796413da731245459cc5dffacc3deb2f P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:28.612928 850 leader_election.cc:304] T 796413da731245459cc5dffacc3deb2f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:28.613204 1326 raft_consensus.cc:2804] T 796413da731245459cc5dffacc3deb2f P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:28.613374 1326 raft_consensus.cc:697] T 796413da731245459cc5dffacc3deb2f P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:28.613646 1326 consensus_queue.cc:237] T 796413da731245459cc5dffacc3deb2f P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.614409 1368 raft_consensus.cc:493] T 87a93778620f4c84b3b3f7a3d7d7e87f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:28.614549 1368 raft_consensus.cc:515] T 87a93778620f4c84b3b3f7a3d7d7e87f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.615191 1368 leader_election.cc:290] T 87a93778620f4c84b3b3f7a3d7d7e87f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:28.622180 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 299572b2dbe244918dcebc5ea5f3c45a (table default.Table_bd5089a7892041649dd05b7fb0961255 [id=840b3ffcb44144548843457c48f46380]) successfully deleted
I20251025 07:52:28.623521 1018 tablet_service.cc:1505] Processing CreateTablet for tablet f43ef51ee45d47a99ec732b49d50d3b4 (DEFAULT_TABLE table=default.Table_0b0bd6b71b92454fb8cdf967ffd8349e [id=e9b21659de7d4276aa1f9effb7076eb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.623757 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 80fe3255ed614270bd8352a2853fbc43 (DEFAULT_TABLE table=default.Table_d2a7e16c6f0d4b2886c16bfe6b685c00 [id=64c18a10455c4694babcc5ec0f463b85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.623729 1020 tablet_service.cc:1505] Processing CreateTablet for tablet b94634ac5077439fadfbbce569f60e8b (DEFAULT_TABLE table=default.Table_1ff54f4df7a2444489ae4e4060084156 [id=b88c37a678344c6b8e8017553e486cb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.623961 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f43ef51ee45d47a99ec732b49d50d3b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.624229 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b94634ac5077439fadfbbce569f60e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.624500 1019 tablet_service.cc:1505] Processing CreateTablet for tablet e9f7cd765d614d59a7c190b744b0844d (DEFAULT_TABLE table=default.Table_5ecc1936fc8447669d0602cfd6133c6b [id=551028aad5c246378c7acb8e443853a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.624604 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80fe3255ed614270bd8352a2853fbc43. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.624897 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9f7cd765d614d59a7c190b744b0844d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:28.634945 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 35ba9d23262248cda6fdf03c51be11a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35ba9d23262248cda6fdf03c51be11a4 already in progress: creating tablet
W20251025 07:52:28.635249 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet c88c46e022ff41a58b02538f2151e260 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c88c46e022ff41a58b02538f2151e260 already in progress: creating tablet
W20251025 07:52:28.636420 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet c4610b0c62e741eca28c21f63cc3de81 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4610b0c62e741eca28c21f63cc3de81 already in progress: creating tablet
W20251025 07:52:28.636656 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet f3001314b56345f5a70314777cae0125 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3001314b56345f5a70314777cae0125 already in progress: creating tablet
I20251025 07:52:28.637878 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet e54fe7cbcffa40c4b04144b47bee2b48
I20251025 07:52:28.638895 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_86de368dc5bb4eb2a051a9db92a580dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:28.631477 525 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: default.Table_20b987167f0e4863824e7bb7c17e5c03 [id=1b5db2f5053c4cb3a3b40eb109d5d5d9] alter complete (version 2)
I20251025 07:52:28.639940 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet e54fe7cbcffa40c4b04144b47bee2b48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:47408
W20251025 07:52:28.640946 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86de368dc5bb4eb2a051a9db92a580dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:28.642590 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87a93778620f4c84b3b3f7a3d7d7e87f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:28.642750 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 27651312cc2e42a8bfcd245f5d301679
W20251025 07:52:28.643540 850 leader_election.cc:343] T 87a93778620f4c84b3b3f7a3d7d7e87f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:28.644141 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 87268e27540c4ec58ce1e62df819f19d consensus state has no leader
I20251025 07:52:28.644654 517 catalog_manager.cc:6719] ReplaceTablet: tablet 27651312cc2e42a8bfcd245f5d301679 deleted and replaced by tablet a4e4b99be0434baaa75d988281797846
I20251025 07:52:28.645784 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 742b3ec3f6604647b9e397532f1673dd
I20251025 07:52:28.645816 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet 27651312cc2e42a8bfcd245f5d301679 with delete_type TABLET_DATA_DELETED (replaced by tablet a4e4b99be0434baaa75d988281797846) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:28.646507 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 887e7c7a5d0e4aaf88f26246306df19c
I20251025 07:52:28.646869 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet 742b3ec3f6604647b9e397532f1673dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:28.647936 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet 887e7c7a5d0e4aaf88f26246306df19c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:28.652067 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet 94408a3a90224645ba3cacccf5de33f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:28.652489 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_f71af05e7bf84337ae6f7da79d58bcfb" } new_table_name: "default.Table_22ddca5b9e8449ef9fbdeb29212ff9a5" modify_external_catalogs: true
I20251025 07:52:28.654263 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 33c608beb4ef4a859a135b11e07c81a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:28.654592 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet d0fd00baa02d48278afaa5b8c19182f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:28.654611 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_dde55dddf5d14de084003c22da08f6cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:28.655184 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dde55dddf5d14de084003c22da08f6cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:28.656561 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_adb643980c8844de93515aee9326e2df" } new_table_name: "default.Table_bb258691f8844d85ab37eb73e9a4c314" modify_external_catalogs: true
I20251025 07:52:28.657089 1019 tablet_service.cc:1505] Processing CreateTablet for tablet 7c7a9d7a4fb3449bb57eafa83230a839 (DEFAULT_TABLE table=default.Table_192549a3e67840f3aefc31029cf2ff52 [id=a37be02f8e994f82baf2f4f5ec800926]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.657049 1023 tablet_service.cc:1505] Processing CreateTablet for tablet f97acc9a638b4d5a968dcc3412a384bc (DEFAULT_TABLE table=default.Table_04c3923fa1624c52b34d968919e406ae [id=0827603890c8466082bc6c306c9c66af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.657689 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f97acc9a638b4d5a968dcc3412a384bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.658048 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_d4c06afc318d4f35a73d15e893b90da8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:28.658540 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d4c06afc318d4f35a73d15e893b90da8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:28.665460 890 tablet_service.cc:1505] Processing CreateTablet for tablet e9f7cd765d614d59a7c190b744b0844d (DEFAULT_TABLE table=default.Table_5ecc1936fc8447669d0602cfd6133c6b [id=551028aad5c246378c7acb8e443853a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.666657 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c7a9d7a4fb3449bb57eafa83230a839. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.667296 1049 raft_consensus.cc:1275] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Refusing update from remote peer 3aea2869ff814cc7b142012faa274382: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 07:52:28.665917 894 tablet_service.cc:1505] Processing CreateTablet for tablet b94634ac5077439fadfbbce569f60e8b (DEFAULT_TABLE table=default.Table_1ff54f4df7a2444489ae4e4060084156 [id=b88c37a678344c6b8e8017553e486cb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.668560 882 tablet_service.cc:1552] Processing DeleteTablet for tablet e54fe7cbcffa40c4b04144b47bee2b48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:28.669116 1368 consensus_queue.cc:1048] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 07:52:28.670661 743 tablet_service.cc:1505] Processing CreateTablet for tablet 943ec703e65f4fd3b0c690e08c0bd226 (DEFAULT_TABLE table=default.Table_db98182d07a54725823623925f51a0cb [id=62cf364c073b47a1939f41dd69222582]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.671063 1170 ts_tablet_manager.cc:1929] T 2fac7b819e2642378e0d3f9230d0259c P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 07:52:28.671087 1171 ts_tablet_manager.cc:1916] T 2fac7b819e2642378e0d3f9230d0259c P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:28.671193 1170 log.cc:1199] T 2fac7b819e2642378e0d3f9230d0259c P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/2fac7b819e2642378e0d3f9230d0259c
I20251025 07:52:28.671277 739 tablet_service.cc:1505] Processing CreateTablet for tablet aa9d9ab8073945d28d52fb02213f201a (DEFAULT_TABLE table=default.Table_c0432016895e4094b7ff582abe612bbd [id=dfd20cf9c3e541c4abe23d59e0aa7f2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.671650 1170 ts_tablet_manager.cc:1950] T 2fac7b819e2642378e0d3f9230d0259c P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:28.671730 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88bfc23acd274041a9457079c1709d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.672396 1247 raft_consensus.cc:493] T d0fd00baa02d48278afaa5b8c19182f0 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:28.672562 1247 raft_consensus.cc:515] T d0fd00baa02d48278afaa5b8c19182f0 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.673236 1247 leader_election.cc:290] T d0fd00baa02d48278afaa5b8c19182f0 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:28.682350 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 943ec703e65f4fd3b0c690e08c0bd226. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.683420 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa9d9ab8073945d28d52fb02213f201a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.683739 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9f7cd765d614d59a7c190b744b0844d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.685534 752 tablet_service.cc:1505] Processing CreateTablet for tablet e54fe7cbcffa40c4b04144b47bee2b48 (DEFAULT_TABLE table=default.Table_590cc281d8b042b19ef12c7f6436752a [id=b1eee5c0243a48c4bbdecc0b4f38ab2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.686093 1374 tablet.cc:1722] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251025 07:52:28.686082 745 tablet_service.cc:1505] Processing CreateTablet for tablet b947ecc352e54f4caa73c6ca97372256 (DEFAULT_TABLE table=default.Table_503b86ee2ab545afa9829bebcc9ea11d [id=395c91146476423dad0455a12eb5a8af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.686408 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b94634ac5077439fadfbbce569f60e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.686591 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 50433881c0eb4aa5a389b6f5b4a4337a with delete_type TABLET_DATA_DELETED (replaced by tablet 50ae9bca6bed4f749d54191ac1543410) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:28.687698 1397 tablet.cc:1722] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P db42be7ad3c741a0aa427f27e245993b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251025 07:52:28.688516 750 tablet_service.cc:1505] Processing CreateTablet for tablet ebf3a2a926754305b5bd95fcebb25d29 (DEFAULT_TABLE table=default.Table_ac4beca0c3bd4cf29565faf5323d9ff7 [id=857bcd1dbe2e477bbc0d77f79a58f7f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.688545 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 7fbae8be73bf4e18b5a5c4ec4cdebc12 with delete_type TABLET_DATA_DELETED (replaced by tablet 4c155aa7fa5b4d68bbab1918b8432372) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:28.689612 746 tablet_service.cc:1505] Processing CreateTablet for tablet f43ef51ee45d47a99ec732b49d50d3b4 (DEFAULT_TABLE table=default.Table_0b0bd6b71b92454fb8cdf967ffd8349e [id=e9b21659de7d4276aa1f9effb7076eb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.689913 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 3abfd52327a04ba88625f391a8658fbb with delete_type TABLET_DATA_DELETED (replaced by tablet 04d27d54fdb7483b869408af65ff85c9) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:28.690116 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 87a93778620f4c84b3b3f7a3d7d7e87f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:28.690526 740 tablet_service.cc:1505] Processing CreateTablet for tablet f4578ce69a0d4322944e5ce7c8a59831 (DEFAULT_TABLE table=default.Table_3bd421885a1b4e06b11d12780c34cad4 [id=4d9e99d6474b4d00bc66e26a33694e69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.690703 1170 tablet_replica.cc:333] T 94408a3a90224645ba3cacccf5de33f1 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:28.690925 1170 raft_consensus.cc:2243] T 94408a3a90224645ba3cacccf5de33f1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:28.691098 1170 raft_consensus.cc:2272] T 94408a3a90224645ba3cacccf5de33f1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:28.691851 747 tablet_service.cc:1505] Processing CreateTablet for tablet 34315f64b89148efbf1f20a43d5c5efb (DEFAULT_TABLE table=default.Table_4b3d7b3a731942189cdc214938a45820 [id=88f1e8b1c5fa4913a5f7b5b2c6cbf6ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.692584 744 tablet_service.cc:1505] Processing CreateTablet for tablet e9f7cd765d614d59a7c190b744b0844d (DEFAULT_TABLE table=default.Table_5ecc1936fc8447669d0602cfd6133c6b [id=551028aad5c246378c7acb8e443853a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.693600 1167 tablet_bootstrap.cc:492] T 9bf486d82b984d2c89ee32e84a85894f P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:28.693758 1167 ts_tablet_manager.cc:1403] T 9bf486d82b984d2c89ee32e84a85894f P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.191s user 0.030s sys 0.046s
I20251025 07:52:28.694046 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 2fac7b819e2642378e0d3f9230d0259c (table default.Table_dda83c899ebe4be999ac2ab537d3d323 [id=6605e11e11804d408a6a4a9d403dde95]) successfully deleted
I20251025 07:52:28.694209 1170 ts_tablet_manager.cc:1916] T 94408a3a90224645ba3cacccf5de33f1 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:28.694293 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 3abfd52327a04ba88625f391a8658fbb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3abfd52327a04ba88625f391a8658fbb already in progress: creating tablet
I20251025 07:52:28.694772 1167 raft_consensus.cc:359] T 9bf486d82b984d2c89ee32e84a85894f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.695020 1167 raft_consensus.cc:385] T 9bf486d82b984d2c89ee32e84a85894f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251025 07:52:28.695075 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 7fbae8be73bf4e18b5a5c4ec4cdebc12 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7fbae8be73bf4e18b5a5c4ec4cdebc12 already in progress: creating tablet
I20251025 07:52:28.695127 1167 raft_consensus.cc:740] T 9bf486d82b984d2c89ee32e84a85894f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
W20251025 07:52:28.695307 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 87a93778620f4c84b3b3f7a3d7d7e87f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87a93778620f4c84b3b3f7a3d7d7e87f already in progress: creating tablet
I20251025 07:52:28.695389 1167 consensus_queue.cc:260] T 9bf486d82b984d2c89ee32e84a85894f P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:28.695503 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet e54fe7cbcffa40c4b04144b47bee2b48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e54fe7cbcffa40c4b04144b47bee2b48 already in progress: creating tablet
W20251025 07:52:28.695690 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 27651312cc2e42a8bfcd245f5d301679 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27651312cc2e42a8bfcd245f5d301679 already in progress: creating tablet
W20251025 07:52:28.695901 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 742b3ec3f6604647b9e397532f1673dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 742b3ec3f6604647b9e397532f1673dd already in progress: creating tablet
W20251025 07:52:28.696110 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 887e7c7a5d0e4aaf88f26246306df19c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 887e7c7a5d0e4aaf88f26246306df19c already in progress: creating tablet
W20251025 07:52:28.697997 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1912e59548874e0e8e989050ec0064c5 has no consensus state
I20251025 07:52:28.698405 1167 ts_tablet_manager.cc:1434] T 9bf486d82b984d2c89ee32e84a85894f P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251025 07:52:28.698696 525 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 50ffbcdd4a9247bf99b8e2361bc02ef5 (table default.Table_9f0ae2442a334cf8ab13f2eba387ebf3 [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 3 got 0
I20251025 07:52:28.699673 1167 tablet_bootstrap.cc:492] T d0fd00baa02d48278afaa5b8c19182f0 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:28.700855 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 34315f64b89148efbf1f20a43d5c5efb (DEFAULT_TABLE table=default.Table_4b3d7b3a731942189cdc214938a45820 [id=88f1e8b1c5fa4913a5f7b5b2c6cbf6ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.701260 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34315f64b89148efbf1f20a43d5c5efb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.701280 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 27651312cc2e42a8bfcd245f5d301679 with delete_type TABLET_DATA_DELETED (replaced by tablet a4e4b99be0434baaa75d988281797846) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:28.701475 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 94408a3a90224645ba3cacccf5de33f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:28.701881 754 tablet_service.cc:1552] Processing DeleteTablet for tablet e54fe7cbcffa40c4b04144b47bee2b48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:28.702363 756 tablet_service.cc:1505] Processing CreateTablet for tablet b94634ac5077439fadfbbce569f60e8b (DEFAULT_TABLE table=default.Table_1ff54f4df7a2444489ae4e4060084156 [id=b88c37a678344c6b8e8017553e486cb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.702704 1167 tablet_bootstrap.cc:654] T d0fd00baa02d48278afaa5b8c19182f0 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:28.703088 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c3eb43209f24592820638fc25486b78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:28.703492 742 tablet_service.cc:1505] Processing CreateTablet for tablet 80fe3255ed614270bd8352a2853fbc43 (DEFAULT_TABLE table=default.Table_d2a7e16c6f0d4b2886c16bfe6b685c00 [id=64c18a10455c4694babcc5ec0f463b85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.703850 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 83084be89b2646b0bda208b3f6f60d77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:28.704792 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 2947502815d742e89cacb474b7fbb249 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:28.706564 1170 ts_tablet_manager.cc:1929] T 94408a3a90224645ba3cacccf5de33f1 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:28.706739 1170 log.cc:1199] T 94408a3a90224645ba3cacccf5de33f1 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/94408a3a90224645ba3cacccf5de33f1
I20251025 07:52:28.707105 1170 ts_tablet_manager.cc:1950] T 94408a3a90224645ba3cacccf5de33f1 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:28.708141 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 94408a3a90224645ba3cacccf5de33f1 (table default.Table_819aedd5270f4d6cbaaaee7c2ff5f676 [id=1e4728f474814b36992fd99de8451b95]) successfully deleted
I20251025 07:52:28.709241 1170 tablet_replica.cc:333] T 33c608beb4ef4a859a135b11e07c81a4 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:28.709432 1170 raft_consensus.cc:2243] T 33c608beb4ef4a859a135b11e07c81a4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:28.709605 1170 raft_consensus.cc:2272] T 33c608beb4ef4a859a135b11e07c81a4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:28.710556 1170 ts_tablet_manager.cc:1916] T 33c608beb4ef4a859a135b11e07c81a4 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:28.710642 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 742b3ec3f6604647b9e397532f1673dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:28.711135 1167 tablet_bootstrap.cc:492] T d0fd00baa02d48278afaa5b8c19182f0 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:28.711297 1167 ts_tablet_manager.cc:1403] T d0fd00baa02d48278afaa5b8c19182f0 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20251025 07:52:28.712199 1167 raft_consensus.cc:359] T d0fd00baa02d48278afaa5b8c19182f0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.712426 1167 raft_consensus.cc:385] T d0fd00baa02d48278afaa5b8c19182f0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:28.712517 1167 raft_consensus.cc:740] T d0fd00baa02d48278afaa5b8c19182f0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:28.714279 751 tablet_service.cc:1505] Processing CreateTablet for tablet 7c7a9d7a4fb3449bb57eafa83230a839 (DEFAULT_TABLE table=default.Table_192549a3e67840f3aefc31029cf2ff52 [id=a37be02f8e994f82baf2f4f5ec800926]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.715018 754 tablet_service.cc:1505] Processing CreateTablet for tablet f97acc9a638b4d5a968dcc3412a384bc (DEFAULT_TABLE table=default.Table_04c3923fa1624c52b34d968919e406ae [id=0827603890c8466082bc6c306c9c66af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.712785 1167 consensus_queue.cc:260] T d0fd00baa02d48278afaa5b8c19182f0 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.716073 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0fd00baa02d48278afaa5b8c19182f0" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:28.716140 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 887e7c7a5d0e4aaf88f26246306df19c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:28.716207 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "796413da731245459cc5dffacc3deb2f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:28.716297 776 raft_consensus.cc:2468] T d0fd00baa02d48278afaa5b8c19182f0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:28.716385 775 raft_consensus.cc:2468] T 796413da731245459cc5dffacc3deb2f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:28.716459 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87a93778620f4c84b3b3f7a3d7d7e87f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:28.717132 982 leader_election.cc:304] T d0fd00baa02d48278afaa5b8c19182f0 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:28.717447 1247 raft_consensus.cc:2804] T d0fd00baa02d48278afaa5b8c19182f0 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:28.717555 1247 raft_consensus.cc:493] T d0fd00baa02d48278afaa5b8c19182f0 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:28.717646 1247 raft_consensus.cc:3060] T d0fd00baa02d48278afaa5b8c19182f0 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:28.718961 1167 ts_tablet_manager.cc:1434] T d0fd00baa02d48278afaa5b8c19182f0 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251025 07:52:28.720830 1247 raft_consensus.cc:515] T d0fd00baa02d48278afaa5b8c19182f0 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.720960 1170 ts_tablet_manager.cc:1929] T 33c608beb4ef4a859a135b11e07c81a4 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:28.721113 1170 log.cc:1199] T 33c608beb4ef4a859a135b11e07c81a4 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/33c608beb4ef4a859a135b11e07c81a4
I20251025 07:52:28.721515 1170 ts_tablet_manager.cc:1950] T 33c608beb4ef4a859a135b11e07c81a4 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:28.721741 1247 leader_election.cc:290] T d0fd00baa02d48278afaa5b8c19182f0 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:28.722473 1167 tablet_bootstrap.cc:492] T b00153c97104470883f6089d71d23708 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:28.722683 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 33c608beb4ef4a859a135b11e07c81a4 (table default.Table_7a19acf479d64317bd910c04b08a8b4f [id=8bbb3b9204634ee6a04f5a4753739476]) successfully deleted
I20251025 07:52:28.723208 1170 tablet_replica.cc:333] T d0fd00baa02d48278afaa5b8c19182f0 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:28.723412 1170 raft_consensus.cc:2243] T d0fd00baa02d48278afaa5b8c19182f0 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:28.723590 1170 raft_consensus.cc:2272] T d0fd00baa02d48278afaa5b8c19182f0 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:28.724916 1170 ts_tablet_manager.cc:1916] T d0fd00baa02d48278afaa5b8c19182f0 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:28.726712 1167 tablet_bootstrap.cc:654] T b00153c97104470883f6089d71d23708 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:28.727115 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 80ee75935d7e461d9d4066fac9819bef consensus state has no leader
I20251025 07:52:28.729475 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 9bf486d82b984d2c89ee32e84a85894f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:28.729554 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0fd00baa02d48278afaa5b8c19182f0" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:28.729794 774 raft_consensus.cc:3060] T d0fd00baa02d48278afaa5b8c19182f0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:28.735445 886 tablet_service.cc:1505] Processing CreateTablet for tablet 80fe3255ed614270bd8352a2853fbc43 (DEFAULT_TABLE table=default.Table_d2a7e16c6f0d4b2886c16bfe6b685c00 [id=64c18a10455c4694babcc5ec0f463b85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.736670 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_db98182d07a54725823623925f51a0cb" } modify_external_catalogs: true
I20251025 07:52:28.736945 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_db98182d07a54725823623925f51a0cb" } modify_external_catalogs: true
I20251025 07:52:28.739753 748 tablet_service.cc:1552] Processing DeleteTablet for tablet d0fd00baa02d48278afaa5b8c19182f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:28.740839 774 raft_consensus.cc:2468] T d0fd00baa02d48278afaa5b8c19182f0 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:28.741626 525 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: default.Table_9f0ae2442a334cf8ab13f2eba387ebf3 [id=fa9b211cb1cb437992aef4ed8f2e7439] alter complete (version 3)
I20251025 07:52:28.743167 1170 ts_tablet_manager.cc:1929] T d0fd00baa02d48278afaa5b8c19182f0 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:28.743418 1170 log.cc:1199] T d0fd00baa02d48278afaa5b8c19182f0 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/d0fd00baa02d48278afaa5b8c19182f0
I20251025 07:52:28.743839 1170 ts_tablet_manager.cc:1950] T d0fd00baa02d48278afaa5b8c19182f0 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:28.744627 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet d0fd00baa02d48278afaa5b8c19182f0 (table default.Table_1d4029fc57dd4ecb84b8621c33d27958 [id=ff352eacb0614dbbbb6ab6369b9192e8]) successfully deleted
I20251025 07:52:28.745772 982 leader_election.cc:304] T d0fd00baa02d48278afaa5b8c19182f0 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
W20251025 07:52:28.746028 982 raft_consensus.cc:2686] T d0fd00baa02d48278afaa5b8c19182f0 P db42be7ad3c741a0aa427f27e245993b: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251025 07:52:28.749400 882 tablet_service.cc:1552] Processing DeleteTablet for tablet 27651312cc2e42a8bfcd245f5d301679 with delete_type TABLET_DATA_DELETED (replaced by tablet a4e4b99be0434baaa75d988281797846) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:28.749868 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80fe3255ed614270bd8352a2853fbc43. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:28.754006 983 leader_election.cc:343] T d0fd00baa02d48278afaa5b8c19182f0 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: d0fd00baa02d48278afaa5b8c19182f0
W20251025 07:52:28.754436 983 leader_election.cc:343] T d0fd00baa02d48278afaa5b8c19182f0 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: d0fd00baa02d48278afaa5b8c19182f0
I20251025 07:52:28.755230 749 tablet_service.cc:1505] Processing CreateTablet for tablet 88eb51447e0f4080baa49dd902deb24f (DEFAULT_TABLE table=default.Table_4a7945d6aa00429b98f77ca4b9f3e3bc [id=438936db6ab24eb09a48b31ca013031b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.755266 748 tablet_service.cc:1505] Processing CreateTablet for tablet 0325d462441349ef81841f1b0cab2a20 (DEFAULT_TABLE table=default.Table_af3343b5c3c244deb6f22b9fba10c2b9 [id=9ce3ba77c56a4ef09cb3b40ea0e488f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.756121 738 tablet_service.cc:1505] Processing CreateTablet for tablet b3a0af893a8947848e12093d0a7f0229 (DEFAULT_TABLE table=default.Table_0f2c837d88c043fda9d4eb94706a5398 [id=c837adb3bd5a4436a4cd8ea6f7be6b9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.757189 737 tablet_service.cc:1505] Processing CreateTablet for tablet 724d0b7e1b65445a9a1f056852983bdb (DEFAULT_TABLE table=default.Table_6fd078e1af7a41f8a24caebf5e12e2b2 [id=14ce70a07f554566bf85c2e37c80fa65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.760083 1019 tablet_service.cc:1505] Processing CreateTablet for tablet 88eb51447e0f4080baa49dd902deb24f (DEFAULT_TABLE table=default.Table_4a7945d6aa00429b98f77ca4b9f3e3bc [id=438936db6ab24eb09a48b31ca013031b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.760378 1025 tablet_service.cc:1505] Processing CreateTablet for tablet c2713942584e4734acbc0775ead772fa (DEFAULT_TABLE table=default.Table_6140b1f390084256848ae1b746059268 [id=adaaf2bcd09f4c779aee11bb7cd4ba15]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.760515 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88eb51447e0f4080baa49dd902deb24f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.760775 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2713942584e4734acbc0775ead772fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.761121 1020 tablet_service.cc:1505] Processing CreateTablet for tablet 724d0b7e1b65445a9a1f056852983bdb (DEFAULT_TABLE table=default.Table_6fd078e1af7a41f8a24caebf5e12e2b2 [id=14ce70a07f554566bf85c2e37c80fa65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.761435 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 724d0b7e1b65445a9a1f056852983bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.762936 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 36262ab8c6964dd684c01b4db6219a4f (DEFAULT_TABLE table=default.Table_4f9a077b76bf42979239aeb6710131ec [id=f249f30497bc450c887d659721469de8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.763337 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36262ab8c6964dd684c01b4db6219a4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.766441 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 0325d462441349ef81841f1b0cab2a20 (DEFAULT_TABLE table=default.Table_af3343b5c3c244deb6f22b9fba10c2b9 [id=9ce3ba77c56a4ef09cb3b40ea0e488f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.766620 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_6b02014a8bfe42b78c2856f646f0c75b" } new_table_name: "default.Table_e8be4237bba844c8963e8955729b4fba" modify_external_catalogs: true
I20251025 07:52:28.766824 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0325d462441349ef81841f1b0cab2a20. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.770448 1023 tablet_service.cc:1505] Processing CreateTablet for tablet b3a0af893a8947848e12093d0a7f0229 (DEFAULT_TABLE table=default.Table_0f2c837d88c043fda9d4eb94706a5398 [id=c837adb3bd5a4436a4cd8ea6f7be6b9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.770809 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3a0af893a8947848e12093d0a7f0229. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.771505 1178 raft_consensus.cc:493] T 50433881c0eb4aa5a389b6f5b4a4337a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:28.771689 1178 raft_consensus.cc:515] T 50433881c0eb4aa5a389b6f5b4a4337a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:28.772385 1178 leader_election.cc:290] T 50433881c0eb4aa5a389b6f5b4a4337a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
W20251025 07:52:28.773525 712 leader_election.cc:343] T 50433881c0eb4aa5a389b6f5b4a4337a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 50433881c0eb4aa5a389b6f5b4a4337a
I20251025 07:52:28.774271 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 887e7c7a5d0e4aaf88f26246306df19c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:28.776479 1178 raft_consensus.cc:493] T 83084be89b2646b0bda208b3f6f60d77 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:28.776647 1178 raft_consensus.cc:515] T 83084be89b2646b0bda208b3f6f60d77 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.777273 1178 leader_election.cc:290] T 83084be89b2646b0bda208b3f6f60d77 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:28.778470 712 leader_election.cc:343] T 83084be89b2646b0bda208b3f6f60d77 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 83084be89b2646b0bda208b3f6f60d77
I20251025 07:52:28.781397 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 943ec703e65f4fd3b0c690e08c0bd226
I20251025 07:52:28.782203 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_8028c8ff15754b788ce12115fde3be3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:28.782581 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_19a3326e4da345c1bbeb2cf083d28e24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:28.787808 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8028c8ff15754b788ce12115fde3be3d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:28.787883 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19a3326e4da345c1bbeb2cf083d28e24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:28.783087 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_bfcdc02fe0dc48a0ae793ecd15f50a70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:28.789309 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bfcdc02fe0dc48a0ae793ecd15f50a70 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:28.783439 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_4bb5ba285e1c46f8b698e4915e37fab9" } new_table_name: "default.Table_d0a06b2f0df7446ead73934e6f603423" modify_external_catalogs: true
I20251025 07:52:28.783886 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_79969e4ace6f4227b2748b59136027d6" } modify_external_catalogs: true
I20251025 07:52:28.790695 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_79969e4ace6f4227b2748b59136027d6" } modify_external_catalogs: true
I20251025 07:52:28.785837 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_380ca630f4144f40a04084d2ef8600e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:28.792176 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_380ca630f4144f40a04084d2ef8600e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:28.796307 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 04449cc97f404a19b6a1b20dea074f95 (DEFAULT_TABLE table=default.Table_f5af41a9cdee45ffa4a44089ba80464b [id=e041946c093b4c789ce175c736cc230b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.796773 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04449cc97f404a19b6a1b20dea074f95. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.798210 882 tablet_service.cc:1552] Processing DeleteTablet for tablet 742b3ec3f6604647b9e397532f1673dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:28.799571 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_4bc4254fb03d4db99f40d7b07b2f89aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:28.799925 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 5caac9c943bf42a5b2ae522d87fdfe8e with delete_type TABLET_DATA_DELETED (replaced by tablet 4f25c9caa05b4da4bd56a8a21fde423c) from {username='slave'} at 127.0.0.1:56590
W20251025 07:52:28.800107 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4bc4254fb03d4db99f40d7b07b2f89aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:28.800105 893 tablet_service.cc:1552] Processing DeleteTablet for tablet dd898450f2604987b08639cf87d0e402 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:28.800307 882 tablet_service.cc:1552] Processing DeleteTablet for tablet a231f3d31b204f7b82f018892ccc8435 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:28.800611 1320 leader_election.cc:290] T 796413da731245459cc5dffacc3deb2f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:28.800876 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 1483c40eee834f20a92e6b4cd89775b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:28.802619 1023 tablet_service.cc:1505] Processing CreateTablet for tablet 292fe04055954afd8c08ac2636b68191 (DEFAULT_TABLE table=default.Table_6588890a31184cc2ab6ccb9b72e60429 [id=a253ce0253bd4da4814741c482193d8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.803048 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 292fe04055954afd8c08ac2636b68191. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:28.804340 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4b28215f61574cf48c447dd01955ac77 consensus state has no leader
I20251025 07:52:28.805917 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 8eaf6c97ab9b41e9ae746a9c5abe1735 (DEFAULT_TABLE table=default.Table_bf5cb8a733934f4ca29ecc9d473e4f2c [id=4bb4741007834a24aca34e608365ea03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.806329 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8eaf6c97ab9b41e9ae746a9c5abe1735. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.810741 881 tablet_service.cc:1505] Processing CreateTablet for tablet 7c7a9d7a4fb3449bb57eafa83230a839 (DEFAULT_TABLE table=default.Table_192549a3e67840f3aefc31029cf2ff52 [id=a37be02f8e994f82baf2f4f5ec800926]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.810904 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet 943ec703e65f4fd3b0c690e08c0bd226 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:28.811362 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c7a9d7a4fb3449bb57eafa83230a839. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.813174 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e54fe7cbcffa40c4b04144b47bee2b48. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.813733 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b947ecc352e54f4caa73c6ca97372256. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.814180 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0325d462441349ef81841f1b0cab2a20. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.814586 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 724d0b7e1b65445a9a1f056852983bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.814599 893 tablet_service.cc:1505] Processing CreateTablet for tablet 0325d462441349ef81841f1b0cab2a20 (DEFAULT_TABLE table=default.Table_af3343b5c3c244deb6f22b9fba10c2b9 [id=9ce3ba77c56a4ef09cb3b40ea0e488f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.815053 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9f7cd765d614d59a7c190b744b0844d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.815469 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4578ce69a0d4322944e5ce7c8a59831. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.815474 888 tablet_service.cc:1505] Processing CreateTablet for tablet f97acc9a638b4d5a968dcc3412a384bc (DEFAULT_TABLE table=default.Table_04c3923fa1624c52b34d968919e406ae [id=0827603890c8466082bc6c306c9c66af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.815896 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f43ef51ee45d47a99ec732b49d50d3b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.816313 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebf3a2a926754305b5bd95fcebb25d29. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.816273 882 tablet_service.cc:1505] Processing CreateTablet for tablet c2713942584e4734acbc0775ead772fa (DEFAULT_TABLE table=default.Table_6140b1f390084256848ae1b746059268 [id=adaaf2bcd09f4c779aee11bb7cd4ba15]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.816728 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34315f64b89148efbf1f20a43d5c5efb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.817174 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88eb51447e0f4080baa49dd902deb24f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.817631 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3a0af893a8947848e12093d0a7f0229. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.818092 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80fe3255ed614270bd8352a2853fbc43. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.818189 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f97acc9a638b4d5a968dcc3412a384bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.818504 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b94634ac5077439fadfbbce569f60e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.818894 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f97acc9a638b4d5a968dcc3412a384bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.819547 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c7a9d7a4fb3449bb57eafa83230a839. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:28.821895 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 943ec703e65f4fd3b0c690e08c0bd226 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 943ec703e65f4fd3b0c690e08c0bd226 already in progress: creating tablet
I20251025 07:52:28.823174 753 tablet_service.cc:1505] Processing CreateTablet for tablet 36262ab8c6964dd684c01b4db6219a4f (DEFAULT_TABLE table=default.Table_4f9a077b76bf42979239aeb6710131ec [id=f249f30497bc450c887d659721469de8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.823544 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36262ab8c6964dd684c01b4db6219a4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.824488 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2713942584e4734acbc0775ead772fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.826920 880 tablet_service.cc:1505] Processing CreateTablet for tablet b3a0af893a8947848e12093d0a7f0229 (DEFAULT_TABLE table=default.Table_0f2c837d88c043fda9d4eb94706a5398 [id=c837adb3bd5a4436a4cd8ea6f7be6b9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.827586 887 tablet_service.cc:1505] Processing CreateTablet for tablet 88eb51447e0f4080baa49dd902deb24f (DEFAULT_TABLE table=default.Table_4a7945d6aa00429b98f77ca4b9f3e3bc [id=438936db6ab24eb09a48b31ca013031b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.828461 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0325d462441349ef81841f1b0cab2a20. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.829918 753 tablet_service.cc:1505] Processing CreateTablet for tablet 292fe04055954afd8c08ac2636b68191 (DEFAULT_TABLE table=default.Table_6588890a31184cc2ab6ccb9b72e60429 [id=a253ce0253bd4da4814741c482193d8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.830168 879 tablet_service.cc:1505] Processing CreateTablet for tablet 724d0b7e1b65445a9a1f056852983bdb (DEFAULT_TABLE table=default.Table_6fd078e1af7a41f8a24caebf5e12e2b2 [id=14ce70a07f554566bf85c2e37c80fa65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.830324 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 292fe04055954afd8c08ac2636b68191. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.830576 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88eb51447e0f4080baa49dd902deb24f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.838294 1019 tablet_service.cc:1505] Processing CreateTablet for tablet d9bfd7c0c3ad4e5b98b6708793d61d41 (DEFAULT_TABLE table=default.Table_580b09b75790418e8dcf68fc88bbbf47 [id=5202bb531f384527a167bf9efe00d9c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.839828 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9bfd7c0c3ad4e5b98b6708793d61d41. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.843302 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_bb258691f8844d85ab37eb73e9a4c314" } modify_external_catalogs: true
I20251025 07:52:28.843518 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_bb258691f8844d85ab37eb73e9a4c314" } modify_external_catalogs: true
I20251025 07:52:28.844558 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 1912e59548874e0e8e989050ec0064c5 (DEFAULT_TABLE table=default.Table_22ddca5b9e8449ef9fbdeb29212ff9a5 [id=74ec1fa4786e40109252885ea225ed66]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.845099 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1912e59548874e0e8e989050ec0064c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.845242 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_2cd70245aa3d4809a0647e7fa499f62f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:28.845780 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2cd70245aa3d4809a0647e7fa499f62f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:28.848495 1026 tablet_service.cc:1505] Processing CreateTablet for tablet a4e4b99be0434baaa75d988281797846 (DEFAULT_TABLE table=default.Table_dac528b081844b7c86027169c6234c27 [id=b0caa78ee8c84aa8bea4818cf9feb70c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.848930 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4e4b99be0434baaa75d988281797846. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.849226 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_b8a12e9a7a974006904fcef12bb8225a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:28.849676 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8a12e9a7a974006904fcef12bb8225a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:28.870222 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 4c155aa7fa5b4d68bbab1918b8432372 (DEFAULT_TABLE table=default.Table_7f531c5ddd174b49b0117a226cd0d91f [id=28bbaad474164a8c8c97a76bb8081791]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.870711 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c155aa7fa5b4d68bbab1918b8432372. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.874641 1023 tablet_service.cc:1505] Processing CreateTablet for tablet aaef24ccdfba45fe8204327723df834e (DEFAULT_TABLE table=default.Table_bd5089a7892041649dd05b7fb0961255 [id=840b3ffcb44144548843457c48f46380]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.875029 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aaef24ccdfba45fe8204327723df834e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.878614 1020 tablet_service.cc:1505] Processing CreateTablet for tablet 0f5f6ca6c3cd4d369690e425387a4344 (DEFAULT_TABLE table=default.Table_e317f8d728da4c76886f67d0995ab7f4 [id=4d2121dea4904850aecb65da5dd2d8f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.879128 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f5f6ca6c3cd4d369690e425387a4344. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.880123 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 04d27d54fdb7483b869408af65ff85c9 (DEFAULT_TABLE table=default.Table_cfa54e1a04154e2eb8747fd3a4812883 [id=bf3a3f8dae184d999f2159dfcc7803b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.880576 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04d27d54fdb7483b869408af65ff85c9. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:28.881071 513 catalog_manager.cc:5058] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): alter failed for tablet 299572b2dbe244918dcebc5ea5f3c45a (table default.Table_bd5089a7892041649dd05b7fb0961255 [id=840b3ffcb44144548843457c48f46380]),no further retry: Not found: Tablet not found: 299572b2dbe244918dcebc5ea5f3c45a
I20251025 07:52:28.882381 1171 ts_tablet_manager.cc:1929] T 2fac7b819e2642378e0d3f9230d0259c P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 07:52:28.882650 1171 log.cc:1199] T 2fac7b819e2642378e0d3f9230d0259c P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/2fac7b819e2642378e0d3f9230d0259c
I20251025 07:52:28.883091 1171 ts_tablet_manager.cc:1950] T 2fac7b819e2642378e0d3f9230d0259c P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:28.886545 755 tablet_service.cc:1505] Processing CreateTablet for tablet c2713942584e4734acbc0775ead772fa (DEFAULT_TABLE table=default.Table_6140b1f390084256848ae1b746059268 [id=adaaf2bcd09f4c779aee11bb7cd4ba15]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.887547 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2713942584e4734acbc0775ead772fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.890266 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 943ec703e65f4fd3b0c690e08c0bd226 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:28.890441 877 tablet_service.cc:1552] Processing DeleteTablet for tablet 943ec703e65f4fd3b0c690e08c0bd226 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:28.890802 752 tablet_service.cc:1505] Processing CreateTablet for tablet d9bfd7c0c3ad4e5b98b6708793d61d41 (DEFAULT_TABLE table=default.Table_580b09b75790418e8dcf68fc88bbbf47 [id=5202bb531f384527a167bf9efe00d9c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.891216 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9bfd7c0c3ad4e5b98b6708793d61d41. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.895078 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 4b28215f61574cf48c447dd01955ac77 (DEFAULT_TABLE table=default.Table_e8be4237bba844c8963e8955729b4fba [id=51b12903e6024e1a9adc2f9fa7400937]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.895498 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b28215f61574cf48c447dd01955ac77. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.895783 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3a0af893a8947848e12093d0a7f0229. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.897326 747 tablet_service.cc:1505] Processing CreateTablet for tablet 04449cc97f404a19b6a1b20dea074f95 (DEFAULT_TABLE table=default.Table_f5af41a9cdee45ffa4a44089ba80464b [id=e041946c093b4c789ce175c736cc230b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.897670 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04449cc97f404a19b6a1b20dea074f95. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.899291 1019 tablet_service.cc:1505] Processing CreateTablet for tablet 26c849d376d143c6bef0c8e0e1d7d03a (DEFAULT_TABLE table=default.Table_f8c38604344c429295d6d0ab9cb85b13 [id=c2f7a6281b6f47c990b096412f2201a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.899663 1218 raft_consensus.cc:493] T 1b31177a062b467187d68ed2034f5690 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:28.899952 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26c849d376d143c6bef0c8e0e1d7d03a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.905208 1018 tablet_service.cc:1505] Processing CreateTablet for tablet cb829649f73d4d3f8862d268c953bd86 (DEFAULT_TABLE table=default.Table_4b70d9acf0f64bcd992a894b129927c3 [id=7944225d3ce7439b881bfb9289d9f040]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.905618 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb829649f73d4d3f8862d268c953bd86. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.908488 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_04c3923fa1624c52b34d968919e406ae" } new_table_name: "default.Table_e50bf47b78d94bbc811e6f3fb3551edb" modify_external_catalogs: true
I20251025 07:52:28.913136 1169 tablet_bootstrap.cc:492] T 2947502815d742e89cacb474b7fbb249 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:28.913286 1169 ts_tablet_manager.cc:1403] T 2947502815d742e89cacb474b7fbb249 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.440s user 0.002s sys 0.003s
I20251025 07:52:28.914075 1169 raft_consensus.cc:359] T 2947502815d742e89cacb474b7fbb249 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.914291 1169 raft_consensus.cc:385] T 2947502815d742e89cacb474b7fbb249 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:28.914376 1169 raft_consensus.cc:740] T 2947502815d742e89cacb474b7fbb249 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:28.914590 1169 consensus_queue.cc:260] T 2947502815d742e89cacb474b7fbb249 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.915107 1169 ts_tablet_manager.cc:1434] T 2947502815d742e89cacb474b7fbb249 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:28.915421 1169 tablet_bootstrap.cc:492] T 9bf486d82b984d2c89ee32e84a85894f P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:28.917131 1169 tablet_bootstrap.cc:654] T 9bf486d82b984d2c89ee32e84a85894f P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:28.919047 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_bd5089a7892041649dd05b7fb0961255" } modify_external_catalogs: true
I20251025 07:52:28.919253 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_bd5089a7892041649dd05b7fb0961255" } modify_external_catalogs: true
I20251025 07:52:28.921836 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 1b31177a062b467187d68ed2034f5690 with delete_type TABLET_DATA_DELETED (replaced by tablet b772025852794a49b072f37cccbd3291) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:28.922050 1166 tablet_bootstrap.cc:492] T f51ec427dd9349a48eb221db04cf242b P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:28.924811 1320 raft_consensus.cc:493] T 1483c40eee834f20a92e6b4cd89775b6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:28.927510 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_0f2c837d88c043fda9d4eb94706a5398" } modify_external_catalogs: true
I20251025 07:52:28.927747 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_0f2c837d88c043fda9d4eb94706a5398" } modify_external_catalogs: true
I20251025 07:52:28.928279 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_5f916bc95ede499b9f4de53471ae1314" } new_table_name: "default.Table_c902adc24467448f9b5d4f585cf614f5" modify_external_catalogs: true
I20251025 07:52:28.929751 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 2947502815d742e89cacb474b7fbb249 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:28.930037 1170 tablet_replica.cc:333] T 2947502815d742e89cacb474b7fbb249 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:28.930249 1170 raft_consensus.cc:2243] T 2947502815d742e89cacb474b7fbb249 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:28.930204 1218 raft_consensus.cc:515] T 1b31177a062b467187d68ed2034f5690 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.930434 1170 raft_consensus.cc:2272] T 2947502815d742e89cacb474b7fbb249 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:28.931034 1218 leader_election.cc:290] T 1b31177a062b467187d68ed2034f5690 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:28.931437 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 80ee75935d7e461d9d4066fac9819bef
I20251025 07:52:28.931509 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_eafaca066a2548a686050fed8c380553"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:28.932024 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eafaca066a2548a686050fed8c380553 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:28.932561 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 80ee75935d7e461d9d4066fac9819bef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:28.932858 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b31177a062b467187d68ed2034f5690" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:28.934209 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_3292d47465544af0970067fb567e7ede"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:28.934883 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3292d47465544af0970067fb567e7ede in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:28.935221 1166 ts_tablet_manager.cc:1403] T f51ec427dd9349a48eb221db04cf242b P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.435s user 0.005s sys 0.000s
I20251025 07:52:28.935346 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 71432a6848944a0da06ef9b81f160dc4
W20251025 07:52:28.936244 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f97acc9a638b4d5a968dcc3412a384bc consensus state has no leader
I20251025 07:52:28.936800 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 71432a6848944a0da06ef9b81f160dc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:47408
W20251025 07:52:28.937047 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5361f08f50b745eb98bf02e979d7c71e consensus state has no leader
I20251025 07:52:28.939349 1170 ts_tablet_manager.cc:1916] T 2947502815d742e89cacb474b7fbb249 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:28.939543 850 leader_election.cc:343] T 1b31177a062b467187d68ed2034f5690 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:28.946184 1169 tablet_bootstrap.cc:492] T 9bf486d82b984d2c89ee32e84a85894f P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:28.946450 1169 ts_tablet_manager.cc:1403] T 9bf486d82b984d2c89ee32e84a85894f P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.001s
I20251025 07:52:28.947340 1169 raft_consensus.cc:359] T 9bf486d82b984d2c89ee32e84a85894f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.947712 1169 raft_consensus.cc:385] T 9bf486d82b984d2c89ee32e84a85894f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:28.947860 1169 raft_consensus.cc:740] T 9bf486d82b984d2c89ee32e84a85894f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:28.948179 1169 consensus_queue.cc:260] T 9bf486d82b984d2c89ee32e84a85894f P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.949682 1320 raft_consensus.cc:515] T 1483c40eee834f20a92e6b4cd89775b6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:28.951004 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1483c40eee834f20a92e6b4cd89775b6" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:28.953658 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 9bf486d82b984d2c89ee32e84a85894f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:28.954926 1169 ts_tablet_manager.cc:1434] T 9bf486d82b984d2c89ee32e84a85894f P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20251025 07:52:28.956676 1172 ts_tablet_manager.cc:1929] T 2fac7b819e2642378e0d3f9230d0259c P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 07:52:28.956835 1172 log.cc:1199] T 2fac7b819e2642378e0d3f9230d0259c P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/2fac7b819e2642378e0d3f9230d0259c
I20251025 07:52:28.957041 1170 ts_tablet_manager.cc:1929] T 2947502815d742e89cacb474b7fbb249 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:28.957348 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_af3343b5c3c244deb6f22b9fba10c2b9" } new_table_name: "default.Table_750ecb905839434fa9a1aa72f87394be" modify_external_catalogs: true
I20251025 07:52:28.960556 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 724d0b7e1b65445a9a1f056852983bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:28.968075 877 tablet_service.cc:1505] Processing CreateTablet for tablet aaef24ccdfba45fe8204327723df834e (DEFAULT_TABLE table=default.Table_bd5089a7892041649dd05b7fb0961255 [id=840b3ffcb44144548843457c48f46380]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.962643 1169 tablet_bootstrap.cc:492] T cf2cab1552bd4e7394797c93b149f450 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:28.970978 1169 tablet_bootstrap.cc:654] T cf2cab1552bd4e7394797c93b149f450 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:28.971138 884 tablet_service.cc:1505] Processing CreateTablet for tablet cb829649f73d4d3f8862d268c953bd86 (DEFAULT_TABLE table=default.Table_4b70d9acf0f64bcd992a894b129927c3 [id=7944225d3ce7439b881bfb9289d9f040]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.971812 1170 log.cc:1199] T 2947502815d742e89cacb474b7fbb249 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/2947502815d742e89cacb474b7fbb249
I20251025 07:52:28.972136 1170 ts_tablet_manager.cc:1950] T 2947502815d742e89cacb474b7fbb249 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:28.973349 1170 tablet_replica.cc:333] T 9bf486d82b984d2c89ee32e84a85894f P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:28.973520 1170 raft_consensus.cc:2243] T 9bf486d82b984d2c89ee32e84a85894f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:28.973667 1170 raft_consensus.cc:2272] T 9bf486d82b984d2c89ee32e84a85894f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:28.962239 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet b3a0af893a8947848e12093d0a7f0229
W20251025 07:52:28.964685 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a6ddefa1738f4ff98586900c344359d2 consensus state has no leader
I20251025 07:52:28.964968 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_563d524d00364bd2819969149a8f436c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:28.973124 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 2947502815d742e89cacb474b7fbb249 (table default.Table_c96e62972b3c4ee48b6c3ea90831a34f [id=e7a5b463914840c980a306b7d46ea871]) successfully deleted
W20251025 07:52:28.977046 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 80ee75935d7e461d9d4066fac9819bef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80ee75935d7e461d9d4066fac9819bef already in progress: creating tablet
W20251025 07:52:28.977149 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_563d524d00364bd2819969149a8f436c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:28.977283 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 71432a6848944a0da06ef9b81f160dc4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71432a6848944a0da06ef9b81f160dc4 already in progress: creating tablet
I20251025 07:52:28.977695 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet b3a0af893a8947848e12093d0a7f0229 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:28.978351 1170 ts_tablet_manager.cc:1916] T 9bf486d82b984d2c89ee32e84a85894f P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:28.980343 890 tablet_service.cc:1505] Processing CreateTablet for tablet 04d27d54fdb7483b869408af65ff85c9 (DEFAULT_TABLE table=default.Table_cfa54e1a04154e2eb8747fd3a4812883 [id=bf3a3f8dae184d999f2159dfcc7803b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.988381 887 tablet_service.cc:1505] Processing CreateTablet for tablet 1912e59548874e0e8e989050ec0064c5 (DEFAULT_TABLE table=default.Table_22ddca5b9e8449ef9fbdeb29212ff9a5 [id=74ec1fa4786e40109252885ea225ed66]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.988106 751 tablet_service.cc:1505] Processing CreateTablet for tablet a4e4b99be0434baaa75d988281797846 (DEFAULT_TABLE table=default.Table_dac528b081844b7c86027169c6234c27 [id=b0caa78ee8c84aa8bea4818cf9feb70c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.995983 1178 raft_consensus.cc:493] T 9bf486d82b984d2c89ee32e84a85894f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:28.997596 745 tablet_service.cc:1505] Processing CreateTablet for tablet 1912e59548874e0e8e989050ec0064c5 (DEFAULT_TABLE table=default.Table_22ddca5b9e8449ef9fbdeb29212ff9a5 [id=74ec1fa4786e40109252885ea225ed66]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.985250 1169 tablet_bootstrap.cc:492] T cf2cab1552bd4e7394797c93b149f450 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:29.000276 1169 ts_tablet_manager.cc:1403] T cf2cab1552bd4e7394797c93b149f450 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.000s
I20251025 07:52:29.001206 1169 raft_consensus.cc:359] T cf2cab1552bd4e7394797c93b149f450 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.001400 1169 raft_consensus.cc:385] T cf2cab1552bd4e7394797c93b149f450 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:29.001483 1169 raft_consensus.cc:740] T cf2cab1552bd4e7394797c93b149f450 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:29.001693 1169 consensus_queue.cc:260] T cf2cab1552bd4e7394797c93b149f450 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.002244 1169 ts_tablet_manager.cc:1434] T cf2cab1552bd4e7394797c93b149f450 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:29.002533 1169 tablet_bootstrap.cc:492] T 5caac9c943bf42a5b2ae522d87fdfe8e P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:29.003271 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_503b86ee2ab545afa9829bebcc9ea11d" } modify_external_catalogs: true
I20251025 07:52:29.003460 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_503b86ee2ab545afa9829bebcc9ea11d" } modify_external_catalogs: true
I20251025 07:52:29.004046 889 tablet_service.cc:1505] Processing CreateTablet for tablet a4e4b99be0434baaa75d988281797846 (DEFAULT_TABLE table=default.Table_dac528b081844b7c86027169c6234c27 [id=b0caa78ee8c84aa8bea4818cf9feb70c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.004261 1169 tablet_bootstrap.cc:654] T 5caac9c943bf42a5b2ae522d87fdfe8e P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:29.009325 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet cf2cab1552bd4e7394797c93b149f450 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:29.009985 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_f34f271c2f9242efa3ef3e78302be7d6" } new_table_name: "default.Table_9afef816a2c94c03bec6f2b52f3992e1" modify_external_catalogs: true
I20251025 07:52:29.012532 1178 raft_consensus.cc:515] T 9bf486d82b984d2c89ee32e84a85894f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.013276 1178 leader_election.cc:290] T 9bf486d82b984d2c89ee32e84a85894f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:29.013437 1178 raft_consensus.cc:493] T 3c3eb43209f24592820638fc25486b78 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:29.013573 1178 raft_consensus.cc:515] T 3c3eb43209f24592820638fc25486b78 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.014119 1178 leader_election.cc:290] T 3c3eb43209f24592820638fc25486b78 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:29.000801 753 tablet_service.cc:1505] Processing CreateTablet for tablet 8eaf6c97ab9b41e9ae746a9c5abe1735 (DEFAULT_TABLE table=default.Table_bf5cb8a733934f4ca29ecc9d473e4f2c [id=4bb4741007834a24aca34e608365ea03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.015183 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 80ee75935d7e461d9d4066fac9819bef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:29.015628 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9bf486d82b984d2c89ee32e84a85894f" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:29.015745 878 tablet_service.cc:1505] Processing CreateTablet for tablet 36262ab8c6964dd684c01b4db6219a4f (DEFAULT_TABLE table=default.Table_4f9a077b76bf42979239aeb6710131ec [id=f249f30497bc450c887d659721469de8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.016301 883 tablet_service.cc:1505] Processing CreateTablet for tablet 8eaf6c97ab9b41e9ae746a9c5abe1735 (DEFAULT_TABLE table=default.Table_bf5cb8a733934f4ca29ecc9d473e4f2c [id=4bb4741007834a24aca34e608365ea03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.016712 894 tablet_service.cc:1505] Processing CreateTablet for tablet d9bfd7c0c3ad4e5b98b6708793d61d41 (DEFAULT_TABLE table=default.Table_580b09b75790418e8dcf68fc88bbbf47 [id=5202bb531f384527a167bf9efe00d9c6]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:29.017174 850 leader_election.cc:343] T 1483c40eee834f20a92e6b4cd89775b6 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:29.017330 850 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:45417, user_credentials={real_user=slave}} blocked reactor thread for 65720.7us
I20251025 07:52:29.017787 892 tablet_service.cc:1505] Processing CreateTablet for tablet f43ef51ee45d47a99ec732b49d50d3b4 (DEFAULT_TABLE table=default.Table_0b0bd6b71b92454fb8cdf967ffd8349e [id=e9b21659de7d4276aa1f9effb7076eb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:28.999259 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet aaef24ccdfba45fe8204327723df834e
I20251025 07:52:29.018883 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4e4b99be0434baaa75d988281797846. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.019363 891 tablet_service.cc:1505] Processing CreateTablet for tablet 4c155aa7fa5b4d68bbab1918b8432372 (DEFAULT_TABLE table=default.Table_7f531c5ddd174b49b0117a226cd0d91f [id=28bbaad474164a8c8c97a76bb8081791]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.019330 876 tablet_service.cc:1505] Processing CreateTablet for tablet 292fe04055954afd8c08ac2636b68191 (DEFAULT_TABLE table=default.Table_6588890a31184cc2ab6ccb9b72e60429 [id=a253ce0253bd4da4814741c482193d8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.019766 1172 ts_tablet_manager.cc:1950] T 2fac7b819e2642378e0d3f9230d0259c P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:29.020663 1172 tablet_replica.cc:333] T 87a93778620f4c84b3b3f7a3d7d7e87f P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:29.020941 1172 raft_consensus.cc:2243] T 87a93778620f4c84b3b3f7a3d7d7e87f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:29.021122 1172 raft_consensus.cc:2272] T 87a93778620f4c84b3b3f7a3d7d7e87f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:29.021778 1172 ts_tablet_manager.cc:1916] T 87a93778620f4c84b3b3f7a3d7d7e87f P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:29.021946 875 tablet_service.cc:1505] Processing CreateTablet for tablet 0f5f6ca6c3cd4d369690e425387a4344 (DEFAULT_TABLE table=default.Table_e317f8d728da4c76886f67d0995ab7f4 [id=4d2121dea4904850aecb65da5dd2d8f9]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:29.022244 712 leader_election.cc:343] T 3c3eb43209f24592820638fc25486b78 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 3c3eb43209f24592820638fc25486b78
I20251025 07:52:29.022746 1166 raft_consensus.cc:359] T f51ec427dd9349a48eb221db04cf242b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.022982 1166 raft_consensus.cc:385] T f51ec427dd9349a48eb221db04cf242b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:29.023079 1166 raft_consensus.cc:740] T f51ec427dd9349a48eb221db04cf242b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:29.023371 1166 consensus_queue.cc:260] T f51ec427dd9349a48eb221db04cf242b P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.024009 1166 ts_tablet_manager.cc:1434] T f51ec427dd9349a48eb221db04cf242b P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.089s user 0.002s sys 0.000s
I20251025 07:52:29.024401 1166 tablet_bootstrap.cc:492] T 8c6ff4c73e624af3bf2de36c4493b462 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:29.024714 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aaef24ccdfba45fe8204327723df834e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.026459 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c155aa7fa5b4d68bbab1918b8432372. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.030033 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 2fac7b819e2642378e0d3f9230d0259c (table default.Table_dda83c899ebe4be999ac2ab537d3d323 [id=6605e11e11804d408a6a4a9d403dde95]) successfully deleted
W20251025 07:52:29.030733 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 7fbae8be73bf4e18b5a5c4ec4cdebc12 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7fbae8be73bf4e18b5a5c4ec4cdebc12 already in progress: creating tablet
I20251025 07:52:29.033922 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet aaef24ccdfba45fe8204327723df834e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:47408
W20251025 07:52:29.035545 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0325d462441349ef81841f1b0cab2a20 consensus state has no leader
I20251025 07:52:29.038033 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36262ab8c6964dd684c01b4db6219a4f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:29.040678 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b17d2a9ad6be49d29f2413666fba7e69 consensus state has no leader
I20251025 07:52:29.045994 1018 tablet_service.cc:1505] Processing CreateTablet for tablet c87ab6a3d08f43e0b2fe58a6d7b11588 (DEFAULT_TABLE table=default.Table_dde55dddf5d14de084003c22da08f6cd [id=c3f37f5469ab4d60a7597994c11357cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.046490 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c87ab6a3d08f43e0b2fe58a6d7b11588. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.046949 1019 tablet_service.cc:1505] Processing CreateTablet for tablet ab42909afc524f2fa3c1041495d13a59 (DEFAULT_TABLE table=default.Table_d4c06afc318d4f35a73d15e893b90da8 [id=40c29df99590408cbd5c583d4fc741eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.048539 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab42909afc524f2fa3c1041495d13a59. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.052279 520 catalog_manager.cc:5649] T 796413da731245459cc5dffacc3deb2f P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } }
I20251025 07:52:29.052728 520 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 796413da731245459cc5dffacc3deb2f (table default.Table_293596c64cca4768bc5869fd9924f856 [id=d1b9d89b22bc4431a7d59ceeb9d75050]). Expected version 2 got 0
I20251025 07:52:29.052994 520 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 50ffbcdd4a9247bf99b8e2361bc02ef5 (table default.Table_9f0ae2442a334cf8ab13f2eba387ebf3 [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 3 got 0
I20251025 07:52:29.053125 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8eaf6c97ab9b41e9ae746a9c5abe1735. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.054944 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 292fe04055954afd8c08ac2636b68191. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.049875 1320 leader_election.cc:290] T 1483c40eee834f20a92e6b4cd89775b6 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:29.056010 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f5f6ca6c3cd4d369690e425387a4344. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.057917 1025 tablet_service.cc:1505] Processing CreateTablet for tablet bc2d16b33ca8426abbd7bd38339c9f1d (DEFAULT_TABLE table=default.Table_86de368dc5bb4eb2a051a9db92a580dc [id=98d2b7797e33485e8872054aa2eb8ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.058386 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc2d16b33ca8426abbd7bd38339c9f1d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:29.058507 847 leader_election.cc:343] T 87a93778620f4c84b3b3f7a3d7d7e87f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:29.065999 756 tablet_service.cc:1505] Processing CreateTablet for tablet 4c155aa7fa5b4d68bbab1918b8432372 (DEFAULT_TABLE table=default.Table_7f531c5ddd174b49b0117a226cd0d91f [id=28bbaad474164a8c8c97a76bb8081791]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.065997 1169 tablet_bootstrap.cc:492] T 5caac9c943bf42a5b2ae522d87fdfe8e P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:29.066313 1169 ts_tablet_manager.cc:1403] T 5caac9c943bf42a5b2ae522d87fdfe8e P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.003s
W20251025 07:52:29.066439 712 leader_election.cc:343] T 9bf486d82b984d2c89ee32e84a85894f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:29.066679 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9bfd7c0c3ad4e5b98b6708793d61d41. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.066697 774 raft_consensus.cc:1275] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Refusing update from remote peer 3aea2869ff814cc7b142012faa274382: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 07:52:29.066664 739 tablet_service.cc:1505] Processing CreateTablet for tablet aaef24ccdfba45fe8204327723df834e (DEFAULT_TABLE table=default.Table_bd5089a7892041649dd05b7fb0961255 [id=840b3ffcb44144548843457c48f46380]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.067648 1169 raft_consensus.cc:359] T 5caac9c943bf42a5b2ae522d87fdfe8e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.067925 1169 raft_consensus.cc:385] T 5caac9c943bf42a5b2ae522d87fdfe8e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:29.068086 1169 raft_consensus.cc:740] T 5caac9c943bf42a5b2ae522d87fdfe8e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:29.068382 1169 consensus_queue.cc:260] T 5caac9c943bf42a5b2ae522d87fdfe8e P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.069370 1169 ts_tablet_manager.cc:1434] T 5caac9c943bf42a5b2ae522d87fdfe8e P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251025 07:52:29.069813 1169 tablet_bootstrap.cc:492] T dd898450f2604987b08639cf87d0e402 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:29.074927 1169 tablet_bootstrap.cc:654] T dd898450f2604987b08639cf87d0e402 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:29.078382 1170 ts_tablet_manager.cc:1929] T 9bf486d82b984d2c89ee32e84a85894f P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:29.078583 1170 log.cc:1199] T 9bf486d82b984d2c89ee32e84a85894f P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/9bf486d82b984d2c89ee32e84a85894f
I20251025 07:52:29.079010 1170 ts_tablet_manager.cc:1950] T 9bf486d82b984d2c89ee32e84a85894f P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:29.080830 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04d27d54fdb7483b869408af65ff85c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.082271 1170 tablet_replica.cc:333] T cf2cab1552bd4e7394797c93b149f450 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:29.082281 847 leader_election.cc:304] T 87a93778620f4c84b3b3f7a3d7d7e87f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:29.082577 1170 raft_consensus.cc:2243] T cf2cab1552bd4e7394797c93b149f450 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:29.082937 1170 raft_consensus.cc:2272] T cf2cab1552bd4e7394797c93b149f450 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:29.084089 1170 ts_tablet_manager.cc:1916] T cf2cab1552bd4e7394797c93b149f450 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:29.086643 1169 tablet_bootstrap.cc:492] T dd898450f2604987b08639cf87d0e402 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:29.087105 1169 ts_tablet_manager.cc:1403] T dd898450f2604987b08639cf87d0e402 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.003s
I20251025 07:52:29.087188 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 9bf486d82b984d2c89ee32e84a85894f (table default.Table_21a2393beb5d49a8ba839143c8328569 [id=5fcd7d82bf9443edbf595c28fcab99ae]) successfully deleted
W20251025 07:52:29.087486 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet b3a0af893a8947848e12093d0a7f0229 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3a0af893a8947848e12093d0a7f0229 already in progress: creating tablet
I20251025 07:52:29.088186 1169 raft_consensus.cc:359] T dd898450f2604987b08639cf87d0e402 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.088436 1169 raft_consensus.cc:385] T dd898450f2604987b08639cf87d0e402 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:29.088588 1169 raft_consensus.cc:740] T dd898450f2604987b08639cf87d0e402 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:29.089053 1169 consensus_queue.cc:260] T dd898450f2604987b08639cf87d0e402 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.089356 1247 raft_consensus.cc:493] T dd898450f2604987b08639cf87d0e402 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:29.089501 1247 raft_consensus.cc:515] T dd898450f2604987b08639cf87d0e402 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.090183 1247 leader_election.cc:290] T dd898450f2604987b08639cf87d0e402 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:29.090876 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 5caac9c943bf42a5b2ae522d87fdfe8e with delete_type TABLET_DATA_DELETED (replaced by tablet 4f25c9caa05b4da4bd56a8a21fde423c) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:29.066999 738 tablet_service.cc:1505] Processing CreateTablet for tablet 0f5f6ca6c3cd4d369690e425387a4344 (DEFAULT_TABLE table=default.Table_e317f8d728da4c76886f67d0995ab7f4 [id=4d2121dea4904850aecb65da5dd2d8f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.094321 1171 tablet_replica.cc:333] T 50433881c0eb4aa5a389b6f5b4a4337a P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:29.096117 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1912e59548874e0e8e989050ec0064c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.097993 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet dd898450f2604987b08639cf87d0e402 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:29.098963 885 tablet_service.cc:1505] Processing CreateTablet for tablet 34315f64b89148efbf1f20a43d5c5efb (DEFAULT_TABLE table=default.Table_4b3d7b3a731942189cdc214938a45820 [id=88f1e8b1c5fa4913a5f7b5b2c6cbf6ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.099324 1169 ts_tablet_manager.cc:1434] T dd898450f2604987b08639cf87d0e402 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20251025 07:52:29.101830 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 71432a6848944a0da06ef9b81f160dc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:29.104332 1170 ts_tablet_manager.cc:1929] T cf2cab1552bd4e7394797c93b149f450 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:29.104561 1170 log.cc:1199] T cf2cab1552bd4e7394797c93b149f450 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/cf2cab1552bd4e7394797c93b149f450
I20251025 07:52:29.105140 1170 ts_tablet_manager.cc:1950] T cf2cab1552bd4e7394797c93b149f450 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:29.106513 1170 tablet_replica.cc:333] T 5caac9c943bf42a5b2ae522d87fdfe8e P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:29.106472 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet cf2cab1552bd4e7394797c93b149f450 (table default.Table_4c12bad86d7946b9ac019c8e3856dcba [id=7de1e93439f94b2ebd4db3ed27f95872]) successfully deleted
W20251025 07:52:29.107033 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet aaef24ccdfba45fe8204327723df834e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aaef24ccdfba45fe8204327723df834e already in progress: creating tablet
I20251025 07:52:29.111600 1170 raft_consensus.cc:2243] T 5caac9c943bf42a5b2ae522d87fdfe8e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:29.111725 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_00b3b394e5ed46faaac0202afe54b437"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:29.112588 881 tablet_service.cc:1505] Processing CreateTablet for tablet 4b28215f61574cf48c447dd01955ac77 (DEFAULT_TABLE table=default.Table_e8be4237bba844c8963e8955729b4fba [id=51b12903e6024e1a9adc2f9fa7400937]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:29.113070 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00b3b394e5ed46faaac0202afe54b437 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:29.113425 893 tablet_service.cc:1552] Processing DeleteTablet for tablet b3a0af893a8947848e12093d0a7f0229 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:29.113122 1170 raft_consensus.cc:2272] T 5caac9c943bf42a5b2ae522d87fdfe8e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:29.113610 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd898450f2604987b08639cf87d0e402" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:29.116598 886 tablet_service.cc:1505] Processing CreateTablet for tablet 26c849d376d143c6bef0c8e0e1d7d03a (DEFAULT_TABLE table=default.Table_f8c38604344c429295d6d0ab9cb85b13 [id=c2f7a6281b6f47c990b096412f2201a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.119151 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet b947ecc352e54f4caa73c6ca97372256
I20251025 07:52:29.120658 880 tablet_service.cc:1505] Processing CreateTablet for tablet 04449cc97f404a19b6a1b20dea074f95 (DEFAULT_TABLE table=default.Table_f5af41a9cdee45ffa4a44089ba80464b [id=e041946c093b4c789ce175c736cc230b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.121066 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4e4b99be0434baaa75d988281797846. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:29.121958 847 raft_consensus.cc:2686] T 87a93778620f4c84b3b3f7a3d7d7e87f P 3aea2869ff814cc7b142012faa274382: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251025 07:52:29.122018 913 raft_consensus.cc:2468] T dd898450f2604987b08639cf87d0e402 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
W20251025 07:52:29.122285 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:46231, user_credentials={real_user=slave}} blocked reactor thread for 131159us
I20251025 07:52:29.122697 983 leader_election.cc:304] T dd898450f2604987b08639cf87d0e402 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:29.122998 1247 raft_consensus.cc:2804] T dd898450f2604987b08639cf87d0e402 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:29.123126 1247 raft_consensus.cc:493] T dd898450f2604987b08639cf87d0e402 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:29.123221 1247 raft_consensus.cc:3060] T dd898450f2604987b08639cf87d0e402 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:29.124146 1320 consensus_queue.cc:1048] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:29.125052 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 2fac7b819e2642378e0d3f9230d0259c (table default.Table_dda83c899ebe4be999ac2ab537d3d323 [id=6605e11e11804d408a6a4a9d403dde95]) successfully deleted
W20251025 07:52:29.125273 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 7fbae8be73bf4e18b5a5c4ec4cdebc12 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7fbae8be73bf4e18b5a5c4ec4cdebc12 already in progress: creating tablet
W20251025 07:52:29.125485 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 3abfd52327a04ba88625f391a8658fbb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3abfd52327a04ba88625f391a8658fbb already in progress: creating tablet
I20251025 07:52:29.126403 888 tablet_service.cc:1505] Processing CreateTablet for tablet bc2d16b33ca8426abbd7bd38339c9f1d (DEFAULT_TABLE table=default.Table_86de368dc5bb4eb2a051a9db92a580dc [id=98d2b7797e33485e8872054aa2eb8ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.126539 1247 raft_consensus.cc:515] T dd898450f2604987b08639cf87d0e402 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.127336 879 tablet_service.cc:1552] Processing DeleteTablet for tablet b947ecc352e54f4caa73c6ca97372256 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:29.127403 1247 leader_election.cc:290] T dd898450f2604987b08639cf87d0e402 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:29.127938 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f43ef51ee45d47a99ec732b49d50d3b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.128207 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b28215f61574cf48c447dd01955ac77. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.131373 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb829649f73d4d3f8862d268c953bd86. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.132256 1320 raft_consensus.cc:493] T a231f3d31b204f7b82f018892ccc8435 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:29.132431 1320 raft_consensus.cc:515] T a231f3d31b204f7b82f018892ccc8435 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.108630 1169 tablet_bootstrap.cc:492] T 1483c40eee834f20a92e6b4cd89775b6 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:29.133070 1166 tablet_bootstrap.cc:654] T 8c6ff4c73e624af3bf2de36c4493b462 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:29.134928 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd898450f2604987b08639cf87d0e402" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:29.135586 914 raft_consensus.cc:3060] T dd898450f2604987b08639cf87d0e402 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:29.136773 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_bc1398c7c53846da95b511e6a442e416"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:29.137368 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc1398c7c53846da95b511e6a442e416 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:29.137566 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_d2a7e16c6f0d4b2886c16bfe6b685c00" } modify_external_catalogs: true
I20251025 07:52:29.137723 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_d2a7e16c6f0d4b2886c16bfe6b685c00" } modify_external_catalogs: true
I20251025 07:52:29.138720 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_7ce7627f71134d5c989f9ba5c6766a2a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:29.139194 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ce7627f71134d5c989f9ba5c6766a2a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:29.140328 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_bcd5afa1072f47898d67504fc72b84c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:29.140496 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_490b6bd20f694746afabf7be7f74fbc0" } new_table_name: "default.Table_ad729c05dcae46f19df251a219aeccc4" modify_external_catalogs: true
W20251025 07:52:29.140848 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bcd5afa1072f47898d67504fc72b84c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:29.141683 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_cd12897d1fc5454ab07cdc13fefb6978"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:29.141940 891 tablet_service.cc:1505] Processing CreateTablet for tablet c87ab6a3d08f43e0b2fe58a6d7b11588 (DEFAULT_TABLE table=default.Table_dde55dddf5d14de084003c22da08f6cd [id=c3f37f5469ab4d60a7597994c11357cf]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:29.142081 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd12897d1fc5454ab07cdc13fefb6978 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:29.144053 882 tablet_service.cc:1505] Processing CreateTablet for tablet ab42909afc524f2fa3c1041495d13a59 (DEFAULT_TABLE table=default.Table_d4c06afc318d4f35a73d15e893b90da8 [id=40c29df99590408cbd5c583d4fc741eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.145531 1171 raft_consensus.cc:2243] T 50433881c0eb4aa5a389b6f5b4a4337a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:29.142381 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_400f06d166054eeb97890ad56453afb2" } modify_external_catalogs: true
I20251025 07:52:29.147173 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_400f06d166054eeb97890ad56453afb2" } modify_external_catalogs: true
I20251025 07:52:29.148417 893 tablet_service.cc:1552] Processing DeleteTablet for tablet aaef24ccdfba45fe8204327723df834e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:29.143062 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_b53575428f5b4c25a5c912428e5d4f2d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:29.149443 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b53575428f5b4c25a5c912428e5d4f2d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:29.155463 1320 leader_election.cc:290] T a231f3d31b204f7b82f018892ccc8435 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:29.156116 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_580b09b75790418e8dcf68fc88bbbf47" } new_table_name: "default.Table_0c74279815ab4a5186497b68bb932523" modify_external_catalogs: true
W20251025 07:52:29.156744 850 leader_election.cc:343] T a231f3d31b204f7b82f018892ccc8435 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: a231f3d31b204f7b82f018892ccc8435
I20251025 07:52:29.159168 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26c849d376d143c6bef0c8e0e1d7d03a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.159067 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_d5abcbf8c8c741a2b40880a3467f2bcd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:29.159579 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5abcbf8c8c741a2b40880a3467f2bcd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:29.162464 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc2d16b33ca8426abbd7bd38339c9f1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.171190 1169 tablet_bootstrap.cc:654] T 1483c40eee834f20a92e6b4cd89775b6 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:29.175241 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 80fe3255ed614270bd8352a2853fbc43
I20251025 07:52:29.175484 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04449cc97f404a19b6a1b20dea074f95. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.181440 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34315f64b89148efbf1f20a43d5c5efb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.113981 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "796413da731245459cc5dffacc3deb2f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:29.184077 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8eaf6c97ab9b41e9ae746a9c5abe1735. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.185273 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b31177a062b467187d68ed2034f5690" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
W20251025 07:52:29.185781 710 leader_election.cc:343] T 50433881c0eb4aa5a389b6f5b4a4337a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 50433881c0eb4aa5a389b6f5b4a4337a
I20251025 07:52:29.188447 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_0f4ba264122a4aad989cf6694f0e7ccb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:29.189082 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f4ba264122a4aad989cf6694f0e7ccb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:29.190404 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_c43ac1adc76d4ccf819411ec76775932"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:29.190899 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c43ac1adc76d4ccf819411ec76775932 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:29.191916 710 leader_election.cc:304] T 50433881c0eb4aa5a389b6f5b4a4337a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:29.192287 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:43563, user_credentials={real_user=slave}} blocked reactor thread for 69662.7us
I20251025 07:52:29.183264 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet b947ecc352e54f4caa73c6ca97372256 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:47408
W20251025 07:52:29.194615 710 leader_election.cc:343] T 83084be89b2646b0bda208b3f6f60d77 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 83084be89b2646b0bda208b3f6f60d77
I20251025 07:52:29.194654 774 raft_consensus.cc:3060] T 796413da731245459cc5dffacc3deb2f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:29.194788 710 leader_election.cc:304] T 83084be89b2646b0bda208b3f6f60d77 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:29.195212 710 leader_election.cc:343] T 9bf486d82b984d2c89ee32e84a85894f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 9bf486d82b984d2c89ee32e84a85894f
I20251025 07:52:29.195322 710 leader_election.cc:304] T 9bf486d82b984d2c89ee32e84a85894f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:29.195616 1178 raft_consensus.cc:2749] T 83084be89b2646b0bda208b3f6f60d77 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:29.195830 1178 raft_consensus.cc:2749] T 9bf486d82b984d2c89ee32e84a85894f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:29.200728 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c155aa7fa5b4d68bbab1918b8432372. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.201972 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_f84af906af804c7a8e98ad602783d747" } modify_external_catalogs: true
I20251025 07:52:29.202205 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_f84af906af804c7a8e98ad602783d747" } modify_external_catalogs: true
I20251025 07:52:29.193688 1170 ts_tablet_manager.cc:1916] T 5caac9c943bf42a5b2ae522d87fdfe8e P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:29.204416 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd898450f2604987b08639cf87d0e402" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:29.207679 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f5f6ca6c3cd4d369690e425387a4344. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:29.209669 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2c50ec95e48e49b59a76fac4fc862d3d consensus state has no leader
I20251025 07:52:29.210876 749 tablet_service.cc:1505] Processing CreateTablet for tablet 04d27d54fdb7483b869408af65ff85c9 (DEFAULT_TABLE table=default.Table_cfa54e1a04154e2eb8747fd3a4812883 [id=bf3a3f8dae184d999f2159dfcc7803b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.212733 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_ac2ef60285674f918695309a38ad40b7" } new_table_name: "default.Table_2c4afce2b14c41e29c5034dc68d31ead" modify_external_catalogs: true
I20251025 07:52:29.213605 1171 raft_consensus.cc:2272] T 50433881c0eb4aa5a389b6f5b4a4337a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:29.213783 1433 raft_consensus.cc:2706] T 50433881c0eb4aa5a389b6f5b4a4337a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Received pre-election callback for term 1 while not running: Illegal state: RaftConsensus is not running: State = Stopped
I20251025 07:52:29.214082 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 80fe3255ed614270bd8352a2853fbc43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:29.216717 750 tablet_service.cc:1505] Processing CreateTablet for tablet cb829649f73d4d3f8862d268c953bd86 (DEFAULT_TABLE table=default.Table_4b70d9acf0f64bcd992a894b129927c3 [id=7944225d3ce7439b881bfb9289d9f040]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.217140 744 tablet_service.cc:1505] Processing CreateTablet for tablet 4b28215f61574cf48c447dd01955ac77 (DEFAULT_TABLE table=default.Table_e8be4237bba844c8963e8955729b4fba [id=51b12903e6024e1a9adc2f9fa7400937]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.217486 1167 tablet_bootstrap.cc:492] T b00153c97104470883f6089d71d23708 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:29.217633 1167 ts_tablet_manager.cc:1403] T b00153c97104470883f6089d71d23708 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.495s user 0.056s sys 0.081s
I20251025 07:52:29.218480 1167 raft_consensus.cc:359] T b00153c97104470883f6089d71d23708 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.218653 1167 raft_consensus.cc:385] T b00153c97104470883f6089d71d23708 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:29.218745 1167 raft_consensus.cc:740] T b00153c97104470883f6089d71d23708 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:29.218955 1167 consensus_queue.cc:260] T b00153c97104470883f6089d71d23708 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.219470 1167 ts_tablet_manager.cc:1434] T b00153c97104470883f6089d71d23708 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:29.219823 1167 tablet_bootstrap.cc:492] T cf2cab1552bd4e7394797c93b149f450 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:29.220268 1178 raft_consensus.cc:493] T 94408a3a90224645ba3cacccf5de33f1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:29.220449 1178 raft_consensus.cc:515] T 94408a3a90224645ba3cacccf5de33f1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.221047 1178 leader_election.cc:290] T 94408a3a90224645ba3cacccf5de33f1 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:29.222544 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_a62ca3a38f5f4dc5a9c776342d8cf293"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:29.223174 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a62ca3a38f5f4dc5a9c776342d8cf293 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:29.223505 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aaef24ccdfba45fe8204327723df834e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.224167 742 tablet_service.cc:1505] Processing CreateTablet for tablet 26c849d376d143c6bef0c8e0e1d7d03a (DEFAULT_TABLE table=default.Table_f8c38604344c429295d6d0ab9cb85b13 [id=c2f7a6281b6f47c990b096412f2201a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.222877 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1483c40eee834f20a92e6b4cd89775b6" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:29.228109 1171 ts_tablet_manager.cc:1916] T 50433881c0eb4aa5a389b6f5b4a4337a P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:29.234241 982 leader_election.cc:343] T dd898450f2604987b08639cf87d0e402 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:29.234719 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_e3b368ac698c46f4a8c947d02f9a2680" } modify_external_catalogs: true
I20251025 07:52:29.235008 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_e3b368ac698c46f4a8c947d02f9a2680" } modify_external_catalogs: true
W20251025 07:52:29.235918 712 leader_election.cc:343] T 94408a3a90224645ba3cacccf5de33f1 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 94408a3a90224645ba3cacccf5de33f1
I20251025 07:52:29.236006 737 tablet_service.cc:1552] Processing DeleteTablet for tablet 80ee75935d7e461d9d4066fac9819bef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:29.236568 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b28215f61574cf48c447dd01955ac77. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.243080 774 raft_consensus.cc:2468] T 796413da731245459cc5dffacc3deb2f P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:29.248788 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_f5af41a9cdee45ffa4a44089ba80464b" } new_table_name: "default.Table_332b308220f64c6dbb5f29711d8463b3" modify_external_catalogs: true
I20251025 07:52:29.250070 743 tablet_service.cc:1552] Processing DeleteTablet for tablet b3a0af893a8947848e12093d0a7f0229 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:29.250321 743 tablet_service.cc:1552] Processing DeleteTablet for tablet aaef24ccdfba45fe8204327723df834e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:29.252899 743 tablet_service.cc:1505] Processing CreateTablet for tablet ab42909afc524f2fa3c1041495d13a59 (DEFAULT_TABLE table=default.Table_d4c06afc318d4f35a73d15e893b90da8 [id=40c29df99590408cbd5c583d4fc741eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.258038 755 tablet_service.cc:1505] Processing CreateTablet for tablet c87ab6a3d08f43e0b2fe58a6d7b11588 (DEFAULT_TABLE table=default.Table_dde55dddf5d14de084003c22da08f6cd [id=c3f37f5469ab4d60a7597994c11357cf]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:29.259614 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d9bfd7c0c3ad4e5b98b6708793d61d41 consensus state has no leader
I20251025 07:52:29.263707 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_d4c06afc318d4f35a73d15e893b90da8" } new_table_name: "default.Table_932ccd57bc1840e7b882d16177ceef9a" modify_external_catalogs: true
I20251025 07:52:29.268687 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 80fe3255ed614270bd8352a2853fbc43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:29.268936 752 tablet_service.cc:1552] Processing DeleteTablet for tablet b00153c97104470883f6089d71d23708 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:29.269091 737 tablet_service.cc:1552] Processing DeleteTablet for tablet 71432a6848944a0da06ef9b81f160dc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:29.261272 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd898450f2604987b08639cf87d0e402" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:29.269570 1169 tablet_bootstrap.cc:492] T 1483c40eee834f20a92e6b4cd89775b6 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:29.269788 1169 ts_tablet_manager.cc:1403] T 1483c40eee834f20a92e6b4cd89775b6 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.161s user 0.000s sys 0.005s
W20251025 07:52:29.269850 982 leader_election.cc:343] T dd898450f2604987b08639cf87d0e402 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:29.270641 1169 raft_consensus.cc:359] T 1483c40eee834f20a92e6b4cd89775b6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.270880 1169 raft_consensus.cc:385] T 1483c40eee834f20a92e6b4cd89775b6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:29.270989 1169 raft_consensus.cc:740] T 1483c40eee834f20a92e6b4cd89775b6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:29.271135 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04d27d54fdb7483b869408af65ff85c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.271291 1169 consensus_queue.cc:260] T 1483c40eee834f20a92e6b4cd89775b6 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.272455 746 tablet_service.cc:1552] Processing DeleteTablet for tablet b947ecc352e54f4caa73c6ca97372256 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:29.275413 754 tablet_service.cc:1505] Processing CreateTablet for tablet bc2d16b33ca8426abbd7bd38339c9f1d (DEFAULT_TABLE table=default.Table_86de368dc5bb4eb2a051a9db92a580dc [id=98d2b7797e33485e8872054aa2eb8ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.275846 1169 ts_tablet_manager.cc:1434] T 1483c40eee834f20a92e6b4cd89775b6 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20251025 07:52:29.278472 1169 tablet_bootstrap.cc:492] T f51ec427dd9349a48eb221db04cf242b P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:29.280589 1169 tablet_bootstrap.cc:654] T f51ec427dd9349a48eb221db04cf242b P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:29.287658 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c87ab6a3d08f43e0b2fe58a6d7b11588. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.306304 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc2d16b33ca8426abbd7bd38339c9f1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.317003 748 tablet_service.cc:1505] Processing CreateTablet for tablet f661d5909657455581dcb5e6eadd3c0b (DEFAULT_TABLE table=default.Table_b8a12e9a7a974006904fcef12bb8225a [id=9bfd1a38f56e4b70850139909aff091f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.317974 746 tablet_service.cc:1505] Processing CreateTablet for tablet 66f0d398e59a4175aa9fe90715124d1a (DEFAULT_TABLE table=default.Table_eafaca066a2548a686050fed8c380553 [id=17525d0c9b664dcdadbf42e85da234ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.318449 741 tablet_service.cc:1505] Processing CreateTablet for tablet 30418858bf95438686bdfe64ac733144 (DEFAULT_TABLE table=default.Table_8028c8ff15754b788ce12115fde3be3d [id=b4379401507b4a6f916f131a1b3a75d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.319167 1320 raft_consensus.cc:493] T 5caac9c943bf42a5b2ae522d87fdfe8e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:29.320749 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c87ab6a3d08f43e0b2fe58a6d7b11588. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.293454 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 1483c40eee834f20a92e6b4cd89775b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:29.313778 1023 tablet_service.cc:1505] Processing CreateTablet for tablet 09898735796541b0a8f3be22cdb6cc30 (DEFAULT_TABLE table=default.Table_bfcdc02fe0dc48a0ae793ecd15f50a70 [id=dec6d4b5bbe34710aed2d981e185f35e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.322453 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09898735796541b0a8f3be22cdb6cc30. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.314384 1020 tablet_service.cc:1505] Processing CreateTablet for tablet f661d5909657455581dcb5e6eadd3c0b (DEFAULT_TABLE table=default.Table_b8a12e9a7a974006904fcef12bb8225a [id=9bfd1a38f56e4b70850139909aff091f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.326416 1320 raft_consensus.cc:515] T 5caac9c943bf42a5b2ae522d87fdfe8e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.327314 1320 leader_election.cc:290] T 5caac9c943bf42a5b2ae522d87fdfe8e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:29.328600 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_892da226f7054e61b5ffc882b1d5c35a" } modify_external_catalogs: true
I20251025 07:52:29.329032 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_892da226f7054e61b5ffc882b1d5c35a" } modify_external_catalogs: true
I20251025 07:52:29.332401 737 tablet_service.cc:1505] Processing CreateTablet for tablet 6dcab53a020847a1aa27930265ba0205 (DEFAULT_TABLE table=default.Table_4bc4254fb03d4db99f40d7b07b2f89aa [id=cb79dc5b6680450cadf2350095389c9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.334506 1218 raft_consensus.cc:493] T f51ec427dd9349a48eb221db04cf242b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:29.334908 893 tablet_service.cc:1552] Processing DeleteTablet for tablet f51ec427dd9349a48eb221db04cf242b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:29.335129 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab42909afc524f2fa3c1041495d13a59. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.337641 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dcab53a020847a1aa27930265ba0205. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.337769 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 80fe3255ed614270bd8352a2853fbc43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:29.344096 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66f0d398e59a4175aa9fe90715124d1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.345084 740 tablet_service.cc:1505] Processing CreateTablet for tablet 09898735796541b0a8f3be22cdb6cc30 (DEFAULT_TABLE table=default.Table_bfcdc02fe0dc48a0ae793ecd15f50a70 [id=dec6d4b5bbe34710aed2d981e185f35e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.349328 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30418858bf95438686bdfe64ac733144. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.350279 752 tablet_service.cc:1505] Processing CreateTablet for tablet 050a5d3b5f934dd19bf2e221f2fe0780 (DEFAULT_TABLE table=default.Table_563d524d00364bd2819969149a8f436c [id=9f09387ac53b4027a147057debd10743]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.340951 1218 raft_consensus.cc:515] T f51ec427dd9349a48eb221db04cf242b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.356400 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5caac9c943bf42a5b2ae522d87fdfe8e" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:29.357208 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 551d07a9242740dcb9366bb2d707a4a8 consensus state has no leader
I20251025 07:52:29.360800 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f661d5909657455581dcb5e6eadd3c0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.362918 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_2bcb67fcbc85452faaadfa683987951c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:29.363540 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2bcb67fcbc85452faaadfa683987951c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:29.364733 850 leader_election.cc:343] T 5caac9c943bf42a5b2ae522d87fdfe8e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: Tablet not RUNNING: STOPPED
I20251025 07:52:29.365450 1170 ts_tablet_manager.cc:1929] T 5caac9c943bf42a5b2ae522d87fdfe8e P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:29.366902 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb829649f73d4d3f8862d268c953bd86. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.373592 1018 tablet_service.cc:1505] Processing CreateTablet for tablet 30418858bf95438686bdfe64ac733144 (DEFAULT_TABLE table=default.Table_8028c8ff15754b788ce12115fde3be3d [id=b4379401507b4a6f916f131a1b3a75d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.374656 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 6dcab53a020847a1aa27930265ba0205 (DEFAULT_TABLE table=default.Table_4bc4254fb03d4db99f40d7b07b2f89aa [id=cb79dc5b6680450cadf2350095389c9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.375878 1172 ts_tablet_manager.cc:1929] T 87a93778620f4c84b3b3f7a3d7d7e87f P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:29.376459 914 raft_consensus.cc:2468] T dd898450f2604987b08639cf87d0e402 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:29.376739 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 36c8fcafd5464b4e999276a8062a812a (DEFAULT_TABLE table=default.Table_19a3326e4da345c1bbeb2cf083d28e24 [id=11c088570f6940b1895d1691465c10b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.377951 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_4e81dfac4c7742c7986eb24e6d54a124"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:29.378578 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e81dfac4c7742c7986eb24e6d54a124 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:29.378643 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f51ec427dd9349a48eb221db04cf242b" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:29.379078 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 66f0d398e59a4175aa9fe90715124d1a (DEFAULT_TABLE table=default.Table_eafaca066a2548a686050fed8c380553 [id=17525d0c9b664dcdadbf42e85da234ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.380985 1172 log.cc:1199] T 87a93778620f4c84b3b3f7a3d7d7e87f P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/87a93778620f4c84b3b3f7a3d7d7e87f
I20251025 07:52:29.382629 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_a62ca3a38f5f4dc5a9c776342d8cf293" } modify_external_catalogs: true
I20251025 07:52:29.382815 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_a62ca3a38f5f4dc5a9c776342d8cf293" } modify_external_catalogs: true
I20251025 07:52:29.382867 1218 leader_election.cc:290] T f51ec427dd9349a48eb221db04cf242b P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:29.383198 893 tablet_service.cc:1505] Processing CreateTablet for tablet 09898735796541b0a8f3be22cdb6cc30 (DEFAULT_TABLE table=default.Table_bfcdc02fe0dc48a0ae793ecd15f50a70 [id=dec6d4b5bbe34710aed2d981e185f35e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.384327 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dcab53a020847a1aa27930265ba0205. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.384888 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30418858bf95438686bdfe64ac733144. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.384930 1024 tablet_service.cc:1505] Processing CreateTablet for tablet c9f7ba3709c745d4a625449d6e1c8ba1 (DEFAULT_TABLE table=default.Table_380ca630f4144f40a04084d2ef8600e3 [id=06b6a39af6764e44972d57ddd722f838]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.385300 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9f7ba3709c745d4a625449d6e1c8ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.385205 887 tablet_service.cc:1505] Processing CreateTablet for tablet f661d5909657455581dcb5e6eadd3c0b (DEFAULT_TABLE table=default.Table_b8a12e9a7a974006904fcef12bb8225a [id=9bfd1a38f56e4b70850139909aff091f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.386010 1172 ts_tablet_manager.cc:1950] T 87a93778620f4c84b3b3f7a3d7d7e87f P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:29.386758 875 tablet_service.cc:1505] Processing CreateTablet for tablet 30418858bf95438686bdfe64ac733144 (DEFAULT_TABLE table=default.Table_8028c8ff15754b788ce12115fde3be3d [id=b4379401507b4a6f916f131a1b3a75d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.388700 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66f0d398e59a4175aa9fe90715124d1a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:29.389225 850 leader_election.cc:343] T f51ec427dd9349a48eb221db04cf242b P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:29.389436 881 tablet_service.cc:1505] Processing CreateTablet for tablet 66f0d398e59a4175aa9fe90715124d1a (DEFAULT_TABLE table=default.Table_eafaca066a2548a686050fed8c380553 [id=17525d0c9b664dcdadbf42e85da234ca]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:29.390302 511 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 80ee75935d7e461d9d4066fac9819bef (table default.Table_bb258691f8844d85ab37eb73e9a4c314 [id=c80bab0ed06a459eb78b1d94df13f337], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 80ee75935d7e461d9d4066fac9819bef consensus state has no leader
I20251025 07:52:29.390528 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36c8fcafd5464b4e999276a8062a812a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.394202 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 050a5d3b5f934dd19bf2e221f2fe0780 (DEFAULT_TABLE table=default.Table_563d524d00364bd2819969149a8f436c [id=9f09387ac53b4027a147057debd10743]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.394575 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 050a5d3b5f934dd19bf2e221f2fe0780. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.395572 884 tablet_service.cc:1505] Processing CreateTablet for tablet 6dcab53a020847a1aa27930265ba0205 (DEFAULT_TABLE table=default.Table_4bc4254fb03d4db99f40d7b07b2f89aa [id=cb79dc5b6680450cadf2350095389c9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.398098 1172 tablet_replica.cc:333] T 5caac9c943bf42a5b2ae522d87fdfe8e P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:29.398198 1021 tablet_service.cc:1505] Processing CreateTablet for tablet b2044ca206634eb285197af4ba2073e7 (DEFAULT_TABLE table=default.Table_3292d47465544af0970067fb567e7ede [id=41fc0b99786749259c0a0059dec97ea1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.398344 1172 raft_consensus.cc:2243] T 5caac9c943bf42a5b2ae522d87fdfe8e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:29.398542 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2044ca206634eb285197af4ba2073e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.399176 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab42909afc524f2fa3c1041495d13a59. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.399981 1172 raft_consensus.cc:2272] T 5caac9c943bf42a5b2ae522d87fdfe8e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:29.400628 1172 ts_tablet_manager.cc:1916] T 5caac9c943bf42a5b2ae522d87fdfe8e P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:29.401294 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_91898f2a551144d18df0309c827855bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:29.401714 878 tablet_service.cc:1505] Processing CreateTablet for tablet 050a5d3b5f934dd19bf2e221f2fe0780 (DEFAULT_TABLE table=default.Table_563d524d00364bd2819969149a8f436c [id=9f09387ac53b4027a147057debd10743]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:29.401888 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91898f2a551144d18df0309c827855bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:29.402631 876 tablet_service.cc:1505] Processing CreateTablet for tablet 0551e5d694894e8fbeb66b5e0c1cd75b (DEFAULT_TABLE table=default.Table_2cd70245aa3d4809a0647e7fa499f62f [id=a003268ec3da425989fa4041699de4d3]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:29.404536 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 04449cc97f404a19b6a1b20dea074f95 consensus state has no leader
I20251025 07:52:29.404804 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f661d5909657455581dcb5e6eadd3c0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.405562 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 050a5d3b5f934dd19bf2e221f2fe0780. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.413599 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_9afef816a2c94c03bec6f2b52f3992e1" } new_table_name: "default.Table_fcddb31989ee4236a4981c12779b3f5d" modify_external_catalogs: true
W20251025 07:52:29.417210 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ab42909afc524f2fa3c1041495d13a59 consensus state has no leader
I20251025 07:52:29.426177 890 tablet_service.cc:1505] Processing CreateTablet for tablet c9f7ba3709c745d4a625449d6e1c8ba1 (DEFAULT_TABLE table=default.Table_380ca630f4144f40a04084d2ef8600e3 [id=06b6a39af6764e44972d57ddd722f838]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.426640 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0551e5d694894e8fbeb66b5e0c1cd75b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.427255 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_1f11da1b07914daabda9c8e29a9230fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:29.427837 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f11da1b07914daabda9c8e29a9230fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:29.430816 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 5fca357d7e24494fb6af4bd73d98833c
I20251025 07:52:29.434561 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9f7ba3709c745d4a625449d6e1c8ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.434767 888 tablet_service.cc:1505] Processing CreateTablet for tablet b2044ca206634eb285197af4ba2073e7 (DEFAULT_TABLE table=default.Table_3292d47465544af0970067fb567e7ede [id=41fc0b99786749259c0a0059dec97ea1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.435377 1170 log.cc:1199] T 5caac9c943bf42a5b2ae522d87fdfe8e P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/5caac9c943bf42a5b2ae522d87fdfe8e
I20251025 07:52:29.440032 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_499a6f88d70a45018371ed3806882506"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:29.440588 1170 ts_tablet_manager.cc:1950] T 5caac9c943bf42a5b2ae522d87fdfe8e P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
W20251025 07:52:29.440784 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_499a6f88d70a45018371ed3806882506 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:29.441831 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 0551e5d694894e8fbeb66b5e0c1cd75b (DEFAULT_TABLE table=default.Table_2cd70245aa3d4809a0647e7fa499f62f [id=a003268ec3da425989fa4041699de4d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.442713 983 leader_election.cc:304] T dd898450f2604987b08639cf87d0e402 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b
I20251025 07:52:29.444166 1247 raft_consensus.cc:2804] T dd898450f2604987b08639cf87d0e402 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:29.446185 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0551e5d694894e8fbeb66b5e0c1cd75b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.446918 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2044ca206634eb285197af4ba2073e7. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:29.449595 983 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:43563, user_credentials={real_user=slave}} blocked reactor thread for 62887.6us
I20251025 07:52:29.449766 894 tablet_service.cc:1505] Processing CreateTablet for tablet 36c8fcafd5464b4e999276a8062a812a (DEFAULT_TABLE table=default.Table_19a3326e4da345c1bbeb2cf083d28e24 [id=11c088570f6940b1895d1691465c10b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.451076 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 5caac9c943bf42a5b2ae522d87fdfe8e (table default.Table_60bfcc345b8649e9bc73fe3d077f46c7 [id=ed06dbbb96a24ab39619f4186ba93191]) successfully deleted
W20251025 07:52:29.463798 847 leader_election.cc:343] T 1b31177a062b467187d68ed2034f5690 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:29.464159 847 leader_election.cc:304] T 1b31177a062b467187d68ed2034f5690 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:29.465992 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30418858bf95438686bdfe64ac733144. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.467765 1218 raft_consensus.cc:2749] T 1b31177a062b467187d68ed2034f5690 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:29.467792 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 8956ad28b35d45f28c683ac055bba441
I20251025 07:52:29.471621 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36c8fcafd5464b4e999276a8062a812a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.474282 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_278dac4ad5644752862d2de5d4f0b9b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:29.474923 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_278dac4ad5644752862d2de5d4f0b9b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:29.478480 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 8956ad28b35d45f28c683ac055bba441 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:29.480608 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 5fca357d7e24494fb6af4bd73d98833c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:29.481370 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66f0d398e59a4175aa9fe90715124d1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.483635 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dcab53a020847a1aa27930265ba0205. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:29.485539 847 leader_election.cc:343] T 1483c40eee834f20a92e6b4cd89775b6 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:29.485842 1166 tablet_bootstrap.cc:492] T 8c6ff4c73e624af3bf2de36c4493b462 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:29.486716 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet bef0ef23d90d480d9c99aef21d7271fd
I20251025 07:52:29.487726 1170 tablet_replica.cc:333] T dd898450f2604987b08639cf87d0e402 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:29.488193 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5caac9c943bf42a5b2ae522d87fdfe8e" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:29.489598 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f51ec427dd9349a48eb221db04cf242b" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:29.490453 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26c849d376d143c6bef0c8e0e1d7d03a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.491068 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet bef0ef23d90d480d9c99aef21d7271fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:29.492623 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_22ddca5b9e8449ef9fbdeb29212ff9a5" } new_table_name: "default.Table_8d39f75f88854f528f5cac09c3b0423a" modify_external_catalogs: true
I20251025 07:52:29.501717 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 4037dee385d04e85bb78c8e77cb00379
I20251025 07:52:29.502424 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09898735796541b0a8f3be22cdb6cc30. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.503510 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 050a5d3b5f934dd19bf2e221f2fe0780. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.506884 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 87a93778620f4c84b3b3f7a3d7d7e87f (table default.Table_0eca32ca64da44198140ae7ec9a77767 [id=2da58c150b514203abed5e20ffa98919]) successfully deleted
W20251025 07:52:29.507347 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet e54fe7cbcffa40c4b04144b47bee2b48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e54fe7cbcffa40c4b04144b47bee2b48 already in progress: creating tablet
W20251025 07:52:29.508062 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 27651312cc2e42a8bfcd245f5d301679 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27651312cc2e42a8bfcd245f5d301679 already in progress: creating tablet
W20251025 07:52:29.508455 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 887e7c7a5d0e4aaf88f26246306df19c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 887e7c7a5d0e4aaf88f26246306df19c already in progress: creating tablet
W20251025 07:52:29.508924 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 742b3ec3f6604647b9e397532f1673dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 742b3ec3f6604647b9e397532f1673dd already in progress: creating tablet
I20251025 07:52:29.511529 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 4037dee385d04e85bb78c8e77cb00379 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:29.510345 847 leader_election.cc:304] T 1483c40eee834f20a92e6b4cd89775b6 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:29.514781 1247 raft_consensus.cc:697] T dd898450f2604987b08639cf87d0e402 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:29.512986 1166 ts_tablet_manager.cc:1403] T 8c6ff4c73e624af3bf2de36c4493b462 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.462s user 0.021s sys 0.037s
I20251025 07:52:29.515193 1247 consensus_queue.cc:237] T dd898450f2604987b08639cf87d0e402 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.516626 1166 raft_consensus.cc:359] T 8c6ff4c73e624af3bf2de36c4493b462 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.518268 1169 tablet_bootstrap.cc:492] T f51ec427dd9349a48eb221db04cf242b P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:29.518455 1169 ts_tablet_manager.cc:1403] T f51ec427dd9349a48eb221db04cf242b P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.240s user 0.007s sys 0.002s
I20251025 07:52:29.519279 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_5261ed9b1f29420eab473df75f903987"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:29.519375 1169 raft_consensus.cc:359] T f51ec427dd9349a48eb221db04cf242b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.519565 1169 raft_consensus.cc:385] T f51ec427dd9349a48eb221db04cf242b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:29.519678 1169 raft_consensus.cc:740] T f51ec427dd9349a48eb221db04cf242b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
W20251025 07:52:29.519955 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5261ed9b1f29420eab473df75f903987 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:29.519930 1169 consensus_queue.cc:260] T f51ec427dd9349a48eb221db04cf242b P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.520562 1169 ts_tablet_manager.cc:1434] T f51ec427dd9349a48eb221db04cf242b P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 07:52:29.520968 1169 tablet_bootstrap.cc:492] T 1b31177a062b467187d68ed2034f5690 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:29.527200 1444 raft_consensus.cc:2749] T 1483c40eee834f20a92e6b4cd89775b6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:29.529989 753 tablet_service.cc:1505] Processing CreateTablet for tablet 0551e5d694894e8fbeb66b5e0c1cd75b (DEFAULT_TABLE table=default.Table_2cd70245aa3d4809a0647e7fa499f62f [id=a003268ec3da425989fa4041699de4d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.530277 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1912e59548874e0e8e989050ec0064c5. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:29.534303 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:46231, user_credentials={real_user=slave}} blocked reactor thread for 45146us
I20251025 07:52:29.536124 1167 tablet_bootstrap.cc:654] T cf2cab1552bd4e7394797c93b149f450 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:29.539201 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f661d5909657455581dcb5e6eadd3c0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.539466 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09898735796541b0a8f3be22cdb6cc30. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.539808 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0551e5d694894e8fbeb66b5e0c1cd75b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.542837 1169 tablet_bootstrap.cc:654] T 1b31177a062b467187d68ed2034f5690 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:29.544298 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_646f48a21c194af3a1611e5e12b2e11b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:29.544961 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_646f48a21c194af3a1611e5e12b2e11b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:29.546630 1169 tablet_bootstrap.cc:492] T 1b31177a062b467187d68ed2034f5690 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:29.546803 1169 ts_tablet_manager.cc:1403] T 1b31177a062b467187d68ed2034f5690 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.003s
I20251025 07:52:29.547569 1169 raft_consensus.cc:359] T 1b31177a062b467187d68ed2034f5690 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.547788 1169 raft_consensus.cc:385] T 1b31177a062b467187d68ed2034f5690 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:29.547868 1169 raft_consensus.cc:740] T 1b31177a062b467187d68ed2034f5690 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:29.549072 1169 consensus_queue.cc:260] T 1b31177a062b467187d68ed2034f5690 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.549666 1169 ts_tablet_manager.cc:1434] T 1b31177a062b467187d68ed2034f5690 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251025 07:52:29.550055 1169 tablet_bootstrap.cc:492] T 87a93778620f4c84b3b3f7a3d7d7e87f P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:29.555559 1169 tablet_bootstrap.cc:654] T 87a93778620f4c84b3b3f7a3d7d7e87f P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:29.559964 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_19354444047b451bbf22bafcf9ae185d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:29.560586 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19354444047b451bbf22bafcf9ae185d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:29.564052 744 tablet_service.cc:1505] Processing CreateTablet for tablet c9f7ba3709c745d4a625449d6e1c8ba1 (DEFAULT_TABLE table=default.Table_380ca630f4144f40a04084d2ef8600e3 [id=06b6a39af6764e44972d57ddd722f838]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.571900 755 tablet_service.cc:1505] Processing CreateTablet for tablet b2044ca206634eb285197af4ba2073e7 (DEFAULT_TABLE table=default.Table_3292d47465544af0970067fb567e7ede [id=41fc0b99786749259c0a0059dec97ea1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.572330 517 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 14aef538520d4a3aba9d6e9ac30a618d
I20251025 07:52:29.573973 737 tablet_service.cc:1505] Processing CreateTablet for tablet 36c8fcafd5464b4e999276a8062a812a (DEFAULT_TABLE table=default.Table_19a3326e4da345c1bbeb2cf083d28e24 [id=11c088570f6940b1895d1691465c10b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.575764 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 5fca357d7e24494fb6af4bd73d98833c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:29.576062 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 8956ad28b35d45f28c683ac055bba441 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:29.576319 756 tablet_service.cc:1552] Processing DeleteTablet for tablet bef0ef23d90d480d9c99aef21d7271fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:29.576555 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 4037dee385d04e85bb78c8e77cb00379 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:29.578500 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_9f0ae2442a334cf8ab13f2eba387ebf3" } new_table_name: "default.Table_5344c3d93df647aba4baf40f77b9191b" modify_external_catalogs: true
I20251025 07:52:29.589398 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_3b10c71d2c1b4bd2849e0cad9ec767d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:29.590057 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b10c71d2c1b4bd2849e0cad9ec767d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:29.594066 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 1b31177a062b467187d68ed2034f5690 with delete_type TABLET_DATA_DELETED (replaced by tablet b772025852794a49b072f37cccbd3291) from {username='slave'} at 127.0.0.1:47408
W20251025 07:52:29.598160 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b17d2a9ad6be49d29f2413666fba7e69 consensus state has no leader
I20251025 07:52:29.608184 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_5f94a1aa9fac44ae8a90d3554215d598" } new_table_name: "default.Table_dfc7f1d4bf8b48bebbb00313419c3fd8" modify_external_catalogs: true
I20251025 07:52:29.613312 1170 raft_consensus.cc:2243] T dd898450f2604987b08639cf87d0e402 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:29.613826 1170 pending_rounds.cc:70] T dd898450f2604987b08639cf87d0e402 P db42be7ad3c741a0aa427f27e245993b: Trying to abort 1 pending ops.
I20251025 07:52:29.613902 1170 pending_rounds.cc:77] T dd898450f2604987b08639cf87d0e402 P db42be7ad3c741a0aa427f27e245993b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214607358017073152 op_type: NO_OP noop_request { }
I20251025 07:52:29.614046 1170 raft_consensus.cc:2889] T dd898450f2604987b08639cf87d0e402 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 07:52:29.614228 1170 raft_consensus.cc:2272] T dd898450f2604987b08639cf87d0e402 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:29.616026 1433 raft_consensus.cc:493] T b00153c97104470883f6089d71d23708 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:29.616204 1433 raft_consensus.cc:515] T b00153c97104470883f6089d71d23708 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.616902 1433 leader_election.cc:290] T b00153c97104470883f6089d71d23708 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:29.618920 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_0f4ba264122a4aad989cf6694f0e7ccb" } new_table_name: "default.Table_32f8a762ec25446dac28e2729d6879bf" modify_external_catalogs: true
W20251025 07:52:29.621348 712 leader_election.cc:343] T b00153c97104470883f6089d71d23708 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: b00153c97104470883f6089d71d23708
I20251025 07:52:29.621767 1170 ts_tablet_manager.cc:1916] T dd898450f2604987b08639cf87d0e402 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:29.621932 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9f7ba3709c745d4a625449d6e1c8ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.622344 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36c8fcafd5464b4e999276a8062a812a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.625208 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2044ca206634eb285197af4ba2073e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.631434 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet f51ec427dd9349a48eb221db04cf242b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:29.641413 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_77761ef857464942be15bf6e67127d6d" } modify_external_catalogs: true
I20251025 07:52:29.641638 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_77761ef857464942be15bf6e67127d6d" } modify_external_catalogs: true
I20251025 07:52:29.645274 1167 tablet_bootstrap.cc:492] T cf2cab1552bd4e7394797c93b149f450 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:29.648389 1167 ts_tablet_manager.cc:1403] T cf2cab1552bd4e7394797c93b149f450 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.429s user 0.001s sys 0.007s
I20251025 07:52:29.648425 1170 ts_tablet_manager.cc:1929] T dd898450f2604987b08639cf87d0e402 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:29.648576 1170 log.cc:1199] T dd898450f2604987b08639cf87d0e402 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/dd898450f2604987b08639cf87d0e402
I20251025 07:52:29.649025 1170 ts_tablet_manager.cc:1950] T dd898450f2604987b08639cf87d0e402 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:29.649504 1167 raft_consensus.cc:359] T cf2cab1552bd4e7394797c93b149f450 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.649808 1167 raft_consensus.cc:385] T cf2cab1552bd4e7394797c93b149f450 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:29.649937 1167 raft_consensus.cc:740] T cf2cab1552bd4e7394797c93b149f450 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:29.650164 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet dd898450f2604987b08639cf87d0e402 (table default.Table_56f5f8c6d70645309e7ea14dcf847a43 [id=2dd014b2a3a94681967660e0399d334a]) successfully deleted
I20251025 07:52:29.650256 1167 consensus_queue.cc:260] T cf2cab1552bd4e7394797c93b149f450 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.651099 1167 ts_tablet_manager.cc:1434] T cf2cab1552bd4e7394797c93b149f450 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251025 07:52:29.651463 1167 tablet_bootstrap.cc:492] T 198db9a45257435f86df2605992ccb34 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:29.653209 1167 tablet_bootstrap.cc:654] T 198db9a45257435f86df2605992ccb34 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:29.654307 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:43563, user_credentials={real_user=slave}} blocked reactor thread for 459103us
W20251025 07:52:29.654623 710 leader_election.cc:343] T 3c3eb43209f24592820638fc25486b78 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 3c3eb43209f24592820638fc25486b78
I20251025 07:52:29.654836 710 leader_election.cc:304] T 3c3eb43209f24592820638fc25486b78 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:29.655259 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1912e59548874e0e8e989050ec0064c5 consensus state has no leader
I20251025 07:52:29.655272 1433 raft_consensus.cc:2749] T 3c3eb43209f24592820638fc25486b78 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:29.659195 1247 raft_consensus.cc:493] T f51ec427dd9349a48eb221db04cf242b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:29.659482 1247 raft_consensus.cc:515] T f51ec427dd9349a48eb221db04cf242b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.660514 1247 leader_election.cc:290] T f51ec427dd9349a48eb221db04cf242b P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:29.663942 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f51ec427dd9349a48eb221db04cf242b" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
W20251025 07:52:29.670409 982 leader_election.cc:343] T f51ec427dd9349a48eb221db04cf242b P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:29.672077 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_8faecb0cb37949a39f4ef5d9d2740c3d" } modify_external_catalogs: true
I20251025 07:52:29.672451 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_8faecb0cb37949a39f4ef5d9d2740c3d" } modify_external_catalogs: true
I20251025 07:52:29.687034 521 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 50ffbcdd4a9247bf99b8e2361bc02ef5 (table default.Table_9f0ae2442a334cf8ab13f2eba387ebf3 [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 3 got 0
I20251025 07:52:29.689165 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_e1d698cd782e4d58b29e9cc7d1f56703" } modify_external_catalogs: true
I20251025 07:52:29.689350 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_e1d698cd782e4d58b29e9cc7d1f56703" } modify_external_catalogs: true
I20251025 07:52:29.709411 1018 tablet_service.cc:1505] Processing CreateTablet for tablet ce1ebb1ec14b4801930a72765bca1a7d (DEFAULT_TABLE table=default.Table_d5abcbf8c8c741a2b40880a3467f2bcd [id=eab2657210e74ee6ac6f59544c9ba12e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.709896 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce1ebb1ec14b4801930a72765bca1a7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.710080 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_e50bf47b78d94bbc811e6f3fb3551edb" } modify_external_catalogs: true
I20251025 07:52:29.710274 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_e50bf47b78d94bbc811e6f3fb3551edb" } modify_external_catalogs: true
I20251025 07:52:29.710510 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 91728987303b41fd840eb56f3a311c5f (DEFAULT_TABLE table=default.Table_b53575428f5b4c25a5c912428e5d4f2d [id=88b6e7cfe0c74d169585a1b064d58e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.710876 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91728987303b41fd840eb56f3a311c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.713653 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 4678f7eab5e8497db308e8a38e387618 (DEFAULT_TABLE table=default.Table_bc1398c7c53846da95b511e6a442e416 [id=10e2dcf046e241a9b881adabc51defbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.714020 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4678f7eab5e8497db308e8a38e387618. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.718411 1019 tablet_service.cc:1505] Processing CreateTablet for tablet ebc824ec4c60411dab1a2ea4a385148e (DEFAULT_TABLE table=default.Table_c43ac1adc76d4ccf819411ec76775932 [id=106e3728b0b846178eb48ccbf076e24e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.718472 1020 tablet_service.cc:1505] Processing CreateTablet for tablet f656945454fe4921ab819c772a089eb3 (DEFAULT_TABLE table=default.Table_7ce7627f71134d5c989f9ba5c6766a2a [id=b5bf17121a4e4e3a8f21dccf1dba4a99]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.718937 1021 tablet_service.cc:1505] Processing CreateTablet for tablet bdaeedbcf35a4aad97f9ef192991978a (DEFAULT_TABLE table=default.Table_cd12897d1fc5454ab07cdc13fefb6978 [id=1655781ab7a046ad8e6dcfacaf2db1b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.719110 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 11bfe0c87ca941a3846462c8a4512d83 (DEFAULT_TABLE table=default.Table_00b3b394e5ed46faaac0202afe54b437 [id=304ef5bba5ed4e198622b8c8dc17f43a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.719390 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebc824ec4c60411dab1a2ea4a385148e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.719643 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdaeedbcf35a4aad97f9ef192991978a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.719954 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f656945454fe4921ab819c772a089eb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.723667 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11bfe0c87ca941a3846462c8a4512d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.730479 1027 tablet_service.cc:1505] Processing CreateTablet for tablet a77eb82f26ae43b48de1e85693372d34 (DEFAULT_TABLE table=default.Table_bcd5afa1072f47898d67504fc72b84c4 [id=3d6f0426d32f481ba8152eaacb298784]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.730875 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a77eb82f26ae43b48de1e85693372d34. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.732398 1471 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:37611 every 8 connection(s)
I20251025 07:52:29.732781 1224 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:37611
I20251025 07:52:29.733765 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_3709b01b0a9a48499f5bd229e3e9f31a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:29.734434 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3709b01b0a9a48499f5bd229e3e9f31a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:29.736565 1224 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-2/data/info.pb
I20251025 07:52:29.744395 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRBHlYt/build/asan/bin/kudu as pid 1224
W20251025 07:52:29.744628 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet b947ecc352e54f4caa73c6ca97372256 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b947ecc352e54f4caa73c6ca97372256 already in progress: creating tablet
W20251025 07:52:29.745313 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 80fe3255ed614270bd8352a2853fbc43 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80fe3255ed614270bd8352a2853fbc43 already in progress: creating tablet
I20251025 07:52:29.745628 1170 tablet_replica.cc:333] T 1483c40eee834f20a92e6b4cd89775b6 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:29.757201 1170 raft_consensus.cc:2243] T 1483c40eee834f20a92e6b4cd89775b6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:29.757930 1170 raft_consensus.cc:2272] T 1483c40eee834f20a92e6b4cd89775b6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:29.761432 1170 ts_tablet_manager.cc:1916] T 1483c40eee834f20a92e6b4cd89775b6 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:29.764796 1472 sys_catalog.cc:263] Verifying existing consensus state
I20251025 07:52:29.765585 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_ba46ca0cf5d941e0879dead1c3544190"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:29.766299 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba46ca0cf5d941e0879dead1c3544190 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:29.774710 1166 raft_consensus.cc:385] T 8c6ff4c73e624af3bf2de36c4493b462 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251025 07:52:29.774801 847 leader_election.cc:343] T a231f3d31b204f7b82f018892ccc8435 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: a231f3d31b204f7b82f018892ccc8435
I20251025 07:52:29.777132 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f51ec427dd9349a48eb221db04cf242b" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:29.777374 914 raft_consensus.cc:2468] T f51ec427dd9349a48eb221db04cf242b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:29.778406 983 leader_election.cc:304] T f51ec427dd9349a48eb221db04cf242b P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b
I20251025 07:52:29.778975 1247 raft_consensus.cc:2804] T f51ec427dd9349a48eb221db04cf242b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:29.779088 1247 raft_consensus.cc:493] T f51ec427dd9349a48eb221db04cf242b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:29.779207 1247 raft_consensus.cc:3060] T f51ec427dd9349a48eb221db04cf242b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:29.781272 741 tablet_service.cc:1505] Processing CreateTablet for tablet 91728987303b41fd840eb56f3a311c5f (DEFAULT_TABLE table=default.Table_b53575428f5b4c25a5c912428e5d4f2d [id=88b6e7cfe0c74d169585a1b064d58e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.781924 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91728987303b41fd840eb56f3a311c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.783211 1247 raft_consensus.cc:515] T f51ec427dd9349a48eb221db04cf242b P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.784096 1247 leader_election.cc:290] T f51ec427dd9349a48eb221db04cf242b P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:29.784715 517 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 50ffbcdd4a9247bf99b8e2361bc02ef5 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 4 got 0
I20251025 07:52:29.785847 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_b53575428f5b4c25a5c912428e5d4f2d" } modify_external_catalogs: true
I20251025 07:52:29.786019 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_b53575428f5b4c25a5c912428e5d4f2d" } modify_external_catalogs: true
I20251025 07:52:29.785943 847 leader_election.cc:304] T a231f3d31b204f7b82f018892ccc8435 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:29.786346 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cd23b8094d7e472eba6bd170248858a9 consensus state has no leader
W20251025 07:52:29.786477 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:46231, user_credentials={real_user=slave}} blocked reactor thread for 84604.8us
I20251025 07:52:29.786496 1444 raft_consensus.cc:2749] T a231f3d31b204f7b82f018892ccc8435 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:29.786926 847 leader_election.cc:343] T 5caac9c943bf42a5b2ae522d87fdfe8e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:29.787055 847 leader_election.cc:304] T 5caac9c943bf42a5b2ae522d87fdfe8e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:29.787297 847 raft_consensus.cc:2686] T 5caac9c943bf42a5b2ae522d87fdfe8e P 3aea2869ff814cc7b142012faa274382: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251025 07:52:29.787635 847 leader_election.cc:343] T f51ec427dd9349a48eb221db04cf242b P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:29.787786 847 leader_election.cc:304] T f51ec427dd9349a48eb221db04cf242b P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:29.788625 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_c1bf6bb7c62c4648853b9a5659c73da0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:29.789209 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1bf6bb7c62c4648853b9a5659c73da0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:29.793144 1166 raft_consensus.cc:740] T 8c6ff4c73e624af3bf2de36c4493b462 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:29.797034 1472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407: Bootstrap starting.
I20251025 07:52:29.799809 876 tablet_service.cc:1552] Processing DeleteTablet for tablet 8956ad28b35d45f28c683ac055bba441 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:29.800953 886 tablet_service.cc:1552] Processing DeleteTablet for tablet bef0ef23d90d480d9c99aef21d7271fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:29.801088 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 5fca357d7e24494fb6af4bd73d98833c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:29.802820 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f51ec427dd9349a48eb221db04cf242b" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:29.805419 1166 consensus_queue.cc:260] T 8c6ff4c73e624af3bf2de36c4493b462 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.805971 1166 ts_tablet_manager.cc:1434] T 8c6ff4c73e624af3bf2de36c4493b462 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.291s user 0.003s sys 0.000s
I20251025 07:52:29.808739 1169 tablet_bootstrap.cc:492] T 87a93778620f4c84b3b3f7a3d7d7e87f P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:29.809909 1169 ts_tablet_manager.cc:1403] T 87a93778620f4c84b3b3f7a3d7d7e87f P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.260s user 0.029s sys 0.059s
I20251025 07:52:29.810762 1169 raft_consensus.cc:359] T 87a93778620f4c84b3b3f7a3d7d7e87f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.810985 1169 raft_consensus.cc:385] T 87a93778620f4c84b3b3f7a3d7d7e87f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:29.811084 1169 raft_consensus.cc:740] T 87a93778620f4c84b3b3f7a3d7d7e87f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:29.811323 1169 consensus_queue.cc:260] T 87a93778620f4c84b3b3f7a3d7d7e87f P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.812744 1169 ts_tablet_manager.cc:1434] T 87a93778620f4c84b3b3f7a3d7d7e87f P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251025 07:52:29.812904 1444 raft_consensus.cc:2749] T f51ec427dd9349a48eb221db04cf242b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:29.813061 1169 tablet_bootstrap.cc:492] T cd23b8094d7e472eba6bd170248858a9 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:29.813989 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 4037dee385d04e85bb78c8e77cb00379 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:29.814832 890 tablet_service.cc:1505] Processing CreateTablet for tablet ebc824ec4c60411dab1a2ea4a385148e (DEFAULT_TABLE table=default.Table_c43ac1adc76d4ccf819411ec76775932 [id=106e3728b0b846178eb48ccbf076e24e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.815441 1169 tablet_bootstrap.cc:654] T cd23b8094d7e472eba6bd170248858a9 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:29.815508 1166 tablet_bootstrap.cc:492] T cd23b8094d7e472eba6bd170248858a9 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:29.818790 1166 tablet_bootstrap.cc:654] T cd23b8094d7e472eba6bd170248858a9 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:29.819972 914 raft_consensus.cc:3060] T f51ec427dd9349a48eb221db04cf242b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:29.821434 754 tablet_service.cc:1505] Processing CreateTablet for tablet bdaeedbcf35a4aad97f9ef192991978a (DEFAULT_TABLE table=default.Table_cd12897d1fc5454ab07cdc13fefb6978 [id=1655781ab7a046ad8e6dcfacaf2db1b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.821710 742 tablet_service.cc:1505] Processing CreateTablet for tablet 11bfe0c87ca941a3846462c8a4512d83 (DEFAULT_TABLE table=default.Table_00b3b394e5ed46faaac0202afe54b437 [id=304ef5bba5ed4e198622b8c8dc17f43a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.822940 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebc824ec4c60411dab1a2ea4a385148e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.823282 740 tablet_service.cc:1505] Processing CreateTablet for tablet ce1ebb1ec14b4801930a72765bca1a7d (DEFAULT_TABLE table=default.Table_d5abcbf8c8c741a2b40880a3467f2bcd [id=eab2657210e74ee6ac6f59544c9ba12e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.823483 892 tablet_service.cc:1505] Processing CreateTablet for tablet a77eb82f26ae43b48de1e85693372d34 (DEFAULT_TABLE table=default.Table_bcd5afa1072f47898d67504fc72b84c4 [id=3d6f0426d32f481ba8152eaacb298784]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.827183 1169 tablet_bootstrap.cc:492] T cd23b8094d7e472eba6bd170248858a9 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:29.827374 1169 ts_tablet_manager.cc:1403] T cd23b8094d7e472eba6bd170248858a9 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.004s
I20251025 07:52:29.828275 1169 raft_consensus.cc:359] T cd23b8094d7e472eba6bd170248858a9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.828521 1169 raft_consensus.cc:385] T cd23b8094d7e472eba6bd170248858a9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:29.828609 1169 raft_consensus.cc:740] T cd23b8094d7e472eba6bd170248858a9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:29.828943 1169 consensus_queue.cc:260] T cd23b8094d7e472eba6bd170248858a9 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.831234 1169 ts_tablet_manager.cc:1434] T cd23b8094d7e472eba6bd170248858a9 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251025 07:52:29.831537 1169 tablet_bootstrap.cc:492] T 1aeaad5c5d01478bb997254ac372d9a6 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:29.832475 739 tablet_service.cc:1505] Processing CreateTablet for tablet 4678f7eab5e8497db308e8a38e387618 (DEFAULT_TABLE table=default.Table_bc1398c7c53846da95b511e6a442e416 [id=10e2dcf046e241a9b881adabc51defbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.833473 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a77eb82f26ae43b48de1e85693372d34. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.833619 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f51ec427dd9349a48eb221db04cf242b" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:29.834858 1169 tablet_bootstrap.cc:654] T 1aeaad5c5d01478bb997254ac372d9a6 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:29.834884 982 leader_election.cc:343] T f51ec427dd9349a48eb221db04cf242b P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:29.835415 880 tablet_service.cc:1505] Processing CreateTablet for tablet 91728987303b41fd840eb56f3a311c5f (DEFAULT_TABLE table=default.Table_b53575428f5b4c25a5c912428e5d4f2d [id=88b6e7cfe0c74d169585a1b064d58e4a]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:29.838989 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5b6383c99e9e4c90a374da9ccd830a6f has no consensus state
I20251025 07:52:29.839272 520 master_service.cc:954] ReplaceTablet: received request to replace tablet a6ddefa1738f4ff98586900c344359d2 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:29.839679 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 53198283718d4b45b30df65b5bde96a6
I20251025 07:52:29.839610 1472 log.cc:826] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407: Log is configured to *not* fsync() on all Append() calls
I20251025 07:52:29.840600 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet be5ab0db6ac34c2c9a370445e251eb46
I20251025 07:52:29.840579 891 tablet_service.cc:1505] Processing CreateTablet for tablet ce1ebb1ec14b4801930a72765bca1a7d (DEFAULT_TABLE table=default.Table_d5abcbf8c8c741a2b40880a3467f2bcd [id=eab2657210e74ee6ac6f59544c9ba12e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.841202 522 master_service.cc:954] ReplaceTablet: received request to replace tablet 5caac9c943bf42a5b2ae522d87fdfe8e from {username='slave'} at 127.0.0.1:45978
W20251025 07:52:29.841823 1462 log.cc:927] Time spent T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 23eeffd3de254ce18db5febfde971c6b: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20251025 07:52:29.843641 752 tablet_service.cc:1505] Processing CreateTablet for tablet f656945454fe4921ab819c772a089eb3 (DEFAULT_TABLE table=default.Table_7ce7627f71134d5c989f9ba5c6766a2a [id=b5bf17121a4e4e3a8f21dccf1dba4a99]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.844987 750 tablet_service.cc:1552] Processing DeleteTablet for tablet be5ab0db6ac34c2c9a370445e251eb46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:29.847558 894 tablet_service.cc:1505] Processing CreateTablet for tablet 11bfe0c87ca941a3846462c8a4512d83 (DEFAULT_TABLE table=default.Table_00b3b394e5ed46faaac0202afe54b437 [id=304ef5bba5ed4e198622b8c8dc17f43a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.853929 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet be5ab0db6ac34c2c9a370445e251eb46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:29.854295 881 tablet_service.cc:1505] Processing CreateTablet for tablet 4678f7eab5e8497db308e8a38e387618 (DEFAULT_TABLE table=default.Table_bc1398c7c53846da95b511e6a442e416 [id=10e2dcf046e241a9b881adabc51defbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.854787 743 tablet_service.cc:1552] Processing DeleteTablet for tablet cf2cab1552bd4e7394797c93b149f450 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:29.856227 749 tablet_service.cc:1505] Processing CreateTablet for tablet a77eb82f26ae43b48de1e85693372d34 (DEFAULT_TABLE table=default.Table_bcd5afa1072f47898d67504fc72b84c4 [id=3d6f0426d32f481ba8152eaacb298784]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.857270 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdaeedbcf35a4aad97f9ef192991978a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.858255 886 tablet_service.cc:1505] Processing CreateTablet for tablet f656945454fe4921ab819c772a089eb3 (DEFAULT_TABLE table=default.Table_7ce7627f71134d5c989f9ba5c6766a2a [id=b5bf17121a4e4e3a8f21dccf1dba4a99]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.860046 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f656945454fe4921ab819c772a089eb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.862394 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4678f7eab5e8497db308e8a38e387618. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.863273 1172 ts_tablet_manager.cc:1929] T 5caac9c943bf42a5b2ae522d87fdfe8e P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:29.863445 1172 log.cc:1199] T 5caac9c943bf42a5b2ae522d87fdfe8e P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/5caac9c943bf42a5b2ae522d87fdfe8e
I20251025 07:52:29.863915 1172 ts_tablet_manager.cc:1950] T 5caac9c943bf42a5b2ae522d87fdfe8e P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:29.864523 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 53198283718d4b45b30df65b5bde96a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:29.866106 914 raft_consensus.cc:2468] T f51ec427dd9349a48eb221db04cf242b P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:29.866852 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91728987303b41fd840eb56f3a311c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.866883 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a77eb82f26ae43b48de1e85693372d34. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.873111 885 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56590 (request call id 242) took 1232 ms. Trace:
I20251025 07:52:29.873237 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_7f5a5345567a48d09dedce1df9d63a3b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:29.873854 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f5a5345567a48d09dedce1df9d63a3b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:29.874536 885 rpcz_store.cc:276] 1025 07:52:28.640832 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:28.903980 (+263148us) service_pool.cc:225] Handling call
1025 07:52:29.099169 (+195189us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 07:52:29.099201 (+ 32us) ts_tablet_manager.cc:682] Creating new metadata...
1025 07:52:29.500391 (+401190us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:29.849309 (+348918us) tablet_replica.cc:206] Creating consensus instance
1025 07:52:29.873071 (+ 23762us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 07:52:29.875258 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_278dac4ad5644752862d2de5d4f0b9b9" } new_table_name: "default.Table_bef97bb8f0a442f3a23cb20cbf85d934" modify_external_catalogs: true
I20251025 07:52:29.877326 983 leader_election.cc:304] T f51ec427dd9349a48eb221db04cf242b P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b
I20251025 07:52:29.877575 525 master_service.cc:954] ReplaceTablet: received request to replace tablet 71432a6848944a0da06ef9b81f160dc4 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:29.877909 1247 raft_consensus.cc:2804] T f51ec427dd9349a48eb221db04cf242b P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:29.878087 1247 raft_consensus.cc:697] T f51ec427dd9349a48eb221db04cf242b P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:29.878422 1247 consensus_queue.cc:237] T f51ec427dd9349a48eb221db04cf242b P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.878089 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_550c910e06364276ae5c4548afa39e6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:29.881821 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_550c910e06364276ae5c4548afa39e6c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:29.881865 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11bfe0c87ca941a3846462c8a4512d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.884053 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 53198283718d4b45b30df65b5bde96a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:29.884589 755 tablet_service.cc:1505] Processing CreateTablet for tablet ebc824ec4c60411dab1a2ea4a385148e (DEFAULT_TABLE table=default.Table_c43ac1adc76d4ccf819411ec76775932 [id=106e3728b0b846178eb48ccbf076e24e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.886327 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 87a93778620f4c84b3b3f7a3d7d7e87f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:29.886921 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11bfe0c87ca941a3846462c8a4512d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.888756 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f656945454fe4921ab819c772a089eb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.889585 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4678f7eab5e8497db308e8a38e387618. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.901163 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet db99a1e68daf4e948a3d66ad276cc5f9
I20251025 07:52:29.902159 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet f97acc9a638b4d5a968dcc3412a384bc
I20251025 07:52:29.906864 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_5eaf42285e474860a44c19683f1fa70f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:29.907444 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5eaf42285e474860a44c19683f1fa70f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:29.907773 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet f97acc9a638b4d5a968dcc3412a384bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:29.908633 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_97c3fbaf37cd4533bb3f4d8f55a0f945" } new_table_name: "default.Table_56bc5dd6e81946b780e4a18414de4c58" modify_external_catalogs: true
I20251025 07:52:29.909001 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce1ebb1ec14b4801930a72765bca1a7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.909865 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_8da73d66704143de8a4b4480c28fd1d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:29.910418 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8da73d66704143de8a4b4480c28fd1d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:29.911230 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebc824ec4c60411dab1a2ea4a385148e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.912879 1171 ts_tablet_manager.cc:1929] T 50433881c0eb4aa5a389b6f5b4a4337a P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:29.914271 1171 log.cc:1199] T 50433881c0eb4aa5a389b6f5b4a4337a P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/50433881c0eb4aa5a389b6f5b4a4337a
I20251025 07:52:29.914619 748 tablet_service.cc:1552] Processing DeleteTablet for tablet f97acc9a638b4d5a968dcc3412a384bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:29.915083 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce1ebb1ec14b4801930a72765bca1a7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.916018 521 master_service.cc:954] ReplaceTablet: received request to replace tablet 2aa522477e934c89834026dc480fe7c0 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:29.916074 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet db99a1e68daf4e948a3d66ad276cc5f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:29.915601 1171 ts_tablet_manager.cc:1950] T 50433881c0eb4aa5a389b6f5b4a4337a P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:29.917496 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 254) took 1118 ms. Trace:
I20251025 07:52:29.917735 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 245) took 1397 ms. Trace:
I20251025 07:52:29.917650 1172 rpcz_store.cc:276] 1025 07:52:28.798741 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:28.798845 (+ 104us) service_pool.cc:225] Handling call
1025 07:52:29.396981 (+598136us) ts_tablet_manager.cc:1151] Deleting tablet 5caac9c943bf42a5b2ae522d87fdfe8e
1025 07:52:29.396989 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:29.844172 (+447183us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:29.863092 (+ 18920us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:29.917457 (+ 54365us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":596929}
I20251025 07:52:29.918108 1172 tablet_replica.cc:333] T dd898450f2604987b08639cf87d0e402 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:29.918020 1171 rpcz_store.cc:276] 1025 07:52:28.520184 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:28.526505 (+ 6321us) service_pool.cc:225] Handling call
1025 07:52:29.017500 (+490995us) ts_tablet_manager.cc:1151] Deleting tablet 50433881c0eb4aa5a389b6f5b4a4337a
1025 07:52:29.017513 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:29.693189 (+675676us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:29.880095 (+186906us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:29.917686 (+ 37591us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":312683}
I20251025 07:52:29.918390 1172 raft_consensus.cc:2243] T dd898450f2604987b08639cf87d0e402 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:29.918576 1172 raft_consensus.cc:2272] T dd898450f2604987b08639cf87d0e402 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:29.919229 1172 ts_tablet_manager.cc:1916] T dd898450f2604987b08639cf87d0e402 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:29.921062 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_f6781cf6e9104f51a278cab3d1781a1a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:29.921641 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f6781cf6e9104f51a278cab3d1781a1a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:29.925487 885 tablet_service.cc:1552] Processing DeleteTablet for tablet f97acc9a638b4d5a968dcc3412a384bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:29.925958 1444 consensus_queue.cc:1048] T 796413da731245459cc5dffacc3deb2f P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:29.926282 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 5caac9c943bf42a5b2ae522d87fdfe8e (table default.Table_60bfcc345b8649e9bc73fe3d077f46c7 [id=ed06dbbb96a24ab39619f4186ba93191]) successfully deleted
I20251025 07:52:29.929351 1218 consensus_queue.cc:1048] T 796413da731245459cc5dffacc3deb2f P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20251025 07:52:29.930418 1170 ts_tablet_manager.cc:1929] T 1483c40eee834f20a92e6b4cd89775b6 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:29.931102 875 tablet_service.cc:1552] Processing DeleteTablet for tablet 53198283718d4b45b30df65b5bde96a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:29.932247 884 tablet_service.cc:1552] Processing DeleteTablet for tablet be5ab0db6ac34c2c9a370445e251eb46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:29.933228 878 tablet_service.cc:1552] Processing DeleteTablet for tablet db99a1e68daf4e948a3d66ad276cc5f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:29.934434 882 tablet_service.cc:1505] Processing CreateTablet for tablet bdaeedbcf35a4aad97f9ef192991978a (DEFAULT_TABLE table=default.Table_cd12897d1fc5454ab07cdc13fefb6978 [id=1655781ab7a046ad8e6dcfacaf2db1b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:29.934962 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdaeedbcf35a4aad97f9ef192991978a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:29.941886 1170 log.cc:1199] T 1483c40eee834f20a92e6b4cd89775b6 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/1483c40eee834f20a92e6b4cd89775b6
I20251025 07:52:29.943768 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 91728987303b41fd840eb56f3a311c5f
I20251025 07:52:29.947134 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 91728987303b41fd840eb56f3a311c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:29.948729 1170 ts_tablet_manager.cc:1950] T 1483c40eee834f20a92e6b4cd89775b6 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:29.949888 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 1483c40eee834f20a92e6b4cd89775b6 (table default.Table_e0bf55e24b084fdbb62f09cce80b50c5 [id=084833e021c24b4a84dd1b1a3bb08dfc]) successfully deleted
I20251025 07:52:29.953325 1166 tablet_bootstrap.cc:492] T cd23b8094d7e472eba6bd170248858a9 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:29.953497 1166 ts_tablet_manager.cc:1403] T cd23b8094d7e472eba6bd170248858a9 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.147s user 0.003s sys 0.002s
I20251025 07:52:29.955109 1166 raft_consensus.cc:359] T cd23b8094d7e472eba6bd170248858a9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.955343 1166 raft_consensus.cc:385] T cd23b8094d7e472eba6bd170248858a9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:29.955437 1166 raft_consensus.cc:740] T cd23b8094d7e472eba6bd170248858a9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:29.955711 1166 consensus_queue.cc:260] T cd23b8094d7e472eba6bd170248858a9 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.960004 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_3353358c7aab4407880b823e1500b6d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:29.960593 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3353358c7aab4407880b823e1500b6d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:29.963384 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 8956ad28b35d45f28c683ac055bba441 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8956ad28b35d45f28c683ac055bba441 already in progress: creating tablet
W20251025 07:52:29.964134 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 5fca357d7e24494fb6af4bd73d98833c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5fca357d7e24494fb6af4bd73d98833c already in progress: creating tablet
I20251025 07:52:29.964263 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 50433881c0eb4aa5a389b6f5b4a4337a (table default.Table_b42187d20a7e4f8783a8d25acb249350 [id=a5c73cb1e5a349ca8675f4ba4cb8a82d]) successfully deleted
I20251025 07:52:29.918957 754 tablet_service.cc:1552] Processing DeleteTablet for tablet db99a1e68daf4e948a3d66ad276cc5f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:29.954159 1476 tablet.cc:1722] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 23eeffd3de254ce18db5febfde971c6b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251025 07:52:29.957188 1167 tablet_bootstrap.cc:492] T 198db9a45257435f86df2605992ccb34 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:29.963469 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 91728987303b41fd840eb56f3a311c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:29.965938 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 252) took 1280 ms. Trace:
W20251025 07:52:29.974326 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 37c7938e8c5e41f08506aff6747801c9 has no consensus state
W20251025 07:52:29.975389 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet bef0ef23d90d480d9c99aef21d7271fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bef0ef23d90d480d9c99aef21d7271fd already in progress: creating tablet
I20251025 07:52:29.976025 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 91728987303b41fd840eb56f3a311c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:29.976148 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_499a6f88d70a45018371ed3806882506" } new_table_name: "default.Table_e2adb17a63724d56b3ad83d8daa4c30c" modify_external_catalogs: true
I20251025 07:52:29.978964 1166 ts_tablet_manager.cc:1434] T cd23b8094d7e472eba6bd170248858a9 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.025s user 0.003s sys 0.000s
I20251025 07:52:29.979254 1166 tablet_bootstrap.cc:492] T 2aa522477e934c89834026dc480fe7c0 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
W20251025 07:52:29.979748 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 4037dee385d04e85bb78c8e77cb00379 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4037dee385d04e85bb78c8e77cb00379 already in progress: creating tablet
I20251025 07:52:29.980116 1170 tablet_replica.cc:333] T 1b31177a062b467187d68ed2034f5690 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:29.980948 1166 tablet_bootstrap.cc:654] T 2aa522477e934c89834026dc480fe7c0 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:29.967170 1167 ts_tablet_manager.cc:1403] T 198db9a45257435f86df2605992ccb34 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.314s user 0.000s sys 0.010s
I20251025 07:52:29.994112 1167 raft_consensus.cc:359] T 198db9a45257435f86df2605992ccb34 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:29.994529 1167 raft_consensus.cc:385] T 198db9a45257435f86df2605992ccb34 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:29.994692 1167 raft_consensus.cc:740] T 198db9a45257435f86df2605992ccb34 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
W20251025 07:52:29.983942 710 leader_election.cc:343] T b00153c97104470883f6089d71d23708 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: b00153c97104470883f6089d71d23708
I20251025 07:52:29.994948 710 leader_election.cc:304] T b00153c97104470883f6089d71d23708 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:29.995090 1167 consensus_queue.cc:260] T 198db9a45257435f86df2605992ccb34 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:29.995484 710 leader_election.cc:343] T 94408a3a90224645ba3cacccf5de33f1 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 94408a3a90224645ba3cacccf5de33f1
I20251025 07:52:29.995769 1433 raft_consensus.cc:2749] T b00153c97104470883f6089d71d23708 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:29.995822 710 leader_election.cc:304] T 94408a3a90224645ba3cacccf5de33f1 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:29.996052 1433 raft_consensus.cc:2749] T 94408a3a90224645ba3cacccf5de33f1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:29.996594 1167 ts_tablet_manager.cc:1434] T 198db9a45257435f86df2605992ccb34 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251025 07:52:29.996964 1167 tablet_bootstrap.cc:492] T 1b31177a062b467187d68ed2034f5690 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:29.998728 1167 tablet_bootstrap.cc:654] T 1b31177a062b467187d68ed2034f5690 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:30.002846 1171 rpcz_store.cc:276] 1025 07:52:28.685468 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:28.686273 (+ 805us) service_pool.cc:225] Handling call
1025 07:52:29.965756 (+1279483us) ts_tablet_manager.cc:1151] Deleting tablet 3abfd52327a04ba88625f391a8658fbb
1025 07:52:29.965766 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:29.965893 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1275718}
W20251025 07:52:30.003412 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 3abfd52327a04ba88625f391a8658fbb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3abfd52327a04ba88625f391a8658fbb already in progress: creating tablet
I20251025 07:52:30.003916 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 253) took 1318 ms. Trace:
I20251025 07:52:30.004081 1171 rpcz_store.cc:276] 1025 07:52:28.685587 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:28.686298 (+ 711us) service_pool.cc:225] Handling call
1025 07:52:30.003752 (+1317454us) ts_tablet_manager.cc:1151] Deleting tablet 87a93778620f4c84b3b3f7a3d7d7e87f
1025 07:52:30.003762 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:30.003877 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1313496}
W20251025 07:52:30.004832 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 87a93778620f4c84b3b3f7a3d7d7e87f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87a93778620f4c84b3b3f7a3d7d7e87f already in progress: creating tablet
I20251025 07:52:30.005367 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 261) took 1304 ms. Trace:
I20251025 07:52:30.005522 1171 rpcz_store.cc:276] 1025 07:52:28.701149 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:28.701197 (+ 48us) service_pool.cc:225] Handling call
1025 07:52:30.005111 (+1303914us) ts_tablet_manager.cc:1151] Deleting tablet 27651312cc2e42a8bfcd245f5d301679
1025 07:52:30.005121 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:30.005336 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1303725}
W20251025 07:52:30.005991 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 27651312cc2e42a8bfcd245f5d301679 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27651312cc2e42a8bfcd245f5d301679 already in progress: creating tablet
I20251025 07:52:30.006671 1171 tablet_replica.cc:333] T 94408a3a90224645ba3cacccf5de33f1 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:30.006994 1171 raft_consensus.cc:2243] T 94408a3a90224645ba3cacccf5de33f1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:30.010071 1170 raft_consensus.cc:2243] T 1b31177a062b467187d68ed2034f5690 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:30.011073 1170 raft_consensus.cc:2272] T 1b31177a062b467187d68ed2034f5690 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:30.012352 1170 ts_tablet_manager.cc:1916] T 1b31177a062b467187d68ed2034f5690 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:30.026682 1172 ts_tablet_manager.cc:1929] T dd898450f2604987b08639cf87d0e402 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:30.026870 1172 log.cc:1199] T dd898450f2604987b08639cf87d0e402 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/dd898450f2604987b08639cf87d0e402
I20251025 07:52:30.027251 1172 ts_tablet_manager.cc:1950] T dd898450f2604987b08639cf87d0e402 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:30.028170 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 256) took 1229 ms. Trace:
I20251025 07:52:30.028285 1172 rpcz_store.cc:276] 1025 07:52:28.799120 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:28.799328 (+ 208us) service_pool.cc:225] Handling call
1025 07:52:29.918035 (+1118707us) ts_tablet_manager.cc:1151] Deleting tablet dd898450f2604987b08639cf87d0e402
1025 07:52:29.918047 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:30.003696 (+ 85649us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:30.026504 (+ 22808us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:30.028117 (+ 1613us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1117831}
I20251025 07:52:30.028774 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet dd898450f2604987b08639cf87d0e402 (table default.Table_56f5f8c6d70645309e7ea14dcf847a43 [id=2dd014b2a3a94681967660e0399d334a]) successfully deleted
I20251025 07:52:30.031633 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet a6ddefa1738f4ff98586900c344359d2
I20251025 07:52:30.032716 520 catalog_manager.cc:6719] ReplaceTablet: tablet a6ddefa1738f4ff98586900c344359d2 deleted and replaced by tablet 4f0c4f0e8f014e59be9e46d2afe0be9d
I20251025 07:52:30.034634 1167 tablet_bootstrap.cc:492] T 1b31177a062b467187d68ed2034f5690 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
W20251025 07:52:30.034708 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ef1854d35a3047e3a9fbe0166ab1295a consensus state has no leader
I20251025 07:52:30.034849 1167 ts_tablet_manager.cc:1403] T 1b31177a062b467187d68ed2034f5690 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.002s
I20251025 07:52:30.035866 1167 raft_consensus.cc:359] T 1b31177a062b467187d68ed2034f5690 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.036125 1167 raft_consensus.cc:385] T 1b31177a062b467187d68ed2034f5690 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:30.036204 1167 raft_consensus.cc:740] T 1b31177a062b467187d68ed2034f5690 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:30.036489 1167 consensus_queue.cc:260] T 1b31177a062b467187d68ed2034f5690 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.036695 1172 tablet_replica.cc:333] T a231f3d31b204f7b82f018892ccc8435 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:30.036881 1172 raft_consensus.cc:2243] T a231f3d31b204f7b82f018892ccc8435 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:30.037025 1167 ts_tablet_manager.cc:1434] T 1b31177a062b467187d68ed2034f5690 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 07:52:30.037066 1172 raft_consensus.cc:2272] T a231f3d31b204f7b82f018892ccc8435 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:30.037382 1167 tablet_bootstrap.cc:492] T 79bc2678df8042e3994b0471afa2c304 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:30.037623 1172 ts_tablet_manager.cc:1916] T a231f3d31b204f7b82f018892ccc8435 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:30.040548 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet a6ddefa1738f4ff98586900c344359d2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f0c4f0e8f014e59be9e46d2afe0be9d) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:30.043165 749 tablet_service.cc:1552] Processing DeleteTablet for tablet a6ddefa1738f4ff98586900c344359d2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f0c4f0e8f014e59be9e46d2afe0be9d) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:30.044028 882 tablet_service.cc:1552] Processing DeleteTablet for tablet a6ddefa1738f4ff98586900c344359d2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f0c4f0e8f014e59be9e46d2afe0be9d) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:30.061975 1169 tablet_bootstrap.cc:492] T 1aeaad5c5d01478bb997254ac372d9a6 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:30.062278 1169 ts_tablet_manager.cc:1403] T 1aeaad5c5d01478bb997254ac372d9a6 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.231s user 0.025s sys 0.021s
I20251025 07:52:30.063261 1169 raft_consensus.cc:359] T 1aeaad5c5d01478bb997254ac372d9a6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.063558 1169 raft_consensus.cc:385] T 1aeaad5c5d01478bb997254ac372d9a6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:30.063747 1169 raft_consensus.cc:740] T 1aeaad5c5d01478bb997254ac372d9a6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:30.064240 1169 consensus_queue.cc:260] T 1aeaad5c5d01478bb997254ac372d9a6 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.067299 821 heartbeater.cc:344] Connected to a master server at 127.0.104.60:37611
I20251025 07:52:30.068147 1167 tablet_bootstrap.cc:654] T 79bc2678df8042e3994b0471afa2c304 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:30.071048 1166 tablet_bootstrap.cc:492] T 2aa522477e934c89834026dc480fe7c0 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:30.071264 1166 ts_tablet_manager.cc:1403] T 2aa522477e934c89834026dc480fe7c0 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.092s user 0.001s sys 0.015s
I20251025 07:52:30.081066 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_0b63d3c003db48709227445b0a0b84ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:30.081733 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b63d3c003db48709227445b0a0b84ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:30.082727 1171 raft_consensus.cc:2272] T 94408a3a90224645ba3cacccf5de33f1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:30.083760 1171 ts_tablet_manager.cc:1916] T 94408a3a90224645ba3cacccf5de33f1 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:30.083756 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_7f531c5ddd174b49b0117a226cd0d91f" } new_table_name: "default.Table_6c5a24747ba14352b0285a41ec2985a4" modify_external_catalogs: true
I20251025 07:52:30.080816 1166 raft_consensus.cc:359] T 2aa522477e934c89834026dc480fe7c0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.085328 1166 raft_consensus.cc:385] T 2aa522477e934c89834026dc480fe7c0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:30.085422 1166 raft_consensus.cc:740] T 2aa522477e934c89834026dc480fe7c0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:30.085763 1166 consensus_queue.cc:260] T 2aa522477e934c89834026dc480fe7c0 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.086439 1166 ts_tablet_manager.cc:1434] T 2aa522477e934c89834026dc480fe7c0 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.015s user 0.003s sys 0.000s
I20251025 07:52:30.086828 1166 tablet_bootstrap.cc:492] T 567a1f5c216a4a519249da66a2a03ea4 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:30.089006 520 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 796413da731245459cc5dffacc3deb2f (table default.Table_293596c64cca4768bc5869fd9924f856 [id=d1b9d89b22bc4431a7d59ceeb9d75050]). Expected version 2 got 0
I20251025 07:52:30.067126 1094 heartbeater.cc:344] Connected to a master server at 127.0.104.60:37611
I20251025 07:52:30.092542 1172 ts_tablet_manager.cc:1929] T a231f3d31b204f7b82f018892ccc8435 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:30.092700 1172 log.cc:1199] T a231f3d31b204f7b82f018892ccc8435 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/a231f3d31b204f7b82f018892ccc8435
I20251025 07:52:30.095153 1167 tablet_bootstrap.cc:492] T 79bc2678df8042e3994b0471afa2c304 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:30.095363 1167 ts_tablet_manager.cc:1403] T 79bc2678df8042e3994b0471afa2c304 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.058s user 0.005s sys 0.001s
I20251025 07:52:30.095892 1169 ts_tablet_manager.cc:1434] T 1aeaad5c5d01478bb997254ac372d9a6 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.033s user 0.000s sys 0.002s
I20251025 07:52:30.096319 1167 raft_consensus.cc:359] T 79bc2678df8042e3994b0471afa2c304 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.096557 1167 raft_consensus.cc:385] T 79bc2678df8042e3994b0471afa2c304 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:30.096681 1167 raft_consensus.cc:740] T 79bc2678df8042e3994b0471afa2c304 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:30.096635 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_6b9cad0ecb4a4d74941d9904282d676d" } new_table_name: "default.Table_434ef1ade0a14af987864563993afae8" modify_external_catalogs: true
I20251025 07:52:30.097002 1167 consensus_queue.cc:260] T 79bc2678df8042e3994b0471afa2c304 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.097764 1167 ts_tablet_manager.cc:1434] T 79bc2678df8042e3994b0471afa2c304 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:30.101873 1172 ts_tablet_manager.cc:1950] T a231f3d31b204f7b82f018892ccc8435 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:30.103549 1167 tablet_bootstrap.cc:492] T 85fba8b37f2c4540aa81701052bf222a P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:30.105361 1169 tablet_bootstrap.cc:492] T 567a1f5c216a4a519249da66a2a03ea4 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:30.105381 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 252) took 1306 ms. Trace:
I20251025 07:52:30.105516 1172 rpcz_store.cc:276] 1025 07:52:28.798472 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:28.798870 (+ 398us) service_pool.cc:225] Handling call
1025 07:52:30.036587 (+1237717us) ts_tablet_manager.cc:1151] Deleting tablet a231f3d31b204f7b82f018892ccc8435
1025 07:52:30.036600 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:30.048619 (+ 12019us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:30.092360 (+ 43741us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:30.105336 (+ 12976us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1236182}
I20251025 07:52:30.106211 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet a231f3d31b204f7b82f018892ccc8435 (table default.Table_1c458711d56d4ddeba1d0bc09392a734 [id=7cebd72e604e497db80a8bc823471e97]) successfully deleted
I20251025 07:52:30.106982 1172 tablet_replica.cc:333] T 1483c40eee834f20a92e6b4cd89775b6 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:30.108162 1166 tablet_bootstrap.cc:654] T 567a1f5c216a4a519249da66a2a03ea4 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:30.109470 1172 raft_consensus.cc:2243] T 1483c40eee834f20a92e6b4cd89775b6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:30.109639 1172 raft_consensus.cc:2272] T 1483c40eee834f20a92e6b4cd89775b6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:30.113096 1476 tablet.cc:1722] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 23eeffd3de254ce18db5febfde971c6b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251025 07:52:30.114671 1167 tablet_bootstrap.cc:654] T 85fba8b37f2c4540aa81701052bf222a P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:30.116175 1507 tablet.cc:1722] T 796413da731245459cc5dffacc3deb2f P db42be7ad3c741a0aa427f27e245993b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251025 07:52:30.122402 959 heartbeater.cc:344] Connected to a master server at 127.0.104.60:37611
I20251025 07:52:30.126574 1472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407: Bootstrap replayed 1/1 log segments. Stats: ops{read=19 overwritten=0 applied=16 ignored=0} inserts{seen=23 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20251025 07:52:30.127312 1472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407: Bootstrap complete.
I20251025 07:52:30.128520 1172 ts_tablet_manager.cc:1916] T 1483c40eee834f20a92e6b4cd89775b6 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:30.133384 526 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 796413da731245459cc5dffacc3deb2f (table default.Table_293596c64cca4768bc5869fd9924f856 [id=d1b9d89b22bc4431a7d59ceeb9d75050]). Expected version 2 got 0
I20251025 07:52:30.133652 526 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 50ffbcdd4a9247bf99b8e2361bc02ef5 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 4 got 3
I20251025 07:52:30.136134 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_4cf617a8064c4be99402936ea4cf514b" } modify_external_catalogs: true
I20251025 07:52:30.136432 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_4cf617a8064c4be99402936ea4cf514b" } modify_external_catalogs: true
I20251025 07:52:30.138231 740 tablet_service.cc:1552] Processing DeleteTablet for tablet 198db9a45257435f86df2605992ccb34 with delete_type TABLET_DATA_DELETED (replaced by tablet 71432a6848944a0da06ef9b81f160dc4) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:30.142340 1247 raft_consensus.cc:493] T 87a93778620f4c84b3b3f7a3d7d7e87f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:30.142513 1247 raft_consensus.cc:515] T 87a93778620f4c84b3b3f7a3d7d7e87f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.143190 1247 leader_election.cc:290] T 87a93778620f4c84b3b3f7a3d7d7e87f P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:30.142395 1472 raft_consensus.cc:359] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [term 1 FOLLOWER]: Replica starting. Triggering 3 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } }
W20251025 07:52:30.144902 983 leader_election.cc:343] T 87a93778620f4c84b3b3f7a3d7d7e87f P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 87a93778620f4c84b3b3f7a3d7d7e87f
I20251025 07:52:30.145154 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87a93778620f4c84b3b3f7a3d7d7e87f" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:30.145443 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_0c74279815ab4a5186497b68bb932523" } modify_external_catalogs: true
I20251025 07:52:30.145637 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_0c74279815ab4a5186497b68bb932523" } modify_external_catalogs: true
I20251025 07:52:30.145741 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 1b31177a062b467187d68ed2034f5690 with delete_type TABLET_DATA_DELETED (replaced by tablet b772025852794a49b072f37cccbd3291) from {username='slave'} at 127.0.0.1:48830
W20251025 07:52:30.146543 982 leader_election.cc:343] T 87a93778620f4c84b3b3f7a3d7d7e87f P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:30.146688 982 leader_election.cc:304] T 87a93778620f4c84b3b3f7a3d7d7e87f P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:30.146931 1247 raft_consensus.cc:2749] T 87a93778620f4c84b3b3f7a3d7d7e87f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:30.149863 1476 tablet.cc:1722] T 796413da731245459cc5dffacc3deb2f P 23eeffd3de254ce18db5febfde971c6b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251025 07:52:30.155191 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c0c6324cbae34dcebade3ef9f89a7150 has no consensus state
I20251025 07:52:30.155711 1167 tablet_bootstrap.cc:492] T 85fba8b37f2c4540aa81701052bf222a P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:30.155882 1167 ts_tablet_manager.cc:1403] T 85fba8b37f2c4540aa81701052bf222a P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.052s user 0.001s sys 0.006s
I20251025 07:52:30.156339 1472 raft_consensus.cc:740] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4a999d754c941ec850c143e4e24c407, State: Initialized, Role: FOLLOWER
I20251025 07:52:30.156911 1167 raft_consensus.cc:359] T 85fba8b37f2c4540aa81701052bf222a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.157142 1167 raft_consensus.cc:385] T 85fba8b37f2c4540aa81701052bf222a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:30.157227 1167 raft_consensus.cc:740] T 85fba8b37f2c4540aa81701052bf222a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:30.157618 1167 consensus_queue.cc:260] T 85fba8b37f2c4540aa81701052bf222a P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.158353 1167 ts_tablet_manager.cc:1434] T 85fba8b37f2c4540aa81701052bf222a P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251025 07:52:30.158433 1472 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 1.19, Last appended by leader: 19, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } }
I20251025 07:52:30.161795 1441 raft_consensus.cc:493] T cf2cab1552bd4e7394797c93b149f450 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:30.161947 1441 raft_consensus.cc:515] T cf2cab1552bd4e7394797c93b149f450 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.162828 1441 leader_election.cc:290] T cf2cab1552bd4e7394797c93b149f450 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:30.163360 1508 tablet.cc:1722] T 796413da731245459cc5dffacc3deb2f P 3aea2869ff814cc7b142012faa274382: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251025 07:52:30.165161 710 leader_election.cc:343] T cf2cab1552bd4e7394797c93b149f450 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: cf2cab1552bd4e7394797c93b149f450
I20251025 07:52:30.168602 1441 raft_consensus.cc:493] T 198db9a45257435f86df2605992ccb34 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:30.168787 1441 raft_consensus.cc:515] T 198db9a45257435f86df2605992ccb34 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.170614 1169 tablet_bootstrap.cc:654] T 567a1f5c216a4a519249da66a2a03ea4 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:30.171362 1472 sys_catalog.cc:565] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 07:52:30.173309 1509 tablet.cc:1722] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P db42be7ad3c741a0aa427f27e245993b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251025 07:52:30.173477 518 master_service.cc:954] ReplaceTablet: received request to replace tablet 74e2d78c024849b0af5c0bde1f48c36f from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:30.175705 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_2c4afce2b14c41e29c5034dc68d31ead" } modify_external_catalogs: true
I20251025 07:52:30.175912 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_2c4afce2b14c41e29c5034dc68d31ead" } modify_external_catalogs: true
I20251025 07:52:30.176633 1167 tablet_bootstrap.cc:492] T 8c6ff4c73e624af3bf2de36c4493b462 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:30.180429 1505 tablet.cc:1722] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251025 07:52:30.184554 1166 tablet_bootstrap.cc:492] T 567a1f5c216a4a519249da66a2a03ea4 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:30.184741 1166 ts_tablet_manager.cc:1403] T 567a1f5c216a4a519249da66a2a03ea4 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.098s user 0.004s sys 0.005s
I20251025 07:52:30.192862 1166 raft_consensus.cc:359] T 567a1f5c216a4a519249da66a2a03ea4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.193077 1166 raft_consensus.cc:385] T 567a1f5c216a4a519249da66a2a03ea4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:30.193185 1166 raft_consensus.cc:740] T 567a1f5c216a4a519249da66a2a03ea4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:30.193428 1166 consensus_queue.cc:260] T 567a1f5c216a4a519249da66a2a03ea4 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.194063 1166 ts_tablet_manager.cc:1434] T 567a1f5c216a4a519249da66a2a03ea4 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251025 07:52:30.194391 1166 tablet_bootstrap.cc:492] T 1aeaad5c5d01478bb997254ac372d9a6 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:30.196868 1167 tablet_bootstrap.cc:654] T 8c6ff4c73e624af3bf2de36c4493b462 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:30.198114 1166 tablet_bootstrap.cc:654] T 1aeaad5c5d01478bb997254ac372d9a6 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:30.198830 1169 tablet_bootstrap.cc:492] T 567a1f5c216a4a519249da66a2a03ea4 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:30.199007 1169 ts_tablet_manager.cc:1403] T 567a1f5c216a4a519249da66a2a03ea4 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.094s user 0.001s sys 0.002s
I20251025 07:52:30.199810 1169 raft_consensus.cc:359] T 567a1f5c216a4a519249da66a2a03ea4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.200016 1169 raft_consensus.cc:385] T 567a1f5c216a4a519249da66a2a03ea4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:30.200106 1169 raft_consensus.cc:740] T 567a1f5c216a4a519249da66a2a03ea4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:30.200361 1169 consensus_queue.cc:260] T 567a1f5c216a4a519249da66a2a03ea4 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.201253 1169 ts_tablet_manager.cc:1434] T 567a1f5c216a4a519249da66a2a03ea4 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:30.201506 1441 leader_election.cc:290] T 198db9a45257435f86df2605992ccb34 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:30.201700 712 leader_election.cc:343] T cf2cab1552bd4e7394797c93b149f450 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: cf2cab1552bd4e7394797c93b149f450
I20251025 07:52:30.202807 712 leader_election.cc:304] T cf2cab1552bd4e7394797c93b149f450 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:30.203349 712 leader_election.cc:343] T 198db9a45257435f86df2605992ccb34 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 198db9a45257435f86df2605992ccb34
I20251025 07:52:30.204373 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 79bc2678df8042e3994b0471afa2c304 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:30.204756 1172 ts_tablet_manager.cc:1929] T 1483c40eee834f20a92e6b4cd89775b6 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:30.204934 1172 log.cc:1199] T 1483c40eee834f20a92e6b4cd89775b6 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/1483c40eee834f20a92e6b4cd89775b6
I20251025 07:52:30.205402 1172 ts_tablet_manager.cc:1950] T 1483c40eee834f20a92e6b4cd89775b6 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:30.206600 1505 tablet.cc:1717] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382: Skipping requested alter to schema version 4, tablet already version 4
W20251025 07:52:30.207157 710 leader_election.cc:343] T 198db9a45257435f86df2605992ccb34 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 198db9a45257435f86df2605992ccb34
I20251025 07:52:30.207299 710 leader_election.cc:304] T 198db9a45257435f86df2605992ccb34 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:30.208093 821 heartbeater.cc:344] Connected to a master server at 127.0.104.60:37611
I20251025 07:52:30.208731 1433 raft_consensus.cc:2749] T 198db9a45257435f86df2605992ccb34 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:30.209584 1441 raft_consensus.cc:2749] T cf2cab1552bd4e7394797c93b149f450 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:30.221009 1170 ts_tablet_manager.cc:1929] T 1b31177a062b467187d68ed2034f5690 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:30.221192 1170 log.cc:1199] T 1b31177a062b467187d68ed2034f5690 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/1b31177a062b467187d68ed2034f5690
I20251025 07:52:30.221594 1170 ts_tablet_manager.cc:1950] T 1b31177a062b467187d68ed2034f5690 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:30.222757 525 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet cd23b8094d7e472eba6bd170248858a9 (table default.Table_dfc7f1d4bf8b48bebbb00313419c3fd8 [id=d82b3e1317434cf08a0d51e8a9427224]). Expected version 2 got 0
I20251025 07:52:30.223737 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_b8a12e9a7a974006904fcef12bb8225a" } modify_external_catalogs: true
I20251025 07:52:30.223903 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_b8a12e9a7a974006904fcef12bb8225a" } modify_external_catalogs: true
I20251025 07:52:30.227164 1019 tablet_service.cc:1505] Processing CreateTablet for tablet 2e2e6893a48449969b71ee1c08d1960e (DEFAULT_TABLE table=default.Table_19354444047b451bbf22bafcf9ae185d [id=b3725485d48c41a1952091d028a783c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.229207 755 tablet_service.cc:1505] Processing CreateTablet for tablet 2e2e6893a48449969b71ee1c08d1960e (DEFAULT_TABLE table=default.Table_19354444047b451bbf22bafcf9ae185d [id=b3725485d48c41a1952091d028a783c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.229696 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e2e6893a48449969b71ee1c08d1960e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.231869 1018 tablet_service.cc:1505] Processing CreateTablet for tablet 570201e7bf05406a97c81dde071d894b (DEFAULT_TABLE table=default.Table_646f48a21c194af3a1611e5e12b2e11b [id=6f395892b09240fea02482b3199cfb5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.233803 882 tablet_service.cc:1505] Processing CreateTablet for tablet 2e2e6893a48449969b71ee1c08d1960e (DEFAULT_TABLE table=default.Table_19354444047b451bbf22bafcf9ae185d [id=b3725485d48c41a1952091d028a783c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.234225 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e2e6893a48449969b71ee1c08d1960e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.237290 881 tablet_service.cc:1505] Processing CreateTablet for tablet 36c414bc700341188b7d85c14173c3e0 (DEFAULT_TABLE table=default.Table_4e81dfac4c7742c7986eb24e6d54a124 [id=b87e43a3dc3c4cd98ee3f44964e8092f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.237654 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36c414bc700341188b7d85c14173c3e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.238977 1166 tablet_bootstrap.cc:492] T 1aeaad5c5d01478bb997254ac372d9a6 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:30.239142 1166 ts_tablet_manager.cc:1403] T 1aeaad5c5d01478bb997254ac372d9a6 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.045s user 0.001s sys 0.003s
I20251025 07:52:30.239698 1169 tablet_bootstrap.cc:492] T 85e27fc52e9a4d62a8622645594f59c6 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:30.239974 1166 raft_consensus.cc:359] T 1aeaad5c5d01478bb997254ac372d9a6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.240185 1166 raft_consensus.cc:385] T 1aeaad5c5d01478bb997254ac372d9a6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:30.240267 1166 raft_consensus.cc:740] T 1aeaad5c5d01478bb997254ac372d9a6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:30.240489 1166 consensus_queue.cc:260] T 1aeaad5c5d01478bb997254ac372d9a6 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.241559 1169 tablet_bootstrap.cc:654] T 85e27fc52e9a4d62a8622645594f59c6 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:30.242436 891 tablet_service.cc:1505] Processing CreateTablet for tablet 570201e7bf05406a97c81dde071d894b (DEFAULT_TABLE table=default.Table_646f48a21c194af3a1611e5e12b2e11b [id=6f395892b09240fea02482b3199cfb5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.242786 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 570201e7bf05406a97c81dde071d894b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.244768 1171 ts_tablet_manager.cc:1929] T 94408a3a90224645ba3cacccf5de33f1 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:30.244912 1171 log.cc:1199] T 94408a3a90224645ba3cacccf5de33f1 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/94408a3a90224645ba3cacccf5de33f1
I20251025 07:52:30.245244 1171 ts_tablet_manager.cc:1950] T 94408a3a90224645ba3cacccf5de33f1 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:30.245512 1166 ts_tablet_manager.cc:1434] T 1aeaad5c5d01478bb997254ac372d9a6 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 07:52:30.246035 1507 tablet.cc:1717] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P db42be7ad3c741a0aa427f27e245993b: Skipping requested alter to schema version 4, tablet already version 4
I20251025 07:52:30.246279 1166 tablet_bootstrap.cc:492] T 90764275825c4ffb8712cb883d37a551 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:30.246417 892 tablet_service.cc:1505] Processing CreateTablet for tablet c0c6324cbae34dcebade3ef9f89a7150 (DEFAULT_TABLE table=default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.246786 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0c6324cbae34dcebade3ef9f89a7150. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.250460 880 tablet_service.cc:1505] Processing CreateTablet for tablet 4a1ec8c2ed7f41abb588f592f62097d1 (DEFAULT_TABLE table=default.Table_91898f2a551144d18df0309c827855bf [id=aaa734f6549741768b0156bdf5cc25af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.250906 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a1ec8c2ed7f41abb588f592f62097d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.253396 886 tablet_service.cc:1505] Processing CreateTablet for tablet c7691c7bdd89437c8db060254c0db25d (DEFAULT_TABLE table=default.Table_1f11da1b07914daabda9c8e29a9230fe [id=27557bb332dd49bb8f386c294bb12a69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.253746 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7691c7bdd89437c8db060254c0db25d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.253793 1166 tablet_bootstrap.cc:654] T 90764275825c4ffb8712cb883d37a551 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:30.256978 890 tablet_service.cc:1505] Processing CreateTablet for tablet 627a9cfd398346f2aa34c8932a11b5a8 (DEFAULT_TABLE table=default.Table_2bcb67fcbc85452faaadfa683987951c [id=63e72692d0b6442396189a2f37170815]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.257302 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 627a9cfd398346f2aa34c8932a11b5a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.260061 1422 raft_consensus.cc:1217] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.16->[1.17-1.431] Dedup: 1.19->[1.20-1.431]
I20251025 07:52:30.261410 1167 tablet_bootstrap.cc:492] T 8c6ff4c73e624af3bf2de36c4493b462 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:30.261592 1167 ts_tablet_manager.cc:1403] T 8c6ff4c73e624af3bf2de36c4493b462 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.085s user 0.004s sys 0.000s
I20251025 07:52:30.261909 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 255) took 1462 ms. Trace:
I20251025 07:52:30.262022 1172 rpcz_store.cc:276] 1025 07:52:28.799011 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:28.799302 (+ 291us) service_pool.cc:225] Handling call
1025 07:52:30.106916 (+1307614us) ts_tablet_manager.cc:1151] Deleting tablet 1483c40eee834f20a92e6b4cd89775b6
1025 07:52:30.106926 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:30.201208 (+ 94282us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:30.204569 (+ 3361us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:30.261842 (+ 57273us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1305925}
I20251025 07:52:30.262425 894 tablet_service.cc:1505] Processing CreateTablet for tablet 5b6383c99e9e4c90a374da9ccd830a6f (DEFAULT_TABLE table=default.Table_32f8a762ec25446dac28e2729d6879bf [id=a13b2a42afca460595cd872ab2cf9ea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.262494 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 1483c40eee834f20a92e6b4cd89775b6 (table default.Table_e0bf55e24b084fdbb62f09cce80b50c5 [id=084833e021c24b4a84dd1b1a3bb08dfc]) successfully deleted
I20251025 07:52:30.262545 1167 raft_consensus.cc:359] T 8c6ff4c73e624af3bf2de36c4493b462 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.262784 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b6383c99e9e4c90a374da9ccd830a6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.262801 1167 raft_consensus.cc:385] T 8c6ff4c73e624af3bf2de36c4493b462 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:30.262902 1167 raft_consensus.cc:740] T 8c6ff4c73e624af3bf2de36c4493b462 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:30.262928 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 265) took 1426 ms. Trace:
I20251025 07:52:30.263008 1172 rpcz_store.cc:276] 1025 07:52:28.836916 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:28.837045 (+ 129us) service_pool.cc:225] Handling call
1025 07:52:30.262738 (+1425693us) ts_tablet_manager.cc:1151] Deleting tablet 943ec703e65f4fd3b0c690e08c0bd226
1025 07:52:30.262747 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:30.262868 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1372100}
I20251025 07:52:30.263207 1167 consensus_queue.cc:260] T 8c6ff4c73e624af3bf2de36c4493b462 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:30.263410 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 943ec703e65f4fd3b0c690e08c0bd226 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 943ec703e65f4fd3b0c690e08c0bd226 already in progress: creating tablet
I20251025 07:52:30.263726 1172 tablet_replica.cc:333] T 1b31177a062b467187d68ed2034f5690 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:30.263937 1172 raft_consensus.cc:2243] T 1b31177a062b467187d68ed2034f5690 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:30.264130 1172 raft_consensus.cc:2272] T 1b31177a062b467187d68ed2034f5690 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:30.264683 1172 ts_tablet_manager.cc:1916] T 1b31177a062b467187d68ed2034f5690 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:30.266440 893 tablet_service.cc:1505] Processing CreateTablet for tablet 37c7938e8c5e41f08506aff6747801c9 (DEFAULT_TABLE table=default.Table_bef97bb8f0a442f3a23cb20cbf85d934 [id=4b4b515be910449395ba4de41a681ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.266829 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37c7938e8c5e41f08506aff6747801c9. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:30.269896 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:43563, user_credentials={real_user=slave}} blocked reactor thread for 77322us
I20251025 07:52:30.270423 878 tablet_service.cc:1505] Processing CreateTablet for tablet 143aa3fce9ba4850b000a94054e5dc3a (DEFAULT_TABLE table=default.Table_5261ed9b1f29420eab473df75f903987 [id=771a5d1dc57e4dffbfe4bf757f57719c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.272186 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 263) took 1570 ms. Trace:
I20251025 07:52:30.272281 1171 rpcz_store.cc:276] 1025 07:52:28.701357 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:28.701408 (+ 51us) service_pool.cc:225] Handling call
1025 07:52:30.006569 (+1305161us) ts_tablet_manager.cc:1151] Deleting tablet 94408a3a90224645ba3cacccf5de33f1
1025 07:52:30.006581 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:30.227109 (+220528us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:30.244633 (+ 17524us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:30.272136 (+ 27503us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1304975}
I20251025 07:52:30.272734 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 260) took 1571 ms. Trace:
I20251025 07:52:30.273146 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e2e6893a48449969b71ee1c08d1960e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.276377 1027 tablet_service.cc:1505] Processing CreateTablet for tablet c0c6324cbae34dcebade3ef9f89a7150 (DEFAULT_TABLE table=default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.276762 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0c6324cbae34dcebade3ef9f89a7150. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.277859 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 627a9cfd398346f2aa34c8932a11b5a8 (DEFAULT_TABLE table=default.Table_2bcb67fcbc85452faaadfa683987951c [id=63e72692d0b6442396189a2f37170815]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.278283 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 627a9cfd398346f2aa34c8932a11b5a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.278455 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 4a1ec8c2ed7f41abb588f592f62097d1 (DEFAULT_TABLE table=default.Table_91898f2a551144d18df0309c827855bf [id=aaa734f6549741768b0156bdf5cc25af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.278853 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a1ec8c2ed7f41abb588f592f62097d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.278961 1170 tablet_replica.cc:333] T f51ec427dd9349a48eb221db04cf242b P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:30.279160 1170 raft_consensus.cc:2243] T f51ec427dd9349a48eb221db04cf242b P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:30.279449 1170 pending_rounds.cc:70] T f51ec427dd9349a48eb221db04cf242b P db42be7ad3c741a0aa427f27e245993b: Trying to abort 1 pending ops.
I20251025 07:52:30.279525 1170 pending_rounds.cc:77] T f51ec427dd9349a48eb221db04cf242b P db42be7ad3c741a0aa427f27e245993b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214607359505289216 op_type: NO_OP noop_request { }
I20251025 07:52:30.279659 1170 raft_consensus.cc:2889] T f51ec427dd9349a48eb221db04cf242b P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 07:52:30.279779 1170 raft_consensus.cc:2272] T f51ec427dd9349a48eb221db04cf242b P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:30.280609 1020 tablet_service.cc:1505] Processing CreateTablet for tablet 5b6383c99e9e4c90a374da9ccd830a6f (DEFAULT_TABLE table=default.Table_32f8a762ec25446dac28e2729d6879bf [id=a13b2a42afca460595cd872ab2cf9ea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.280943 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b6383c99e9e4c90a374da9ccd830a6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.282403 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 36c414bc700341188b7d85c14173c3e0 (DEFAULT_TABLE table=default.Table_4e81dfac4c7742c7986eb24e6d54a124 [id=b87e43a3dc3c4cd98ee3f44964e8092f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.282722 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36c414bc700341188b7d85c14173c3e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.286522 1021 tablet_service.cc:1505] Processing CreateTablet for tablet 37c7938e8c5e41f08506aff6747801c9 (DEFAULT_TABLE table=default.Table_bef97bb8f0a442f3a23cb20cbf85d934 [id=4b4b515be910449395ba4de41a681ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.286978 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37c7938e8c5e41f08506aff6747801c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.290477 1026 tablet_service.cc:1505] Processing CreateTablet for tablet c7691c7bdd89437c8db060254c0db25d (DEFAULT_TABLE table=default.Table_1f11da1b07914daabda9c8e29a9230fe [id=27557bb332dd49bb8f386c294bb12a69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.290969 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7691c7bdd89437c8db060254c0db25d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.294399 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 143aa3fce9ba4850b000a94054e5dc3a (DEFAULT_TABLE table=default.Table_5261ed9b1f29420eab473df75f903987 [id=771a5d1dc57e4dffbfe4bf757f57719c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.294744 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 143aa3fce9ba4850b000a94054e5dc3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.297698 1170 ts_tablet_manager.cc:1916] T f51ec427dd9349a48eb221db04cf242b P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:30.298885 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 570201e7bf05406a97c81dde071d894b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.272809 1171 rpcz_store.cc:276] 1025 07:52:28.701062 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:28.701796 (+ 734us) service_pool.cc:225] Handling call
1025 07:52:30.272586 (+1570790us) ts_tablet_manager.cc:1151] Deleting tablet e54fe7cbcffa40c4b04144b47bee2b48
1025 07:52:30.272594 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:30.272691 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1570603}
I20251025 07:52:30.302681 1171 tablet_replica.cc:333] T 3c3eb43209f24592820638fc25486b78 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:30.303273 742 tablet_service.cc:1505] Processing CreateTablet for tablet c0c6324cbae34dcebade3ef9f89a7150 (DEFAULT_TABLE table=default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.303740 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0c6324cbae34dcebade3ef9f89a7150. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.304458 1170 ts_tablet_manager.cc:1929] T f51ec427dd9349a48eb221db04cf242b P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:30.304631 1170 log.cc:1199] T f51ec427dd9349a48eb221db04cf242b P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/f51ec427dd9349a48eb221db04cf242b
I20251025 07:52:30.305001 1170 ts_tablet_manager.cc:1950] T f51ec427dd9349a48eb221db04cf242b P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:30.305534 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 94408a3a90224645ba3cacccf5de33f1 (table default.Table_819aedd5270f4d6cbaaaee7c2ff5f676 [id=1e4728f474814b36992fd99de8451b95]) successfully deleted
W20251025 07:52:30.305824 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet e54fe7cbcffa40c4b04144b47bee2b48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e54fe7cbcffa40c4b04144b47bee2b48 already in progress: creating tablet
I20251025 07:52:30.306293 1170 tablet_replica.cc:333] T 87a93778620f4c84b3b3f7a3d7d7e87f P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:30.306488 1170 raft_consensus.cc:2243] T 87a93778620f4c84b3b3f7a3d7d7e87f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:30.306679 1170 raft_consensus.cc:2272] T 87a93778620f4c84b3b3f7a3d7d7e87f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:30.306862 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet f51ec427dd9349a48eb221db04cf242b (table default.Table_cc49d933a4b343aeb2220c3939495834 [id=e5e538e59868418e9805492aa80f3240]) successfully deleted
W20251025 07:52:30.307092 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet be5ab0db6ac34c2c9a370445e251eb46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be5ab0db6ac34c2c9a370445e251eb46 already in progress: creating tablet
W20251025 07:52:30.307276 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 53198283718d4b45b30df65b5bde96a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53198283718d4b45b30df65b5bde96a6 already in progress: creating tablet
I20251025 07:52:30.307250 752 tablet_service.cc:1505] Processing CreateTablet for tablet 5b6383c99e9e4c90a374da9ccd830a6f (DEFAULT_TABLE table=default.Table_32f8a762ec25446dac28e2729d6879bf [id=a13b2a42afca460595cd872ab2cf9ea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.307423 1170 ts_tablet_manager.cc:1916] T 87a93778620f4c84b3b3f7a3d7d7e87f P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:30.307701 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b6383c99e9e4c90a374da9ccd830a6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.310393 1476 tablet.cc:1717] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 23eeffd3de254ce18db5febfde971c6b: Skipping requested alter to schema version 4, tablet already version 4
I20251025 07:52:30.310506 748 tablet_service.cc:1505] Processing CreateTablet for tablet 37c7938e8c5e41f08506aff6747801c9 (DEFAULT_TABLE table=default.Table_bef97bb8f0a442f3a23cb20cbf85d934 [id=4b4b515be910449395ba4de41a681ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.310860 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37c7938e8c5e41f08506aff6747801c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.314658 739 tablet_service.cc:1505] Processing CreateTablet for tablet 143aa3fce9ba4850b000a94054e5dc3a (DEFAULT_TABLE table=default.Table_5261ed9b1f29420eab473df75f903987 [id=771a5d1dc57e4dffbfe4bf757f57719c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.315109 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 143aa3fce9ba4850b000a94054e5dc3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.315279 1170 ts_tablet_manager.cc:1929] T 87a93778620f4c84b3b3f7a3d7d7e87f P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:30.315397 1170 log.cc:1199] T 87a93778620f4c84b3b3f7a3d7d7e87f P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/87a93778620f4c84b3b3f7a3d7d7e87f
I20251025 07:52:30.315730 1170 ts_tablet_manager.cc:1950] T 87a93778620f4c84b3b3f7a3d7d7e87f P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:30.316788 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 87a93778620f4c84b3b3f7a3d7d7e87f (table default.Table_0eca32ca64da44198140ae7ec9a77767 [id=2da58c150b514203abed5e20ffa98919]) successfully deleted
I20251025 07:52:30.318277 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 143aa3fce9ba4850b000a94054e5dc3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.318259 1247 raft_consensus.cc:493] T 1aeaad5c5d01478bb997254ac372d9a6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:30.318436 1247 raft_consensus.cc:515] T 1aeaad5c5d01478bb997254ac372d9a6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.319260 1247 leader_election.cc:290] T 1aeaad5c5d01478bb997254ac372d9a6 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:30.320551 740 tablet_service.cc:1505] Processing CreateTablet for tablet 4a1ec8c2ed7f41abb588f592f62097d1 (DEFAULT_TABLE table=default.Table_91898f2a551144d18df0309c827855bf [id=aaa734f6549741768b0156bdf5cc25af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.320911 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a1ec8c2ed7f41abb588f592f62097d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.324393 741 tablet_service.cc:1505] Processing CreateTablet for tablet 627a9cfd398346f2aa34c8932a11b5a8 (DEFAULT_TABLE table=default.Table_2bcb67fcbc85452faaadfa683987951c [id=63e72692d0b6442396189a2f37170815]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.324890 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 627a9cfd398346f2aa34c8932a11b5a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.326412 754 tablet_service.cc:1505] Processing CreateTablet for tablet c7691c7bdd89437c8db060254c0db25d (DEFAULT_TABLE table=default.Table_1f11da1b07914daabda9c8e29a9230fe [id=27557bb332dd49bb8f386c294bb12a69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.326874 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7691c7bdd89437c8db060254c0db25d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.306530 749 tablet_service.cc:1505] Processing CreateTablet for tablet 36c414bc700341188b7d85c14173c3e0 (DEFAULT_TABLE table=default.Table_4e81dfac4c7742c7986eb24e6d54a124 [id=b87e43a3dc3c4cd98ee3f44964e8092f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.336706 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36c414bc700341188b7d85c14173c3e0. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:30.342715 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet f97acc9a638b4d5a968dcc3412a384bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f97acc9a638b4d5a968dcc3412a384bc already in progress: creating tablet
W20251025 07:52:30.342990 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet db99a1e68daf4e948a3d66ad276cc5f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db99a1e68daf4e948a3d66ad276cc5f9 already in progress: creating tablet
W20251025 07:52:30.343179 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 91728987303b41fd840eb56f3a311c5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91728987303b41fd840eb56f3a311c5f already in progress: creating tablet
W20251025 07:52:30.343398 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet a6ddefa1738f4ff98586900c344359d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6ddefa1738f4ff98586900c344359d2 already in progress: creating tablet
I20251025 07:52:30.348927 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aeaad5c5d01478bb997254ac372d9a6" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
W20251025 07:52:30.349711 982 leader_election.cc:343] T 1aeaad5c5d01478bb997254ac372d9a6 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:30.297545 1167 ts_tablet_manager.cc:1434] T 8c6ff4c73e624af3bf2de36c4493b462 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.036s user 0.002s sys 0.000s
I20251025 07:52:30.353163 755 tablet_service.cc:1505] Processing CreateTablet for tablet 570201e7bf05406a97c81dde071d894b (DEFAULT_TABLE table=default.Table_646f48a21c194af3a1611e5e12b2e11b [id=6f395892b09240fea02482b3199cfb5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.354187 1444 raft_consensus.cc:493] T 1aeaad5c5d01478bb997254ac372d9a6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:30.357159 1444 raft_consensus.cc:515] T 1aeaad5c5d01478bb997254ac372d9a6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.359200 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aeaad5c5d01478bb997254ac372d9a6" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:30.362723 1169 tablet_bootstrap.cc:492] T 85e27fc52e9a4d62a8622645594f59c6 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:30.362893 1169 ts_tablet_manager.cc:1403] T 85e27fc52e9a4d62a8622645594f59c6 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.123s user 0.002s sys 0.005s
I20251025 07:52:30.363693 1169 raft_consensus.cc:359] T 85e27fc52e9a4d62a8622645594f59c6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.363873 1169 raft_consensus.cc:385] T 85e27fc52e9a4d62a8622645594f59c6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:30.363978 1169 raft_consensus.cc:740] T 85e27fc52e9a4d62a8622645594f59c6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:30.364717 1169 consensus_queue.cc:260] T 85e27fc52e9a4d62a8622645594f59c6 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:30.366993 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c155aa7fa5b4d68bbab1918b8432372 consensus state has no leader
I20251025 07:52:30.367743 1169 ts_tablet_manager.cc:1434] T 85e27fc52e9a4d62a8622645594f59c6 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20251025 07:52:30.370184 1169 tablet_bootstrap.cc:492] T 627a10a0714f4be8bdbced503fd6388d P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:30.370373 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 2aa522477e934c89834026dc480fe7c0
I20251025 07:52:30.370782 521 catalog_manager.cc:6719] ReplaceTablet: tablet 2aa522477e934c89834026dc480fe7c0 deleted and replaced by tablet 1068f566477e4c6db7dde566fff4c2fb
I20251025 07:52:30.371132 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_25f26dbeafcf4afdaaa118dc47494351"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:30.371693 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_25f26dbeafcf4afdaaa118dc47494351 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:30.371917 1169 tablet_bootstrap.cc:654] T 627a10a0714f4be8bdbced503fd6388d P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:30.373057 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 467324e2432f4caca148951329caa292
I20251025 07:52:30.374429 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 551d07a9242740dcb9366bb2d707a4a8
I20251025 07:52:30.375267 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_e08a3ff70f3a411e82b4bb292d768961"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:30.375921 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e08a3ff70f3a411e82b4bb292d768961 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:30.380076 1169 tablet_bootstrap.cc:492] T 627a10a0714f4be8bdbced503fd6388d P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:30.380264 1169 ts_tablet_manager.cc:1403] T 627a10a0714f4be8bdbced503fd6388d P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20251025 07:52:30.380494 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 74e2d78c024849b0af5c0bde1f48c36f
I20251025 07:52:30.381445 518 catalog_manager.cc:6719] ReplaceTablet: tablet 74e2d78c024849b0af5c0bde1f48c36f deleted and replaced by tablet fd265cdbf6a8440393a2b5d8240d1872
I20251025 07:52:30.381467 1169 raft_consensus.cc:359] T 627a10a0714f4be8bdbced503fd6388d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.381671 1169 raft_consensus.cc:385] T 627a10a0714f4be8bdbced503fd6388d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:30.381752 1169 raft_consensus.cc:740] T 627a10a0714f4be8bdbced503fd6388d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:30.381768 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_7ce7627f71134d5c989f9ba5c6766a2a" } modify_external_catalogs: true
I20251025 07:52:30.381933 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_7ce7627f71134d5c989f9ba5c6766a2a" } modify_external_catalogs: true
I20251025 07:52:30.382045 1169 consensus_queue.cc:260] T 627a10a0714f4be8bdbced503fd6388d P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:30.384928 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 74e2d78c024849b0af5c0bde1f48c36f consensus state has no leader
I20251025 07:52:30.385291 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_bd168da90c704b7987fc731f5ab28045"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:30.385823 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd168da90c704b7987fc731f5ab28045 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:30.386906 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 467324e2432f4caca148951329caa292 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:30.387506 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 570201e7bf05406a97c81dde071d894b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.387701 744 tablet_service.cc:1552] Processing DeleteTablet for tablet 551d07a9242740dcb9366bb2d707a4a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:30.387807 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 2aa522477e934c89834026dc480fe7c0 with delete_type TABLET_DATA_DELETED (replaced by tablet 1068f566477e4c6db7dde566fff4c2fb) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:30.388070 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 467324e2432f4caca148951329caa292 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:48830
W20251025 07:52:30.388257 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 467324e2432f4caca148951329caa292 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 467324e2432f4caca148951329caa292 already in progress: creating tablet
I20251025 07:52:30.388290 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 74e2d78c024849b0af5c0bde1f48c36f with delete_type TABLET_DATA_DELETED (replaced by tablet fd265cdbf6a8440393a2b5d8240d1872) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:30.389060 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 2aa522477e934c89834026dc480fe7c0 with delete_type TABLET_DATA_DELETED (replaced by tablet 1068f566477e4c6db7dde566fff4c2fb) from {username='slave'} at 127.0.0.1:47408
W20251025 07:52:30.389860 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 2aa522477e934c89834026dc480fe7c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2aa522477e934c89834026dc480fe7c0 already in progress: creating tablet
I20251025 07:52:30.394755 1169 ts_tablet_manager.cc:1434] T 627a10a0714f4be8bdbced503fd6388d P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
I20251025 07:52:30.395360 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_2070d67c804b4eceb6d29bfb97a42604"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:30.395859 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2070d67c804b4eceb6d29bfb97a42604 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:30.397054 1169 tablet_bootstrap.cc:492] T 47284f96928749e6a5201103a8ee5f9f P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:30.398793 1169 tablet_bootstrap.cc:654] T 47284f96928749e6a5201103a8ee5f9f P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:30.401561 1094 heartbeater.cc:344] Connected to a master server at 127.0.104.60:37611
W20251025 07:52:30.407676 511 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 4c155aa7fa5b4d68bbab1918b8432372 (table default.Table_6c5a24747ba14352b0285a41ec2985a4 [id=28bbaad474164a8c8c97a76bb8081791], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4c155aa7fa5b4d68bbab1918b8432372 consensus state has no leader
I20251025 07:52:30.411650 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aeaad5c5d01478bb997254ac372d9a6" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:30.411872 1048 raft_consensus.cc:2468] T 1aeaad5c5d01478bb997254ac372d9a6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:30.413488 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet f661d5909657455581dcb5e6eadd3c0b
I20251025 07:52:30.414218 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_4bc4254fb03d4db99f40d7b07b2f89aa" } new_table_name: "default.Table_9b7fc5a1f4944e749c90c1119d0b0054" modify_external_catalogs: true
I20251025 07:52:30.414810 1171 raft_consensus.cc:2243] T 3c3eb43209f24592820638fc25486b78 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:30.414850 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet d9bfd7c0c3ad4e5b98b6708793d61d41
I20251025 07:52:30.415376 524 master_service.cc:954] ReplaceTablet: received request to replace tablet 896bd7870d87408e8adfdb7a78998111 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:30.416451 1171 raft_consensus.cc:2272] T 3c3eb43209f24592820638fc25486b78 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:30.417025 1171 ts_tablet_manager.cc:1916] T 3c3eb43209f24592820638fc25486b78 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:30.417251 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet d9bfd7c0c3ad4e5b98b6708793d61d41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:30.420063 850 leader_election.cc:304] T 1aeaad5c5d01478bb997254ac372d9a6 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:30.421204 752 tablet_service.cc:1552] Processing DeleteTablet for tablet d9bfd7c0c3ad4e5b98b6708793d61d41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:30.423000 740 tablet_service.cc:1552] Processing DeleteTablet for tablet f661d5909657455581dcb5e6eadd3c0b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:30.423741 1167 tablet_bootstrap.cc:492] T 5caac9c943bf42a5b2ae522d87fdfe8e P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:30.424747 520 master_service.cc:954] ReplaceTablet: received request to replace tablet ebc824ec4c60411dab1a2ea4a385148e from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:30.425472 1167 tablet_bootstrap.cc:654] T 5caac9c943bf42a5b2ae522d87fdfe8e P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:30.426700 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet d9bfd7c0c3ad4e5b98b6708793d61d41 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9bfd7c0c3ad4e5b98b6708793d61d41 already in progress: creating tablet
I20251025 07:52:30.426811 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet f656945454fe4921ab819c772a089eb3
I20251025 07:52:30.427604 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_ce27d8ed3ede46579f2ed3ae81536572"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:30.428095 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce27d8ed3ede46579f2ed3ae81536572 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:30.430867 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_8028c8ff15754b788ce12115fde3be3d" } new_table_name: "default.Table_7da05a15ca474e779a81fecd62328481" modify_external_catalogs: true
I20251025 07:52:30.426013 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_3193d4af1aaa42a28d4eda999726f033"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:30.433034 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3193d4af1aaa42a28d4eda999726f033 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:30.434980 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_9c39679c7a244c7f9dd7d0f97f16ccb3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:30.435545 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c39679c7a244c7f9dd7d0f97f16ccb3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:30.435834 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6dcab53a020847a1aa27930265ba0205 consensus state has no leader
I20251025 07:52:30.436254 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_dbface3af7f6413d9060b881447d34b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:30.436755 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dbface3af7f6413d9060b881447d34b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:30.437486 914 raft_consensus.cc:2468] T 1aeaad5c5d01478bb997254ac372d9a6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:30.438864 1444 leader_election.cc:290] T 1aeaad5c5d01478bb997254ac372d9a6 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:30.439074 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 2aa522477e934c89834026dc480fe7c0 with delete_type TABLET_DATA_DELETED (replaced by tablet 1068f566477e4c6db7dde566fff4c2fb) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:30.439118 1166 tablet_bootstrap.cc:492] T 90764275825c4ffb8712cb883d37a551 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:30.439251 1500 raft_consensus.cc:2804] T 1aeaad5c5d01478bb997254ac372d9a6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:30.439293 1166 ts_tablet_manager.cc:1403] T 90764275825c4ffb8712cb883d37a551 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.193s user 0.013s sys 0.028s
I20251025 07:52:30.439510 1500 raft_consensus.cc:493] T 1aeaad5c5d01478bb997254ac372d9a6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:30.439638 1500 raft_consensus.cc:3060] T 1aeaad5c5d01478bb997254ac372d9a6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:30.442435 754 tablet_service.cc:1552] Processing DeleteTablet for tablet f656945454fe4921ab819c772a089eb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:30.445915 1496 raft_consensus.cc:493] T 2aa522477e934c89834026dc480fe7c0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:30.446062 1171 ts_tablet_manager.cc:1929] T 3c3eb43209f24592820638fc25486b78 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:30.446224 1171 log.cc:1199] T 3c3eb43209f24592820638fc25486b78 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/3c3eb43209f24592820638fc25486b78
I20251025 07:52:30.448051 1171 ts_tablet_manager.cc:1950] T 3c3eb43209f24592820638fc25486b78 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:30.448906 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 262) took 1747 ms. Trace:
I20251025 07:52:30.449031 1171 rpcz_store.cc:276] 1025 07:52:28.701243 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:28.702992 (+ 1749us) service_pool.cc:225] Handling call
1025 07:52:30.302573 (+1599581us) ts_tablet_manager.cc:1151] Deleting tablet 3c3eb43209f24592820638fc25486b78
1025 07:52:30.302584 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:30.427537 (+124953us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:30.445900 (+ 18363us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:30.448869 (+ 2969us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1599372}
I20251025 07:52:30.449501 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 3c3eb43209f24592820638fc25486b78 (table default.Table_9b8e648894e14f8595796e4698407495 [id=89d0b4e609a74dde8540eb740446be5a]) successfully deleted
I20251025 07:52:30.454680 1171 tablet_replica.cc:333] T 83084be89b2646b0bda208b3f6f60d77 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:30.455050 1171 raft_consensus.cc:2243] T 83084be89b2646b0bda208b3f6f60d77 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:30.455766 1171 raft_consensus.cc:2272] T 83084be89b2646b0bda208b3f6f60d77 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:30.456499 1171 ts_tablet_manager.cc:1916] T 83084be89b2646b0bda208b3f6f60d77 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:30.456413 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_86de368dc5bb4eb2a051a9db92a580dc" } new_table_name: "default.Table_4ea985df4c5b415c9570c64506a98cc4" modify_external_catalogs: true
I20251025 07:52:30.456557 1166 raft_consensus.cc:359] T 90764275825c4ffb8712cb883d37a551 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.456772 1166 raft_consensus.cc:385] T 90764275825c4ffb8712cb883d37a551 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:30.457343 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 551d07a9242740dcb9366bb2d707a4a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:56590
W20251025 07:52:30.458988 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 30418858bf95438686bdfe64ac733144 consensus state has no leader
I20251025 07:52:30.459429 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_34b975902f884c658093a51baa54b095"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:30.460007 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34b975902f884c658093a51baa54b095 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:30.461078 875 tablet_service.cc:1552] Processing DeleteTablet for tablet f661d5909657455581dcb5e6eadd3c0b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:30.446086 1496 raft_consensus.cc:515] T 2aa522477e934c89834026dc480fe7c0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.457808 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 467324e2432f4caca148951329caa292 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:30.462829 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 74e2d78c024849b0af5c0bde1f48c36f with delete_type TABLET_DATA_DELETED (replaced by tablet fd265cdbf6a8440393a2b5d8240d1872) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:30.466339 1167 tablet_bootstrap.cc:492] T 5caac9c943bf42a5b2ae522d87fdfe8e P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:30.466384 892 tablet_service.cc:1552] Processing DeleteTablet for tablet d9bfd7c0c3ad4e5b98b6708793d61d41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:30.466506 1167 ts_tablet_manager.cc:1403] T 5caac9c943bf42a5b2ae522d87fdfe8e P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.115s user 0.003s sys 0.000s
I20251025 07:52:30.467326 1166 raft_consensus.cc:740] T 90764275825c4ffb8712cb883d37a551 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:30.467370 1167 raft_consensus.cc:359] T 5caac9c943bf42a5b2ae522d87fdfe8e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.467635 1167 raft_consensus.cc:385] T 5caac9c943bf42a5b2ae522d87fdfe8e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:30.467718 1167 raft_consensus.cc:740] T 5caac9c943bf42a5b2ae522d87fdfe8e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:30.468030 1167 consensus_queue.cc:260] T 5caac9c943bf42a5b2ae522d87fdfe8e P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.469799 1171 ts_tablet_manager.cc:1929] T 83084be89b2646b0bda208b3f6f60d77 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:30.469941 1171 log.cc:1199] T 83084be89b2646b0bda208b3f6f60d77 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/83084be89b2646b0bda208b3f6f60d77
I20251025 07:52:30.470366 1171 ts_tablet_manager.cc:1950] T 83084be89b2646b0bda208b3f6f60d77 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:30.473723 1167 ts_tablet_manager.cc:1434] T 5caac9c943bf42a5b2ae522d87fdfe8e P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251025 07:52:30.474678 1496 leader_election.cc:290] T 2aa522477e934c89834026dc480fe7c0 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:30.475332 1166 consensus_queue.cc:260] T 90764275825c4ffb8712cb883d37a551 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.476418 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 264) took 1774 ms. Trace:
I20251025 07:52:30.476595 1166 ts_tablet_manager.cc:1434] T 90764275825c4ffb8712cb883d37a551 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.036s user 0.003s sys 0.000s
I20251025 07:52:30.476524 1171 rpcz_store.cc:276] 1025 07:52:28.701748 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:28.703774 (+ 2026us) service_pool.cc:225] Handling call
1025 07:52:30.454364 (+1750590us) ts_tablet_manager.cc:1151] Deleting tablet 83084be89b2646b0bda208b3f6f60d77
1025 07:52:30.454376 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:30.466358 (+ 11982us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:30.469688 (+ 3330us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:30.476357 (+ 6669us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1750394}
I20251025 07:52:30.476876 1171 tablet_replica.cc:333] T 2947502815d742e89cacb474b7fbb249 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:30.476902 1166 tablet_bootstrap.cc:492] T 85e27fc52e9a4d62a8622645594f59c6 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:30.477048 1171 raft_consensus.cc:2243] T 2947502815d742e89cacb474b7fbb249 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:30.477217 1171 raft_consensus.cc:2272] T 2947502815d742e89cacb474b7fbb249 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:30.477726 1171 ts_tablet_manager.cc:1916] T 2947502815d742e89cacb474b7fbb249 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:30.477840 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 83084be89b2646b0bda208b3f6f60d77 (table default.Table_7c2f0e6e0edf4ae6920557e0d5130468 [id=31c55cf000ed4e4dbefe4a407fa1828c]) successfully deleted
I20251025 07:52:30.478606 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2aa522477e934c89834026dc480fe7c0" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:30.478772 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aeaad5c5d01478bb997254ac372d9a6" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:30.479549 983 leader_election.cc:304] T 1aeaad5c5d01478bb997254ac372d9a6 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b
I20251025 07:52:30.482268 892 tablet_service.cc:1552] Processing DeleteTablet for tablet f656945454fe4921ab819c772a089eb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:30.482841 1167 tablet_bootstrap.cc:492] T f51ec427dd9349a48eb221db04cf242b P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:30.486222 1247 raft_consensus.cc:2804] T 1aeaad5c5d01478bb997254ac372d9a6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:30.486925 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 74e2d78c024849b0af5c0bde1f48c36f with delete_type TABLET_DATA_DELETED (replaced by tablet fd265cdbf6a8440393a2b5d8240d1872) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:30.487223 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2aa522477e934c89834026dc480fe7c0" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:30.487445 1500 raft_consensus.cc:515] T 1aeaad5c5d01478bb997254ac372d9a6 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:30.487620 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 74e2d78c024849b0af5c0bde1f48c36f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74e2d78c024849b0af5c0bde1f48c36f already in progress: creating tablet
I20251025 07:52:30.486840 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet f661d5909657455581dcb5e6eadd3c0b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:30.489444 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet f656945454fe4921ab819c772a089eb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:30.489482 1247 raft_consensus.cc:493] T 1aeaad5c5d01478bb997254ac372d9a6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:30.489620 1247 raft_consensus.cc:3060] T 1aeaad5c5d01478bb997254ac372d9a6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:30.491132 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bc2d16b33ca8426abbd7bd38339c9f1d consensus state has no leader
W20251025 07:52:30.492539 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet f661d5909657455581dcb5e6eadd3c0b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f661d5909657455581dcb5e6eadd3c0b already in progress: creating tablet
I20251025 07:52:30.492556 1247 raft_consensus.cc:515] T 1aeaad5c5d01478bb997254ac372d9a6 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:30.492815 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet f656945454fe4921ab819c772a089eb3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f656945454fe4921ab819c772a089eb3 already in progress: creating tablet
I20251025 07:52:30.489617 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 551d07a9242740dcb9366bb2d707a4a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:30.493219 1247 leader_election.cc:290] T 1aeaad5c5d01478bb997254ac372d9a6 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:30.493394 1247 raft_consensus.cc:493] T 567a1f5c216a4a519249da66a2a03ea4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:30.493525 1247 raft_consensus.cc:515] T 567a1f5c216a4a519249da66a2a03ea4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.493959 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aeaad5c5d01478bb997254ac372d9a6" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
W20251025 07:52:30.494947 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 551d07a9242740dcb9366bb2d707a4a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 551d07a9242740dcb9366bb2d707a4a8 already in progress: creating tablet
I20251025 07:52:30.495143 1247 leader_election.cc:290] T 567a1f5c216a4a519249da66a2a03ea4 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:30.495549 982 leader_election.cc:343] T 1aeaad5c5d01478bb997254ac372d9a6 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:30.495823 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "567a1f5c216a4a519249da66a2a03ea4" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:30.496876 519 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet cd23b8094d7e472eba6bd170248858a9 (table default.Table_dfc7f1d4bf8b48bebbb00313419c3fd8 [id=d82b3e1317434cf08a0d51e8a9427224]). Expected version 2 got 0
I20251025 07:52:30.498576 517 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439] alter complete (version 4)
W20251025 07:52:30.502115 982 leader_election.cc:343] T 567a1f5c216a4a519249da66a2a03ea4 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:30.503037 1171 ts_tablet_manager.cc:1929] T 2947502815d742e89cacb474b7fbb249 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:30.503177 1171 log.cc:1199] T 2947502815d742e89cacb474b7fbb249 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/2947502815d742e89cacb474b7fbb249
I20251025 07:52:30.503123 748 tablet_service.cc:1505] Processing CreateTablet for tablet 48ff4ce7846b4e56a37eee9bcb17bd5e (DEFAULT_TABLE table=default.Table_5eaf42285e474860a44c19683f1fa70f [id=0da8afa0245842cfa80b852852e5235a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.503515 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48ff4ce7846b4e56a37eee9bcb17bd5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.503720 1171 ts_tablet_manager.cc:1950] T 2947502815d742e89cacb474b7fbb249 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:30.504501 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 265) took 1800 ms. Trace:
I20251025 07:52:30.504642 1171 rpcz_store.cc:276] 1025 07:52:28.704135 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:28.704176 (+ 41us) service_pool.cc:225] Handling call
1025 07:52:30.476796 (+1772620us) ts_tablet_manager.cc:1151] Deleting tablet 2947502815d742e89cacb474b7fbb249
1025 07:52:30.476804 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:30.498624 (+ 21820us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:30.502906 (+ 4282us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:30.504463 (+ 1557us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1771883}
I20251025 07:52:30.505206 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 266) took 1800 ms. Trace:
I20251025 07:52:30.505352 1171 rpcz_store.cc:276] 1025 07:52:28.705068 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:28.710515 (+ 5447us) service_pool.cc:225] Handling call
1025 07:52:30.505064 (+1794549us) ts_tablet_manager.cc:1151] Deleting tablet 742b3ec3f6604647b9e397532f1673dd
1025 07:52:30.505073 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:30.505160 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1794294}
I20251025 07:52:30.505832 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 267) took 1792 ms. Trace:
I20251025 07:52:30.505939 1171 rpcz_store.cc:276] 1025 07:52:28.713806 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:28.716058 (+ 2252us) service_pool.cc:225] Handling call
1025 07:52:30.505697 (+1789639us) ts_tablet_manager.cc:1151] Deleting tablet 887e7c7a5d0e4aaf88f26246306df19c
1025 07:52:30.505705 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:30.505796 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1789436}
I20251025 07:52:30.506317 1171 tablet_replica.cc:333] T 9bf486d82b984d2c89ee32e84a85894f P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:30.506438 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 48ff4ce7846b4e56a37eee9bcb17bd5e (DEFAULT_TABLE table=default.Table_5eaf42285e474860a44c19683f1fa70f [id=0da8afa0245842cfa80b852852e5235a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.506532 1171 raft_consensus.cc:2243] T 9bf486d82b984d2c89ee32e84a85894f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:30.506693 1171 raft_consensus.cc:2272] T 9bf486d82b984d2c89ee32e84a85894f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:30.506830 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48ff4ce7846b4e56a37eee9bcb17bd5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.509056 517 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: default.Table_293596c64cca4768bc5869fd9924f856 [id=d1b9d89b22bc4431a7d59ceeb9d75050] alter complete (version 2)
I20251025 07:52:30.510823 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 2947502815d742e89cacb474b7fbb249 (table default.Table_c96e62972b3c4ee48b6c3ea90831a34f [id=e7a5b463914840c980a306b7d46ea871]) successfully deleted
W20251025 07:52:30.511101 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 742b3ec3f6604647b9e397532f1673dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 742b3ec3f6604647b9e397532f1673dd already in progress: creating tablet
I20251025 07:52:30.511054 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 716fb61312e94a4d9eb8ab8336ca140c (DEFAULT_TABLE table=default.Table_ba46ca0cf5d941e0879dead1c3544190 [id=3cf1d7f589ff457b9d47c88a1ccc3b2b]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:30.511336 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 887e7c7a5d0e4aaf88f26246306df19c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 887e7c7a5d0e4aaf88f26246306df19c already in progress: creating tablet
I20251025 07:52:30.511453 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 716fb61312e94a4d9eb8ab8336ca140c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.511762 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 896bd7870d87408e8adfdb7a78998111
I20251025 07:52:30.512290 524 catalog_manager.cc:6719] ReplaceTablet: tablet 896bd7870d87408e8adfdb7a78998111 deleted and replaced by tablet e8402fd33ba842e981c6c68787c7c548
I20251025 07:52:30.512998 740 tablet_service.cc:1505] Processing CreateTablet for tablet a0ea1edeca7347fc844faa5e4be1ae94 (DEFAULT_TABLE table=default.Table_7f5a5345567a48d09dedce1df9d63a3b [id=7eaeb32957a946998e90658207639ed7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.513356 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0ea1edeca7347fc844faa5e4be1ae94. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.516207 1020 tablet_service.cc:1505] Processing CreateTablet for tablet 7e6544202d4f489d82f22d6073ef240d (DEFAULT_TABLE table=default.Table_8da73d66704143de8a4b4480c28fd1d9 [id=93fcfeca6dbf48baa3691b7939d0e4ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.516472 739 tablet_service.cc:1505] Processing CreateTablet for tablet 9f039d9107ae493486d799ab6a1d731d (DEFAULT_TABLE table=default.Table_3b10c71d2c1b4bd2849e0cad9ec767d7 [id=ee09c3f16a194bbf97dd2768b5231c06]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.516561 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e6544202d4f489d82f22d6073ef240d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.516805 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f039d9107ae493486d799ab6a1d731d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.517269 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 4f0c4f0e8f014e59be9e46d2afe0be9d (DEFAULT_TABLE table=default.Table_c902adc24467448f9b5d4f585cf614f5 [id=56e169f79a37465d82dd1d1c3c285740]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.517690 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f0c4f0e8f014e59be9e46d2afe0be9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.518522 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet ebc824ec4c60411dab1a2ea4a385148e
I20251025 07:52:30.518957 520 catalog_manager.cc:6719] ReplaceTablet: tablet ebc824ec4c60411dab1a2ea4a385148e deleted and replaced by tablet 29dbaf9149134f548a974a5779f7da67
I20251025 07:52:30.519774 1019 tablet_service.cc:1505] Processing CreateTablet for tablet 682c1e22d7764dce862d970b33b88cb3 (DEFAULT_TABLE table=default.Table_550c910e06364276ae5c4548afa39e6c [id=03294f958266430589e89052d6c729df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.520143 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 682c1e22d7764dce862d970b33b88cb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.520076 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet 896bd7870d87408e8adfdb7a78998111 with delete_type TABLET_DATA_DELETED (replaced by tablet e8402fd33ba842e981c6c68787c7c548) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:30.520990 741 tablet_service.cc:1505] Processing CreateTablet for tablet 4f0c4f0e8f014e59be9e46d2afe0be9d (DEFAULT_TABLE table=default.Table_c902adc24467448f9b5d4f585cf614f5 [id=56e169f79a37465d82dd1d1c3c285740]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.521340 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f0c4f0e8f014e59be9e46d2afe0be9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.521548 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet ebc824ec4c60411dab1a2ea4a385148e with delete_type TABLET_DATA_DELETED (replaced by tablet 29dbaf9149134f548a974a5779f7da67) from {username='slave'} at 127.0.0.1:47408
W20251025 07:52:30.521838 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 896bd7870d87408e8adfdb7a78998111 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 896bd7870d87408e8adfdb7a78998111 already in progress: creating tablet
I20251025 07:52:30.522264 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 567a1f5c216a4a519249da66a2a03ea4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:30.522573 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 1aeaad5c5d01478bb997254ac372d9a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:30.523295 752 tablet_service.cc:1505] Processing CreateTablet for tablet 716fb61312e94a4d9eb8ab8336ca140c (DEFAULT_TABLE table=default.Table_ba46ca0cf5d941e0879dead1c3544190 [id=3cf1d7f589ff457b9d47c88a1ccc3b2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.523653 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 716fb61312e94a4d9eb8ab8336ca140c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.525362 1024 tablet_service.cc:1505] Processing CreateTablet for tablet e9e8fffb8e984463b5fa2c7dd9e93227 (DEFAULT_TABLE table=default.Table_3353358c7aab4407880b823e1500b6d3 [id=70586b36140141d9b7579a32df907ae5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.525708 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9e8fffb8e984463b5fa2c7dd9e93227. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.526392 742 tablet_service.cc:1505] Processing CreateTablet for tablet 7e6544202d4f489d82f22d6073ef240d (DEFAULT_TABLE table=default.Table_8da73d66704143de8a4b4480c28fd1d9 [id=93fcfeca6dbf48baa3691b7939d0e4ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.526754 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e6544202d4f489d82f22d6073ef240d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.516216 1023 tablet_service.cc:1505] Processing CreateTablet for tablet 34c63b69904d499bb0d286941a8303f2 (DEFAULT_TABLE table=default.Table_3709b01b0a9a48499f5bd229e3e9f31a [id=433d4cda7efb41349209f3e82cc41e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.527750 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34c63b69904d499bb0d286941a8303f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.530405 745 tablet_service.cc:1505] Processing CreateTablet for tablet 682c1e22d7764dce862d970b33b88cb3 (DEFAULT_TABLE table=default.Table_550c910e06364276ae5c4548afa39e6c [id=03294f958266430589e89052d6c729df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.530766 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 682c1e22d7764dce862d970b33b88cb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.531232 1021 tablet_service.cc:1505] Processing CreateTablet for tablet 9f039d9107ae493486d799ab6a1d731d (DEFAULT_TABLE table=default.Table_3b10c71d2c1b4bd2849e0cad9ec767d7 [id=ee09c3f16a194bbf97dd2768b5231c06]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.531590 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f039d9107ae493486d799ab6a1d731d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.534268 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 85fba8b37f2c4540aa81701052bf222a with delete_type TABLET_DATA_DELETED (replaced by tablet 6a58931e7cd7454fb583f1c70cd2ad15) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:30.534643 753 tablet_service.cc:1505] Processing CreateTablet for tablet 34c63b69904d499bb0d286941a8303f2 (DEFAULT_TABLE table=default.Table_3709b01b0a9a48499f5bd229e3e9f31a [id=433d4cda7efb41349209f3e82cc41e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.534969 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34c63b69904d499bb0d286941a8303f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.538403 754 tablet_service.cc:1505] Processing CreateTablet for tablet 1b5ac7e845a24b2eb5b2429d47361232 (DEFAULT_TABLE table=default.Table_f6781cf6e9104f51a278cab3d1781a1a [id=8b4207028ba74a5ebd5d4f6690f46893]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.538765 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b5ac7e845a24b2eb5b2429d47361232. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.538857 749 tablet_service.cc:1505] Processing CreateTablet for tablet dda0b26e4f1a44928bc4b84f333901c0 (DEFAULT_TABLE table=default.Table_c1bf6bb7c62c4648853b9a5659c73da0 [id=d0b9f20f28d340f4944e85ee0f563fab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.539258 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dda0b26e4f1a44928bc4b84f333901c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.540601 1017 tablet_service.cc:1505] Processing CreateTablet for tablet 9a8e2658a81e436298fe2190726080d4 (DEFAULT_TABLE table=default.Table_3193d4af1aaa42a28d4eda999726f033 [id=5c3691c247a448eb985d64e94b1e4ac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.541136 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a8e2658a81e436298fe2190726080d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.543699 1016 tablet_service.cc:1505] Processing CreateTablet for tablet 141e413d2faa4c67b4c822192e631417 (DEFAULT_TABLE table=default.Table_25f26dbeafcf4afdaaa118dc47494351 [id=8c364800b3f34fa39a375f2f8196b525]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.544190 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 141e413d2faa4c67b4c822192e631417. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.544780 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 1b5ac7e845a24b2eb5b2429d47361232 (DEFAULT_TABLE table=default.Table_f6781cf6e9104f51a278cab3d1781a1a [id=8b4207028ba74a5ebd5d4f6690f46893]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.545068 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 6d3c9954693d4e8782ee0128ef066aa9 (DEFAULT_TABLE table=default.Table_2070d67c804b4eceb6d29bfb97a42604 [id=2df257976bfe40a68d54f0cf87dc65a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.545610 1018 tablet_service.cc:1505] Processing CreateTablet for tablet a0ea1edeca7347fc844faa5e4be1ae94 (DEFAULT_TABLE table=default.Table_7f5a5345567a48d09dedce1df9d63a3b [id=7eaeb32957a946998e90658207639ed7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.546352 1170 tablet_replica.cc:333] T 567a1f5c216a4a519249da66a2a03ea4 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:30.546563 1170 raft_consensus.cc:2243] T 567a1f5c216a4a519249da66a2a03ea4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:30.546736 1170 raft_consensus.cc:2272] T 567a1f5c216a4a519249da66a2a03ea4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:30.546970 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aeaad5c5d01478bb997254ac372d9a6" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
W20251025 07:52:30.547449 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet ebc824ec4c60411dab1a2ea4a385148e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ebc824ec4c60411dab1a2ea4a385148e already in progress: creating tablet
I20251025 07:52:30.548475 1170 ts_tablet_manager.cc:1916] T 567a1f5c216a4a519249da66a2a03ea4 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:30.545148 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b5ac7e845a24b2eb5b2429d47361232. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.549197 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d3c9954693d4e8782ee0128ef066aa9. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:30.543789 847 leader_election.cc:343] T 2aa522477e934c89834026dc480fe7c0 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:30.551260 1247 raft_consensus.cc:493] T cd23b8094d7e472eba6bd170248858a9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:30.551431 1247 raft_consensus.cc:515] T cd23b8094d7e472eba6bd170248858a9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.551563 914 raft_consensus.cc:2393] T 1aeaad5c5d01478bb997254ac372d9a6 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
W20251025 07:52:30.551788 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:46231, user_credentials={real_user=slave}} blocked reactor thread for 70058us
W20251025 07:52:30.552012 847 leader_election.cc:343] T 1aeaad5c5d01478bb997254ac372d9a6 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:30.552062 1247 leader_election.cc:290] T cd23b8094d7e472eba6bd170248858a9 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:30.553333 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd23b8094d7e472eba6bd170248858a9" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:30.553797 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0ea1edeca7347fc844faa5e4be1ae94. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.553807 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aeaad5c5d01478bb997254ac372d9a6" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:30.554463 881 tablet_service.cc:1505] Processing CreateTablet for tablet 7e6544202d4f489d82f22d6073ef240d (DEFAULT_TABLE table=default.Table_8da73d66704143de8a4b4480c28fd1d9 [id=93fcfeca6dbf48baa3691b7939d0e4ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.554735 884 tablet_service.cc:1505] Processing CreateTablet for tablet 9f039d9107ae493486d799ab6a1d731d (DEFAULT_TABLE table=default.Table_3b10c71d2c1b4bd2849e0cad9ec767d7 [id=ee09c3f16a194bbf97dd2768b5231c06]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.554936 890 tablet_service.cc:1505] Processing CreateTablet for tablet e9e8fffb8e984463b5fa2c7dd9e93227 (DEFAULT_TABLE table=default.Table_3353358c7aab4407880b823e1500b6d3 [id=70586b36140141d9b7579a32df907ae5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.555192 1496 raft_consensus.cc:493] T 567a1f5c216a4a519249da66a2a03ea4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:30.555222 875 tablet_service.cc:1505] Processing CreateTablet for tablet 4f0c4f0e8f014e59be9e46d2afe0be9d (DEFAULT_TABLE table=default.Table_c902adc24467448f9b5d4f585cf614f5 [id=56e169f79a37465d82dd1d1c3c285740]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.555316 882 tablet_service.cc:1505] Processing CreateTablet for tablet 1b5ac7e845a24b2eb5b2429d47361232 (DEFAULT_TABLE table=default.Table_f6781cf6e9104f51a278cab3d1781a1a [id=8b4207028ba74a5ebd5d4f6690f46893]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.555361 1496 raft_consensus.cc:515] T 567a1f5c216a4a519249da66a2a03ea4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.555585 892 tablet_service.cc:1505] Processing CreateTablet for tablet a0ea1edeca7347fc844faa5e4be1ae94 (DEFAULT_TABLE table=default.Table_7f5a5345567a48d09dedce1df9d63a3b [id=7eaeb32957a946998e90658207639ed7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.555616 885 tablet_service.cc:1505] Processing CreateTablet for tablet 682c1e22d7764dce862d970b33b88cb3 (DEFAULT_TABLE table=default.Table_550c910e06364276ae5c4548afa39e6c [id=03294f958266430589e89052d6c729df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.554457 894 tablet_service.cc:1505] Processing CreateTablet for tablet dda0b26e4f1a44928bc4b84f333901c0 (DEFAULT_TABLE table=default.Table_c1bf6bb7c62c4648853b9a5659c73da0 [id=d0b9f20f28d340f4944e85ee0f563fab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.555917 888 tablet_service.cc:1505] Processing CreateTablet for tablet 34c63b69904d499bb0d286941a8303f2 (DEFAULT_TABLE table=default.Table_3709b01b0a9a48499f5bd229e3e9f31a [id=433d4cda7efb41349209f3e82cc41e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.555989 1496 leader_election.cc:290] T 567a1f5c216a4a519249da66a2a03ea4 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:30.556170 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 896bd7870d87408e8adfdb7a78998111 with delete_type TABLET_DATA_DELETED (replaced by tablet e8402fd33ba842e981c6c68787c7c548) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:30.556643 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e6544202d4f489d82f22d6073ef240d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.556620 880 tablet_service.cc:1505] Processing CreateTablet for tablet 48ff4ce7846b4e56a37eee9bcb17bd5e (DEFAULT_TABLE table=default.Table_5eaf42285e474860a44c19683f1fa70f [id=0da8afa0245842cfa80b852852e5235a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.558692 891 tablet_service.cc:1505] Processing CreateTablet for tablet 716fb61312e94a4d9eb8ab8336ca140c (DEFAULT_TABLE table=default.Table_ba46ca0cf5d941e0879dead1c3544190 [id=3cf1d7f589ff457b9d47c88a1ccc3b2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.558964 1500 leader_election.cc:290] T 1aeaad5c5d01478bb997254ac372d9a6 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:30.559691 876 tablet_service.cc:1552] Processing DeleteTablet for tablet ebc824ec4c60411dab1a2ea4a385148e with delete_type TABLET_DATA_DELETED (replaced by tablet 29dbaf9149134f548a974a5779f7da67) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:30.560088 889 tablet_service.cc:1505] Processing CreateTablet for tablet c11f674385184ae6b235f1db8bf1fb91 (DEFAULT_TABLE table=default.Table_ce27d8ed3ede46579f2ed3ae81536572 [id=c4c41c3b236141eb8e03dbb171f6b56d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.560488 877 tablet_service.cc:1505] Processing CreateTablet for tablet 9a8e2658a81e436298fe2190726080d4 (DEFAULT_TABLE table=default.Table_3193d4af1aaa42a28d4eda999726f033 [id=5c3691c247a448eb985d64e94b1e4ac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.560511 520 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 627a10a0714f4be8bdbced503fd6388d (table default.Table_6b937442c9614efeb8cc1562ac306174 [id=7fc07b0c33f74604be526c4a0df949e9]). Expected version 1 got 0
I20251025 07:52:30.560889 879 tablet_service.cc:1505] Processing CreateTablet for tablet 6d3c9954693d4e8782ee0128ef066aa9 (DEFAULT_TABLE table=default.Table_2070d67c804b4eceb6d29bfb97a42604 [id=2df257976bfe40a68d54f0cf87dc65a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.561390 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 567a1f5c216a4a519249da66a2a03ea4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:30.562530 1026 tablet_service.cc:1505] Processing CreateTablet for tablet dda0b26e4f1a44928bc4b84f333901c0 (DEFAULT_TABLE table=default.Table_c1bf6bb7c62c4648853b9a5659c73da0 [id=d0b9f20f28d340f4944e85ee0f563fab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.562944 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dda0b26e4f1a44928bc4b84f333901c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.563827 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 896bd7870d87408e8adfdb7a78998111 with delete_type TABLET_DATA_DELETED (replaced by tablet e8402fd33ba842e981c6c68787c7c548) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:30.563880 744 tablet_service.cc:1552] Processing DeleteTablet for tablet ebc824ec4c60411dab1a2ea4a385148e with delete_type TABLET_DATA_DELETED (replaced by tablet 29dbaf9149134f548a974a5779f7da67) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:30.564061 737 tablet_service.cc:1552] Processing DeleteTablet for tablet 5caac9c943bf42a5b2ae522d87fdfe8e with delete_type TABLET_DATA_DELETED (replaced by tablet 4f25c9caa05b4da4bd56a8a21fde423c) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:30.565069 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "567a1f5c216a4a519249da66a2a03ea4" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
W20251025 07:52:30.565507 982 leader_election.cc:343] T cd23b8094d7e472eba6bd170248858a9 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:30.565069 744 tablet_service.cc:1505] Processing CreateTablet for tablet 6d3c9954693d4e8782ee0128ef066aa9 (DEFAULT_TABLE table=default.Table_2070d67c804b4eceb6d29bfb97a42604 [id=2df257976bfe40a68d54f0cf87dc65a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.565904 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d3c9954693d4e8782ee0128ef066aa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.566816 738 tablet_service.cc:1505] Processing CreateTablet for tablet c11f674385184ae6b235f1db8bf1fb91 (DEFAULT_TABLE table=default.Table_ce27d8ed3ede46579f2ed3ae81536572 [id=c4c41c3b236141eb8e03dbb171f6b56d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.567174 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c11f674385184ae6b235f1db8bf1fb91. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.568099 983 leader_election.cc:304] T 1aeaad5c5d01478bb997254ac372d9a6 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:30.569069 737 tablet_service.cc:1505] Processing CreateTablet for tablet 9a8e2658a81e436298fe2190726080d4 (DEFAULT_TABLE table=default.Table_3193d4af1aaa42a28d4eda999726f033 [id=5c3691c247a448eb985d64e94b1e4ac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.569832 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_cfa54e1a04154e2eb8747fd3a4812883" } new_table_name: "default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95" modify_external_catalogs: true
I20251025 07:52:30.571077 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9e8fffb8e984463b5fa2c7dd9e93227. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.571137 1247 raft_consensus.cc:2749] T 1aeaad5c5d01478bb997254ac372d9a6 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251025 07:52:30.573532 1171 ts_tablet_manager.cc:1916] T 9bf486d82b984d2c89ee32e84a85894f P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:30.577327 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "567a1f5c216a4a519249da66a2a03ea4" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:30.577816 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a8e2658a81e436298fe2190726080d4. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:30.579778 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 04d27d54fdb7483b869408af65ff85c9 consensus state has no leader
I20251025 07:52:30.582283 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c11f674385184ae6b235f1db8bf1fb91. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.582321 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd23b8094d7e472eba6bd170248858a9" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:30.582929 1218 raft_consensus.cc:493] T 8c6ff4c73e624af3bf2de36c4493b462 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:30.583437 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34c63b69904d499bb0d286941a8303f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.583724 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet 85e27fc52e9a4d62a8622645594f59c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47408
W20251025 07:52:30.584287 847 leader_election.cc:343] T 1aeaad5c5d01478bb997254ac372d9a6 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:30.584873 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0ea1edeca7347fc844faa5e4be1ae94. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.585678 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b5ac7e845a24b2eb5b2429d47361232. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.586663 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dda0b26e4f1a44928bc4b84f333901c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.588001 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d3c9954693d4e8782ee0128ef066aa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.589170 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f0c4f0e8f014e59be9e46d2afe0be9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.589401 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f039d9107ae493486d799ab6a1d731d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:30.590396 847 leader_election.cc:343] T 567a1f5c216a4a519249da66a2a03ea4 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:30.589203 1218 raft_consensus.cc:515] T 8c6ff4c73e624af3bf2de36c4493b462 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.592850 913 raft_consensus.cc:2468] T cd23b8094d7e472eba6bd170248858a9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:30.593654 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_1de0b151cf4c4c10aca35e45623caf2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:30.593782 914 raft_consensus.cc:2468] T 567a1f5c216a4a519249da66a2a03ea4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:30.593782 983 leader_election.cc:304] T cd23b8094d7e472eba6bd170248858a9 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b
I20251025 07:52:30.594105 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48ff4ce7846b4e56a37eee9bcb17bd5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.594254 983 leader_election.cc:304] T 567a1f5c216a4a519249da66a2a03ea4 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b
W20251025 07:52:30.594322 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1de0b151cf4c4c10aca35e45623caf2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:30.594430 983 raft_consensus.cc:2686] T 567a1f5c216a4a519249da66a2a03ea4 P db42be7ad3c741a0aa427f27e245993b: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251025 07:52:30.595486 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_618540173e8a45828c270476360228f0" } new_table_name: "default.Table_14e4b2d5ef6240d6a0e17581b7785e69" modify_external_catalogs: true
I20251025 07:52:30.595647 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_8da73d66704143de8a4b4480c28fd1d9" } modify_external_catalogs: true
I20251025 07:52:30.595826 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_8da73d66704143de8a4b4480c28fd1d9" } modify_external_catalogs: true
I20251025 07:52:30.596975 1247 raft_consensus.cc:2804] T cd23b8094d7e472eba6bd170248858a9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:30.597084 1247 raft_consensus.cc:493] T cd23b8094d7e472eba6bd170248858a9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:30.597174 1247 raft_consensus.cc:3060] T cd23b8094d7e472eba6bd170248858a9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:30.590478 1166 tablet_bootstrap.cc:654] T 85e27fc52e9a4d62a8622645594f59c6 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:30.599895 1247 raft_consensus.cc:515] T cd23b8094d7e472eba6bd170248858a9 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.607241 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd23b8094d7e472eba6bd170248858a9" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:30.607463 914 raft_consensus.cc:3060] T cd23b8094d7e472eba6bd170248858a9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:30.607717 1247 leader_election.cc:290] T cd23b8094d7e472eba6bd170248858a9 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:30.613268 1172 ts_tablet_manager.cc:1929] T 1b31177a062b467187d68ed2034f5690 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:30.613425 1172 log.cc:1199] T 1b31177a062b467187d68ed2034f5690 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/1b31177a062b467187d68ed2034f5690
I20251025 07:52:30.613502 914 raft_consensus.cc:2468] T cd23b8094d7e472eba6bd170248858a9 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:30.613823 1172 ts_tablet_manager.cc:1950] T 1b31177a062b467187d68ed2034f5690 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:30.613901 983 leader_election.cc:304] T cd23b8094d7e472eba6bd170248858a9 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:30.614151 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 253) took 1815 ms. Trace:
I20251025 07:52:30.614233 1247 raft_consensus.cc:2804] T cd23b8094d7e472eba6bd170248858a9 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:30.614353 1247 raft_consensus.cc:697] T cd23b8094d7e472eba6bd170248858a9 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:30.614250 1172 rpcz_store.cc:276] 1025 07:52:28.798615 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:28.883819 (+ 85204us) service_pool.cc:225] Handling call
1025 07:52:30.263637 (+1379818us) ts_tablet_manager.cc:1151] Deleting tablet 1b31177a062b467187d68ed2034f5690
1025 07:52:30.263646 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:30.470105 (+206459us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:30.613128 (+143023us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:30.614100 (+ 972us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1341656}
I20251025 07:52:30.614718 1218 leader_election.cc:290] T 8c6ff4c73e624af3bf2de36c4493b462 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:30.614624 1247 consensus_queue.cc:237] T cd23b8094d7e472eba6bd170248858a9 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:30.615276 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 92aa605fabbc4257a6a204c087e7cc7f consensus state has no leader
I20251025 07:52:30.615686 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 7e6544202d4f489d82f22d6073ef240d
I20251025 07:52:30.616438 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 279) took 1634 ms. Trace:
I20251025 07:52:30.616575 1172 rpcz_store.cc:276] 1025 07:52:28.982184 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:29.015042 (+ 32858us) service_pool.cc:225] Handling call
1025 07:52:30.614496 (+1599454us) ts_tablet_manager.cc:1151] Deleting tablet 80ee75935d7e461d9d4066fac9819bef
1025 07:52:30.614505 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:30.616394 (+ 1889us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1599050}
I20251025 07:52:30.616971 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 280) took 1634 ms. Trace:
I20251025 07:52:30.617028 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e6544202d4f489d82f22d6073ef240d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:30.617077 1172 rpcz_store.cc:276] 1025 07:52:28.982325 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:29.015503 (+ 33178us) service_pool.cc:225] Handling call
1025 07:52:30.616837 (+1601334us) ts_tablet_manager.cc:1151] Deleting tablet 71432a6848944a0da06ef9b81f160dc4
1025 07:52:30.616845 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:30.616932 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1514872}
I20251025 07:52:30.617416 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 281) took 1635 ms. Trace:
I20251025 07:52:30.617520 1172 rpcz_store.cc:276] 1025 07:52:28.982421 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:29.071202 (+ 88781us) service_pool.cc:225] Handling call
1025 07:52:30.617283 (+1546081us) ts_tablet_manager.cc:1151] Deleting tablet b3a0af893a8947848e12093d0a7f0229
1025 07:52:30.617293 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:30.617388 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1503691}
I20251025 07:52:30.617847 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 286) took 1494 ms. Trace:
I20251025 07:52:30.617944 1172 rpcz_store.cc:276] 1025 07:52:29.123700 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:29.127244 (+ 3544us) service_pool.cc:225] Handling call
1025 07:52:30.617721 (+1490477us) ts_tablet_manager.cc:1151] Deleting tablet b947ecc352e54f4caa73c6ca97372256
1025 07:52:30.617731 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:30.617806 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1490261}
I20251025 07:52:30.618275 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 282) took 1495 ms. Trace:
I20251025 07:52:30.618363 1172 rpcz_store.cc:276] 1025 07:52:29.123192 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:29.148296 (+ 25104us) service_pool.cc:225] Handling call
1025 07:52:30.618148 (+1469852us) ts_tablet_manager.cc:1151] Deleting tablet aaef24ccdfba45fe8204327723df834e
1025 07:52:30.618159 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:30.618235 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1469600}
I20251025 07:52:30.618608 1172 tablet_replica.cc:333] T f51ec427dd9349a48eb221db04cf242b P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:30.618834 1172 raft_consensus.cc:2243] T f51ec427dd9349a48eb221db04cf242b P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:30.618997 1172 raft_consensus.cc:2272] T f51ec427dd9349a48eb221db04cf242b P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:30.619570 1172 ts_tablet_manager.cc:1916] T f51ec427dd9349a48eb221db04cf242b P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:30.621392 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_39bd22f18d6540d3be94520671683032" } modify_external_catalogs: true
I20251025 07:52:30.621595 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_39bd22f18d6540d3be94520671683032" } modify_external_catalogs: true
I20251025 07:52:30.629634 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd23b8094d7e472eba6bd170248858a9" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
W20251025 07:52:30.630256 982 leader_election.cc:343] T cd23b8094d7e472eba6bd170248858a9 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:30.630268 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c6ff4c73e624af3bf2de36c4493b462" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:30.630436 775 raft_consensus.cc:2468] T 8c6ff4c73e624af3bf2de36c4493b462 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:30.630991 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_1ff54f4df7a2444489ae4e4060084156" } modify_external_catalogs: true
I20251025 07:52:30.631165 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_1ff54f4df7a2444489ae4e4060084156" } modify_external_catalogs: true
I20251025 07:52:30.632818 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_3b25f9f08fa04ed5a0d2a758b95e8bbc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:30.633343 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b25f9f08fa04ed5a0d2a758b95e8bbc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:30.633513 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_32f8a762ec25446dac28e2729d6879bf" } modify_external_catalogs: true
I20251025 07:52:30.633687 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_32f8a762ec25446dac28e2729d6879bf" } modify_external_catalogs: true
I20251025 07:52:30.635183 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_e42a1f7287a34753b7e4daf36893d9c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:30.635685 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e42a1f7287a34753b7e4daf36893d9c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:30.637006 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a8e2658a81e436298fe2190726080d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.640045 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 682c1e22d7764dce862d970b33b88cb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.643620 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_4d3dff3c6fda405f84600e93d3818380"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:30.644178 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d3dff3c6fda405f84600e93d3818380 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:30.645376 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_8129e15bf5c549bd853c04b227a7ef26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:30.645823 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8129e15bf5c549bd853c04b227a7ef26 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:30.646368 1167 tablet_bootstrap.cc:654] T f51ec427dd9349a48eb221db04cf242b P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:30.647099 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_14e4b2d5ef6240d6a0e17581b7785e69" } new_table_name: "default.Table_78724078b4574f35879fd2db388fc4f4" modify_external_catalogs: true
I20251025 07:52:30.651213 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 716fb61312e94a4d9eb8ab8336ca140c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.654453 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_5eaf42285e474860a44c19683f1fa70f" } modify_external_catalogs: true
I20251025 07:52:30.656432 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_5eaf42285e474860a44c19683f1fa70f" } modify_external_catalogs: true
I20251025 07:52:30.660023 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_8e00811c122448288ec358cc8e7867fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:30.660567 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e00811c122448288ec358cc8e7867fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:30.661449 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aeaad5c5d01478bb997254ac372d9a6" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:30.661710 1048 raft_consensus.cc:2393] T 1aeaad5c5d01478bb997254ac372d9a6 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3aea2869ff814cc7b142012faa274382 in current term 1: Already voted for candidate db42be7ad3c741a0aa427f27e245993b in this term.
I20251025 07:52:30.662779 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "567a1f5c216a4a519249da66a2a03ea4" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:30.663369 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c6ff4c73e624af3bf2de36c4493b462" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:30.667371 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 2b594d4718344aafbdaec34040695589 (DEFAULT_TABLE table=default.Table_e08a3ff70f3a411e82b4bb292d768961 [id=8d075a6652b14a37a2bfbfddc6139eb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.667765 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b594d4718344aafbdaec34040695589. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.668421 1170 ts_tablet_manager.cc:1929] T 567a1f5c216a4a519249da66a2a03ea4 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:30.668571 1170 log.cc:1199] T 567a1f5c216a4a519249da66a2a03ea4 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/567a1f5c216a4a519249da66a2a03ea4
I20251025 07:52:30.669021 1170 ts_tablet_manager.cc:1950] T 567a1f5c216a4a519249da66a2a03ea4 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:30.670960 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 1aeba18be0cb4eb9949aeaf6ad73a812 (DEFAULT_TABLE table=default.Table_bd168da90c704b7987fc731f5ab28045 [id=e7683b98787e4d55b81303f4b58e4a6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.673708 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_4a7945d6aa00429b98f77ca4b9f3e3bc" } new_table_name: "default.Table_b210463b66db4877b9a791f7cd741492" modify_external_catalogs: true
I20251025 07:52:30.674460 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_c4716673eec04f3baa6fc852c668fed8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:30.675048 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4716673eec04f3baa6fc852c668fed8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:30.675124 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 5b6383c99e9e4c90a374da9ccd830a6f
I20251025 07:52:30.676795 1028 tablet_service.cc:1505] Processing CreateTablet for tablet fd265cdbf6a8440393a2b5d8240d1872 (DEFAULT_TABLE table=default.Table_434ef1ade0a14af987864563993afae8 [id=3a3af35e11944a6f97fc0144cb9d08d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.677147 1022 tablet_service.cc:1505] Processing CreateTablet for tablet c501ef5d70e742598c1a902efe47feb1 (DEFAULT_TABLE table=default.Table_9c39679c7a244c7f9dd7d0f97f16ccb3 [id=ad812862698d421184ebf501ce79b7be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.677348 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b6383c99e9e4c90a374da9ccd830a6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:30.677726 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_dbe39538a228480a9ff9cd45159e39f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:30.678202 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 410ef150a7014c6c867a021b544a2f4e
W20251025 07:52:30.678238 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dbe39538a228480a9ff9cd45159e39f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:30.679292 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 410ef150a7014c6c867a021b544a2f4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:30.680521 1018 tablet_service.cc:1505] Processing CreateTablet for tablet 1068f566477e4c6db7dde566fff4c2fb (DEFAULT_TABLE table=default.Table_7ce70fea0fe04c52a62033afb8eef977 [id=a8b2a16fd037435893c8f60e92fd1627]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.681033 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 567a1f5c216a4a519249da66a2a03ea4 (table default.Table_b6cf4ff8abe642088cedf32609854bdd [id=e55f743deb3c45f9a90c2f3a0ab68a3c]) successfully deleted
I20251025 07:52:30.681450 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1068f566477e4c6db7dde566fff4c2fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.681546 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet b94634ac5077439fadfbbce569f60e8b
I20251025 07:52:30.682535 525 catalog_manager.cc:5649] T cd23b8094d7e472eba6bd170248858a9 P db42be7ad3c741a0aa427f27e245993b reported cstate change: term changed from 0 to 1, leader changed from <none> to db42be7ad3c741a0aa427f27e245993b (127.0.104.3). New cstate: current_term: 1 leader_uuid: "db42be7ad3c741a0aa427f27e245993b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: UNKNOWN } } }
I20251025 07:52:30.682865 1170 tablet_replica.cc:333] T 1aeaad5c5d01478bb997254ac372d9a6 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:30.682966 525 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet cd23b8094d7e472eba6bd170248858a9 (table default.Table_dfc7f1d4bf8b48bebbb00313419c3fd8 [id=d82b3e1317434cf08a0d51e8a9427224]). Expected version 2 got 0
I20251025 07:52:30.683058 1170 raft_consensus.cc:2243] T 1aeaad5c5d01478bb997254ac372d9a6 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:30.683262 1170 raft_consensus.cc:2272] T 1aeaad5c5d01478bb997254ac372d9a6 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:30.683830 1170 ts_tablet_manager.cc:1916] T 1aeaad5c5d01478bb997254ac372d9a6 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:30.685307 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet b94634ac5077439fadfbbce569f60e8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:30.685561 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c501ef5d70e742598c1a902efe47feb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.689407 1023 tablet_service.cc:1505] Processing CreateTablet for tablet fa826b85f0c94150a17d4e417dd3d62f (DEFAULT_TABLE table=default.Table_0b63d3c003db48709227445b0a0b84ca [id=b2486c1de45d47eeb272a605ce7fdf84]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.689787 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa826b85f0c94150a17d4e417dd3d62f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.691373 1020 tablet_service.cc:1505] Processing CreateTablet for tablet c187a40f8ff14c6a8c184fbeb030ad13 (DEFAULT_TABLE table=default.Table_34b975902f884c658093a51baa54b095 [id=ec0cc315ecba4988a97d68f00ff22697]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.691684 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c187a40f8ff14c6a8c184fbeb030ad13. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.694257 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd265cdbf6a8440393a2b5d8240d1872. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.698441 1017 tablet_service.cc:1505] Processing CreateTablet for tablet dff7f86ae8924685b26011e504e2247b (DEFAULT_TABLE table=default.Table_dbface3af7f6413d9060b881447d34b3 [id=f74f5abb40284e08a990f9caaa828636]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.698797 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dff7f86ae8924685b26011e504e2247b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.699709 1169 tablet_bootstrap.cc:492] T 47284f96928749e6a5201103a8ee5f9f P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:30.702483 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aeba18be0cb4eb9949aeaf6ad73a812. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.707538 1170 ts_tablet_manager.cc:1929] T 1aeaad5c5d01478bb997254ac372d9a6 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:30.707671 1170 log.cc:1199] T 1aeaad5c5d01478bb997254ac372d9a6 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/1aeaad5c5d01478bb997254ac372d9a6
I20251025 07:52:30.708072 1170 ts_tablet_manager.cc:1950] T 1aeaad5c5d01478bb997254ac372d9a6 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:30.709322 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 1aeaad5c5d01478bb997254ac372d9a6 (table default.Table_ebcfacbbec3147a189ff08cec113fc55 [id=8383812e49d24d98bf0ef5f985376d98]) successfully deleted
I20251025 07:52:30.712028 1170 tablet_replica.cc:333] T 85e27fc52e9a4d62a8622645594f59c6 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:30.712239 1170 raft_consensus.cc:2243] T 85e27fc52e9a4d62a8622645594f59c6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:30.712402 1170 raft_consensus.cc:2272] T 85e27fc52e9a4d62a8622645594f59c6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:30.717986 1170 ts_tablet_manager.cc:1916] T 85e27fc52e9a4d62a8622645594f59c6 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:30.720598 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 1aeaad5c5d01478bb997254ac372d9a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:30.723291 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 90764275825c4ffb8712cb883d37a551 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:30.725504 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e6544202d4f489d82f22d6073ef240d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:56590
W20251025 07:52:30.726661 850 leader_election.cc:343] T 2aa522477e934c89834026dc480fe7c0 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:30.727355 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_262a9d20d20840aa85bb3dc06aa5dde6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:30.728124 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_262a9d20d20840aa85bb3dc06aa5dde6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:30.729687 1169 ts_tablet_manager.cc:1403] T 47284f96928749e6a5201103a8ee5f9f P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.333s user 0.008s sys 0.003s
I20251025 07:52:30.731567 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_4b70d9acf0f64bcd992a894b129927c3" } new_table_name: "default.Table_ba4b8cc7b930482690b7e192e0edff12" modify_external_catalogs: true
I20251025 07:52:30.733078 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_c8e45b61c6d4479a99fede45d02dbc1d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:30.733701 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c8e45b61c6d4479a99fede45d02dbc1d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:30.737455 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 92aa605fabbc4257a6a204c087e7cc7f consensus state has no leader
I20251025 07:52:30.737913 879 tablet_service.cc:1505] Processing CreateTablet for tablet 141e413d2faa4c67b4c822192e631417 (DEFAULT_TABLE table=default.Table_25f26dbeafcf4afdaaa118dc47494351 [id=8c364800b3f34fa39a375f2f8196b525]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.738375 850 leader_election.cc:304] T 2aa522477e934c89834026dc480fe7c0 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:30.744143 1218 raft_consensus.cc:2749] T 2aa522477e934c89834026dc480fe7c0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:30.744388 894 tablet_service.cc:1505] Processing CreateTablet for tablet 2b594d4718344aafbdaec34040695589 (DEFAULT_TABLE table=default.Table_e08a3ff70f3a411e82b4bb292d768961 [id=8d075a6652b14a37a2bfbfddc6139eb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.744477 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b6383c99e9e4c90a374da9ccd830a6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:56590
W20251025 07:52:30.744451 850 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:45417, user_credentials={real_user=slave}} blocked reactor thread for 80193.2us
I20251025 07:52:30.745024 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 141e413d2faa4c67b4c822192e631417. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.745587 1169 raft_consensus.cc:359] T 47284f96928749e6a5201103a8ee5f9f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.745807 1169 raft_consensus.cc:385] T 47284f96928749e6a5201103a8ee5f9f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:30.745901 1169 raft_consensus.cc:740] T 47284f96928749e6a5201103a8ee5f9f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:30.746176 1169 consensus_queue.cc:260] T 47284f96928749e6a5201103a8ee5f9f P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.746905 1169 ts_tablet_manager.cc:1434] T 47284f96928749e6a5201103a8ee5f9f P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20251025 07:52:30.747290 1169 tablet_bootstrap.cc:492] T 2aa522477e934c89834026dc480fe7c0 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:30.749269 1169 tablet_bootstrap.cc:654] T 2aa522477e934c89834026dc480fe7c0 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:30.750263 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 410ef150a7014c6c867a021b544a2f4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:30.750365 878 tablet_service.cc:1505] Processing CreateTablet for tablet 1aeba18be0cb4eb9949aeaf6ad73a812 (DEFAULT_TABLE table=default.Table_bd168da90c704b7987fc731f5ab28045 [id=e7683b98787e4d55b81303f4b58e4a6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.750720 893 tablet_service.cc:1505] Processing CreateTablet for tablet fd265cdbf6a8440393a2b5d8240d1872 (DEFAULT_TABLE table=default.Table_434ef1ade0a14af987864563993afae8 [id=3a3af35e11944a6f97fc0144cb9d08d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.752398 886 tablet_service.cc:1505] Processing CreateTablet for tablet c501ef5d70e742598c1a902efe47feb1 (DEFAULT_TABLE table=default.Table_9c39679c7a244c7f9dd7d0f97f16ccb3 [id=ad812862698d421184ebf501ce79b7be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.752699 850 leader_election.cc:304] T 1aeaad5c5d01478bb997254ac372d9a6 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:30.753587 881 tablet_service.cc:1552] Processing DeleteTablet for tablet b94634ac5077439fadfbbce569f60e8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:30.754839 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b594d4718344aafbdaec34040695589. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:30.756105 850 leader_election.cc:343] T 567a1f5c216a4a519249da66a2a03ea4 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: Tablet not RUNNING: STOPPED
I20251025 07:52:30.756237 850 leader_election.cc:304] T 567a1f5c216a4a519249da66a2a03ea4 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:30.757275 1530 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8603bddd8a084ff7a73a25641768dfa9. Latest consensus state: current_term: 1 leader_uuid: "8603bddd8a084ff7a73a25641768dfa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } } }
I20251025 07:52:30.757407 1514 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8603bddd8a084ff7a73a25641768dfa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } } }
I20251025 07:52:30.757642 1530 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 07:52:30.757742 1514 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 07:52:30.760675 1218 raft_consensus.cc:2749] T 1aeaad5c5d01478bb997254ac372d9a6 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251025 07:52:30.761047 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aeba18be0cb4eb9949aeaf6ad73a812. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:30.762346 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 80ee75935d7e461d9d4066fac9819bef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80ee75935d7e461d9d4066fac9819bef already in progress: creating tablet
W20251025 07:52:30.762630 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 71432a6848944a0da06ef9b81f160dc4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71432a6848944a0da06ef9b81f160dc4 already in progress: creating tablet
W20251025 07:52:30.762832 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet b3a0af893a8947848e12093d0a7f0229 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3a0af893a8947848e12093d0a7f0229 already in progress: creating tablet
W20251025 07:52:30.763054 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet b947ecc352e54f4caa73c6ca97372256 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b947ecc352e54f4caa73c6ca97372256 already in progress: creating tablet
W20251025 07:52:30.763236 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet aaef24ccdfba45fe8204327723df834e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aaef24ccdfba45fe8204327723df834e already in progress: creating tablet
I20251025 07:52:30.763710 887 tablet_service.cc:1505] Processing CreateTablet for tablet fa826b85f0c94150a17d4e417dd3d62f (DEFAULT_TABLE table=default.Table_0b63d3c003db48709227445b0a0b84ca [id=b2486c1de45d47eeb272a605ce7fdf84]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.766093 1500 raft_consensus.cc:2749] T 567a1f5c216a4a519249da66a2a03ea4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:30.766613 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd265cdbf6a8440393a2b5d8240d1872. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.767323 881 tablet_service.cc:1505] Processing CreateTablet for tablet 1068f566477e4c6db7dde566fff4c2fb (DEFAULT_TABLE table=default.Table_7ce70fea0fe04c52a62033afb8eef977 [id=a8b2a16fd037435893c8f60e92fd1627]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:30.770112 850 leader_election.cc:343] T 8c6ff4c73e624af3bf2de36c4493b462 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:30.775724 883 tablet_service.cc:1505] Processing CreateTablet for tablet dff7f86ae8924685b26011e504e2247b (DEFAULT_TABLE table=default.Table_dbface3af7f6413d9060b881447d34b3 [id=f74f5abb40284e08a990f9caaa828636]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.775997 518 master_service.cc:954] ReplaceTablet: received request to replace tablet 50ffbcdd4a9247bf99b8e2361bc02ef5 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:30.778601 876 tablet_service.cc:1505] Processing CreateTablet for tablet c187a40f8ff14c6a8c184fbeb030ad13 (DEFAULT_TABLE table=default.Table_34b975902f884c658093a51baa54b095 [id=ec0cc315ecba4988a97d68f00ff22697]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.787163 847 leader_election.cc:304] T 8c6ff4c73e624af3bf2de36c4493b462 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:30.788225 1171 ts_tablet_manager.cc:1929] T 9bf486d82b984d2c89ee32e84a85894f P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:30.790311 1171 log.cc:1199] T 9bf486d82b984d2c89ee32e84a85894f P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/9bf486d82b984d2c89ee32e84a85894f
I20251025 07:52:30.790724 1171 ts_tablet_manager.cc:1950] T 9bf486d82b984d2c89ee32e84a85894f P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:30.791527 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 270) took 2062 ms. Trace:
I20251025 07:52:30.791621 1171 rpcz_store.cc:276] 1025 07:52:28.729247 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:28.729337 (+ 90us) service_pool.cc:225] Handling call
1025 07:52:30.506263 (+1776926us) ts_tablet_manager.cc:1151] Deleting tablet 9bf486d82b984d2c89ee32e84a85894f
1025 07:52:30.506270 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:30.637101 (+130831us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:30.788103 (+151002us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:30.791471 (+ 3368us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1776617}
I20251025 07:52:30.791976 1171 tablet_replica.cc:333] T d0fd00baa02d48278afaa5b8c19182f0 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:30.792150 1171 raft_consensus.cc:2243] T d0fd00baa02d48278afaa5b8c19182f0 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:30.792323 1171 raft_consensus.cc:2272] T d0fd00baa02d48278afaa5b8c19182f0 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:30.792837 1171 ts_tablet_manager.cc:1916] T d0fd00baa02d48278afaa5b8c19182f0 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:30.793226 1500 raft_consensus.cc:2804] T 8c6ff4c73e624af3bf2de36c4493b462 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:30.793368 1500 raft_consensus.cc:493] T 8c6ff4c73e624af3bf2de36c4493b462 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:30.793380 755 tablet_service.cc:1505] Processing CreateTablet for tablet e9e8fffb8e984463b5fa2c7dd9e93227 (DEFAULT_TABLE table=default.Table_3353358c7aab4407880b823e1500b6d3 [id=70586b36140141d9b7579a32df907ae5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.795579 739 tablet_service.cc:1505] Processing CreateTablet for tablet 141e413d2faa4c67b4c822192e631417 (DEFAULT_TABLE table=default.Table_25f26dbeafcf4afdaaa118dc47494351 [id=8c364800b3f34fa39a375f2f8196b525]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.796185 742 tablet_service.cc:1505] Processing CreateTablet for tablet 2b594d4718344aafbdaec34040695589 (DEFAULT_TABLE table=default.Table_e08a3ff70f3a411e82b4bb292d768961 [id=8d075a6652b14a37a2bfbfddc6139eb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.796990 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c501ef5d70e742598c1a902efe47feb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.802472 1166 tablet_bootstrap.cc:492] T 85e27fc52e9a4d62a8622645594f59c6 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:30.759467 1170 ts_tablet_manager.cc:1929] T 85e27fc52e9a4d62a8622645594f59c6 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:30.803037 1170 log.cc:1199] T 85e27fc52e9a4d62a8622645594f59c6 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/85e27fc52e9a4d62a8622645594f59c6
I20251025 07:52:30.803414 1170 ts_tablet_manager.cc:1950] T 85e27fc52e9a4d62a8622645594f59c6 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:30.804033 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9e8fffb8e984463b5fa2c7dd9e93227. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.804212 737 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e6544202d4f489d82f22d6073ef240d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:30.805719 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 85e27fc52e9a4d62a8622645594f59c6 (table default.Table_4bafafcc9e8641b687d438725186f639 [id=73f69c9574ee4019ada5af76bd902107]) successfully deleted
W20251025 07:52:30.806006 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 7e6544202d4f489d82f22d6073ef240d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e6544202d4f489d82f22d6073ef240d already in progress: creating tablet
W20251025 07:52:30.806228 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 5b6383c99e9e4c90a374da9ccd830a6f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b6383c99e9e4c90a374da9ccd830a6f already in progress: creating tablet
W20251025 07:52:30.806419 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 410ef150a7014c6c867a021b544a2f4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 410ef150a7014c6c867a021b544a2f4e already in progress: creating tablet
W20251025 07:52:30.806676 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet b94634ac5077439fadfbbce569f60e8b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b94634ac5077439fadfbbce569f60e8b already in progress: creating tablet
I20251025 07:52:30.806749 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b594d4718344aafbdaec34040695589. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.767608 1247 raft_consensus.cc:493] T 47284f96928749e6a5201103a8ee5f9f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:30.781646 1029 tablet_service.cc:1505] Processing CreateTablet for tablet c11f674385184ae6b235f1db8bf1fb91 (DEFAULT_TABLE table=default.Table_ce27d8ed3ede46579f2ed3ae81536572 [id=c4c41c3b236141eb8e03dbb171f6b56d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.807149 1247 raft_consensus.cc:515] T 47284f96928749e6a5201103a8ee5f9f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.807790 1247 leader_election.cc:290] T 47284f96928749e6a5201103a8ee5f9f P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:30.808703 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c11f674385184ae6b235f1db8bf1fb91. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.809266 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1068f566477e4c6db7dde566fff4c2fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.809785 1500 raft_consensus.cc:3060] T 8c6ff4c73e624af3bf2de36c4493b462 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:30.812222 752 tablet_service.cc:1505] Processing CreateTablet for tablet 1aeba18be0cb4eb9949aeaf6ad73a812 (DEFAULT_TABLE table=default.Table_bd168da90c704b7987fc731f5ab28045 [id=e7683b98787e4d55b81303f4b58e4a6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.814029 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 141e413d2faa4c67b4c822192e631417. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.814512 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa826b85f0c94150a17d4e417dd3d62f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.818117 1166 ts_tablet_manager.cc:1403] T 85e27fc52e9a4d62a8622645594f59c6 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.326s user 0.011s sys 0.018s
I20251025 07:52:30.819311 750 tablet_service.cc:1505] Processing CreateTablet for tablet 1068f566477e4c6db7dde566fff4c2fb (DEFAULT_TABLE table=default.Table_7ce70fea0fe04c52a62033afb8eef977 [id=a8b2a16fd037435893c8f60e92fd1627]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.819489 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c187a40f8ff14c6a8c184fbeb030ad13. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.819911 743 tablet_service.cc:1505] Processing CreateTablet for tablet c501ef5d70e742598c1a902efe47feb1 (DEFAULT_TABLE table=default.Table_9c39679c7a244c7f9dd7d0f97f16ccb3 [id=ad812862698d421184ebf501ce79b7be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.820523 751 tablet_service.cc:1505] Processing CreateTablet for tablet fa826b85f0c94150a17d4e417dd3d62f (DEFAULT_TABLE table=default.Table_0b63d3c003db48709227445b0a0b84ca [id=b2486c1de45d47eeb272a605ce7fdf84]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.821015 746 tablet_service.cc:1505] Processing CreateTablet for tablet c187a40f8ff14c6a8c184fbeb030ad13 (DEFAULT_TABLE table=default.Table_34b975902f884c658093a51baa54b095 [id=ec0cc315ecba4988a97d68f00ff22697]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.822383 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 9bf486d82b984d2c89ee32e84a85894f (table default.Table_21a2393beb5d49a8ba839143c8328569 [id=5fcd7d82bf9443edbf595c28fcab99ae]) successfully deleted
I20251025 07:52:30.823014 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aeba18be0cb4eb9949aeaf6ad73a812. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.823074 1158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8603bddd8a084ff7a73a25641768dfa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } } }
I20251025 07:52:30.823283 1158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [sys.catalog]: This master's current role is: LEADER
I20251025 07:52:30.823915 747 tablet_service.cc:1552] Processing DeleteTablet for tablet b94634ac5077439fadfbbce569f60e8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:30.824077 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1068f566477e4c6db7dde566fff4c2fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.824251 756 tablet_service.cc:1505] Processing CreateTablet for tablet dff7f86ae8924685b26011e504e2247b (DEFAULT_TABLE table=default.Table_dbface3af7f6413d9060b881447d34b3 [id=f74f5abb40284e08a990f9caaa828636]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.824651 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dff7f86ae8924685b26011e504e2247b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.824679 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 48ff4ce7846b4e56a37eee9bcb17bd5e
I20251025 07:52:30.825405 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_0b63d3c003db48709227445b0a0b84ca" } new_table_name: "default.Table_54ea01577e6f4faea38bddfc2d04aeb2" modify_external_catalogs: true
I20251025 07:52:30.825481 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47284f96928749e6a5201103a8ee5f9f" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:30.826007 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c187a40f8ff14c6a8c184fbeb030ad13. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.826826 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa826b85f0c94150a17d4e417dd3d62f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.826982 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_d5ce3dfc91174e268c17c2d5b3f93b86"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:30.827479 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5ce3dfc91174e268c17c2d5b3f93b86 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:30.828132 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c501ef5d70e742598c1a902efe47feb1. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:30.830837 982 leader_election.cc:343] T 47284f96928749e6a5201103a8ee5f9f P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:30.831776 741 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b6383c99e9e4c90a374da9ccd830a6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:30.831987 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 48ff4ce7846b4e56a37eee9bcb17bd5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:30.836761 1542 catalog_manager.cc:1269] Loaded cluster ID: aee99068aba246df9094aaec3350d6ff
I20251025 07:52:30.837126 1542 catalog_manager.cc:1562] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407: loading cluster ID for follower catalog manager: success
I20251025 07:52:30.838909 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47284f96928749e6a5201103a8ee5f9f" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:30.839232 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dff7f86ae8924685b26011e504e2247b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:30.843751 983 leader_election.cc:343] T 47284f96928749e6a5201103a8ee5f9f P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:30.843938 983 leader_election.cc:304] T 47284f96928749e6a5201103a8ee5f9f P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:30.844375 1247 raft_consensus.cc:2749] T 47284f96928749e6a5201103a8ee5f9f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:30.852478 880 tablet_service.cc:1552] Processing DeleteTablet for tablet 48ff4ce7846b4e56a37eee9bcb17bd5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:30.853114 737 tablet_service.cc:1505] Processing CreateTablet for tablet fd265cdbf6a8440393a2b5d8240d1872 (DEFAULT_TABLE table=default.Table_434ef1ade0a14af987864563993afae8 [id=3a3af35e11944a6f97fc0144cb9d08d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.853557 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd265cdbf6a8440393a2b5d8240d1872. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.854396 1166 raft_consensus.cc:359] T 85e27fc52e9a4d62a8622645594f59c6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.858265 1166 raft_consensus.cc:385] T 85e27fc52e9a4d62a8622645594f59c6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:30.861477 1166 raft_consensus.cc:740] T 85e27fc52e9a4d62a8622645594f59c6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:30.864861 1166 consensus_queue.cc:260] T 85e27fc52e9a4d62a8622645594f59c6 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.868460 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 48ff4ce7846b4e56a37eee9bcb17bd5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:47408
W20251025 07:52:30.869222 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 48ff4ce7846b4e56a37eee9bcb17bd5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48ff4ce7846b4e56a37eee9bcb17bd5e already in progress: creating tablet
I20251025 07:52:30.869817 1169 tablet_bootstrap.cc:492] T 2aa522477e934c89834026dc480fe7c0 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:30.870020 1169 ts_tablet_manager.cc:1403] T 2aa522477e934c89834026dc480fe7c0 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.123s user 0.005s sys 0.001s
I20251025 07:52:30.868382 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 410ef150a7014c6c867a021b544a2f4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:30.870867 1169 raft_consensus.cc:359] T 2aa522477e934c89834026dc480fe7c0 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.871074 1169 raft_consensus.cc:385] T 2aa522477e934c89834026dc480fe7c0 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:30.871153 1169 raft_consensus.cc:740] T 2aa522477e934c89834026dc480fe7c0 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:30.871398 1169 consensus_queue.cc:260] T 2aa522477e934c89834026dc480fe7c0 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.877703 1441 raft_consensus.cc:493] T 79bc2678df8042e3994b0471afa2c304 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:30.878329 1441 raft_consensus.cc:515] T 79bc2678df8042e3994b0471afa2c304 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.880532 1441 leader_election.cc:290] T 79bc2678df8042e3994b0471afa2c304 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:30.881358 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79bc2678df8042e3994b0471afa2c304" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:30.881991 712 leader_election.cc:343] T 79bc2678df8042e3994b0471afa2c304 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:30.883587 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79bc2678df8042e3994b0471afa2c304" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:30.871974 1169 ts_tablet_manager.cc:1434] T 2aa522477e934c89834026dc480fe7c0 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 07:52:30.884121 1542 catalog_manager.cc:1584] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407: acquiring CA information for follower catalog manager: success
I20251025 07:52:30.884158 1411 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" instance_seqno: 1761378744097060) as {username='slave'} at 127.0.104.3:37647; Asking this server to re-register.
I20251025 07:52:30.886693 1094 heartbeater.cc:461] Registering TS with master...
W20251025 07:52:30.887704 710 leader_election.cc:343] T 79bc2678df8042e3994b0471afa2c304 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:30.887881 710 leader_election.cc:304] T 79bc2678df8042e3994b0471afa2c304 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:30.890905 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 88eb51447e0f4080baa49dd902deb24f consensus state has no leader
I20251025 07:52:30.892096 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_241aaedabd0245dc85e8692b83d51965"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:30.892223 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_0ebd49dea97c4b0790902d42d65e749f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:30.892694 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_241aaedabd0245dc85e8692b83d51965 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:30.894337 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 50ffbcdd4a9247bf99b8e2361bc02ef5
W20251025 07:52:30.894672 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fa826b85f0c94150a17d4e417dd3d62f consensus state has no leader
W20251025 07:52:30.894721 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cb829649f73d4d3f8862d268c953bd86 consensus state has no leader
I20251025 07:52:30.894833 518 catalog_manager.cc:6719] ReplaceTablet: tablet 50ffbcdd4a9247bf99b8e2361bc02ef5 deleted and replaced by tablet e47c4f741f354a19b49777bd06e54764
I20251025 07:52:30.895198 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_8a4e7e8d20264b62b1db8a111b1285b8" } new_table_name: "default.Table_99fd36369f9a4e0fb14f234c98635203" modify_external_catalogs: true
W20251025 07:52:30.896404 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ebd49dea97c4b0790902d42d65e749f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:30.896644 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_8e00811c122448288ec358cc8e7867fe" } modify_external_catalogs: true
I20251025 07:52:30.896826 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_8e00811c122448288ec358cc8e7867fe" } modify_external_catalogs: true
I20251025 07:52:30.897365 1500 raft_consensus.cc:515] T 8c6ff4c73e624af3bf2de36c4493b462 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.897895 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_c902adc24467448f9b5d4f585cf614f5" } modify_external_catalogs: true
I20251025 07:52:30.898059 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_c902adc24467448f9b5d4f585cf614f5" } modify_external_catalogs: true
I20251025 07:52:30.901346 1166 ts_tablet_manager.cc:1434] T 85e27fc52e9a4d62a8622645594f59c6 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.075s user 0.003s sys 0.000s
I20251025 07:52:30.904874 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c6ff4c73e624af3bf2de36c4493b462" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:30.905706 875 tablet_service.cc:1552] Processing DeleteTablet for tablet 50ffbcdd4a9247bf99b8e2361bc02ef5 with delete_type TABLET_DATA_DELETED (replaced by tablet e47c4f741f354a19b49777bd06e54764) from {username='slave'} at 127.0.0.1:56590
W20251025 07:52:30.905879 850 leader_election.cc:343] T 8c6ff4c73e624af3bf2de36c4493b462 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:30.908810 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 47284f96928749e6a5201103a8ee5f9f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:30.909685 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 2aa522477e934c89834026dc480fe7c0 with delete_type TABLET_DATA_DELETED (replaced by tablet 1068f566477e4c6db7dde566fff4c2fb) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:30.909721 1170 tablet_replica.cc:333] T 47284f96928749e6a5201103a8ee5f9f P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:30.909948 1170 raft_consensus.cc:2243] T 47284f96928749e6a5201103a8ee5f9f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:30.910192 1170 raft_consensus.cc:2272] T 47284f96928749e6a5201103a8ee5f9f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:30.910360 1169 tablet_bootstrap.cc:492] T 742b3ec3f6604647b9e397532f1673dd P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:30.911347 1170 ts_tablet_manager.cc:1916] T 47284f96928749e6a5201103a8ee5f9f P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:30.912086 1169 tablet_bootstrap.cc:654] T 742b3ec3f6604647b9e397532f1673dd P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:30.913880 1411 ts_manager.cc:194] Registered new tserver with Master: db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:30.916522 1542 catalog_manager.cc:1612] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251025 07:52:30.918119 1166 tablet_bootstrap.cc:492] T 79bc2678df8042e3994b0471afa2c304 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:30.924242 1500 leader_election.cc:290] T 8c6ff4c73e624af3bf2de36c4493b462 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:30.928359 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_45faec266a9b4b1b9f5ef3ac915e3bbc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:30.928792 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:43563, user_credentials={real_user=slave}} blocked reactor thread for 44306us
W20251025 07:52:30.928911 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45faec266a9b4b1b9f5ef3ac915e3bbc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:30.929368 1441 raft_consensus.cc:2749] T 79bc2678df8042e3994b0471afa2c304 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:30.929678 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c6ff4c73e624af3bf2de36c4493b462" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:30.929824 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_bef94ac5c2bb4f3e8cbbc68037145d6a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:30.930032 775 raft_consensus.cc:3060] T 8c6ff4c73e624af3bf2de36c4493b462 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:30.930373 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bef94ac5c2bb4f3e8cbbc68037145d6a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:30.931481 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_500d949c884c4644b70943543bd3e0c5" } modify_external_catalogs: true
I20251025 07:52:30.931627 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_500d949c884c4644b70943543bd3e0c5" } modify_external_catalogs: true
I20251025 07:52:30.933348 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_293596c64cca4768bc5869fd9924f856" } modify_external_catalogs: true
I20251025 07:52:30.933509 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_293596c64cca4768bc5869fd9924f856" } modify_external_catalogs: true
I20251025 07:52:30.938961 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_56bc5dd6e81946b780e4a18414de4c58" } new_table_name: "default.Table_088aad2a46b041bfa29ed8e6929760b1" modify_external_catalogs: true
I20251025 07:52:30.943876 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 85e27fc52e9a4d62a8622645594f59c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:30.945778 1172 ts_tablet_manager.cc:1929] T f51ec427dd9349a48eb221db04cf242b P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:30.947021 775 raft_consensus.cc:2468] T 8c6ff4c73e624af3bf2de36c4493b462 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
W20251025 07:52:30.947482 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b97e39aaac3048ed93dbfe873b757254 consensus state has no leader
I20251025 07:52:30.948197 524 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 7a89d0a473074c55b8f8b44ad1632359
I20251025 07:52:30.953493 1170 ts_tablet_manager.cc:1929] T 47284f96928749e6a5201103a8ee5f9f P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:30.955116 847 leader_election.cc:304] T 8c6ff4c73e624af3bf2de36c4493b462 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:30.956085 1410 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3aea2869ff814cc7b142012faa274382" instance_seqno: 1761378743627123) as {username='slave'} at 127.0.104.2:51205; Asking this server to re-register.
I20251025 07:52:30.956406 1500 raft_consensus.cc:2804] T 8c6ff4c73e624af3bf2de36c4493b462 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:30.957365 1500 raft_consensus.cc:697] T 8c6ff4c73e624af3bf2de36c4493b462 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:30.957703 1500 consensus_queue.cc:237] T 8c6ff4c73e624af3bf2de36c4493b462 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.960819 881 tablet_service.cc:1505] Processing CreateTablet for tablet 6a33cfb2f2de48d4a5e9e39730d4900e (DEFAULT_TABLE table=default.Table_4d3dff3c6fda405f84600e93d3818380 [id=86fcd91f390d4661bcbb6d43190e6f2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.961282 1172 log.cc:1199] T f51ec427dd9349a48eb221db04cf242b P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/f51ec427dd9349a48eb221db04cf242b
I20251025 07:52:30.962998 1167 tablet_bootstrap.cc:492] T f51ec427dd9349a48eb221db04cf242b P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:30.963253 1167 ts_tablet_manager.cc:1403] T f51ec427dd9349a48eb221db04cf242b P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.481s user 0.014s sys 0.000s
I20251025 07:52:30.963359 1218 raft_consensus.cc:493] T 85e27fc52e9a4d62a8622645594f59c6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:30.963508 1218 raft_consensus.cc:515] T 85e27fc52e9a4d62a8622645594f59c6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.964452 1167 raft_consensus.cc:359] T f51ec427dd9349a48eb221db04cf242b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.964658 1167 raft_consensus.cc:385] T f51ec427dd9349a48eb221db04cf242b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:30.964761 1167 raft_consensus.cc:740] T f51ec427dd9349a48eb221db04cf242b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:30.965050 1167 consensus_queue.cc:260] T f51ec427dd9349a48eb221db04cf242b P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:30.965615 1167 ts_tablet_manager.cc:1434] T f51ec427dd9349a48eb221db04cf242b P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 07:52:30.965785 959 heartbeater.cc:461] Registering TS with master...
I20251025 07:52:30.966363 1218 leader_election.cc:290] T 85e27fc52e9a4d62a8622645594f59c6 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
W20251025 07:52:30.966846 850 leader_election.cc:343] T 85e27fc52e9a4d62a8622645594f59c6 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 85e27fc52e9a4d62a8622645594f59c6
I20251025 07:52:30.967263 1172 ts_tablet_manager.cc:1950] T f51ec427dd9349a48eb221db04cf242b P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:30.967948 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a33cfb2f2de48d4a5e9e39730d4900e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.971328 746 tablet_service.cc:1505] Processing CreateTablet for tablet 29dbaf9149134f548a974a5779f7da67 (DEFAULT_TABLE table=default.Table_c43ac1adc76d4ccf819411ec76775932 [id=106e3728b0b846178eb48ccbf076e24e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.972043 894 tablet_service.cc:1505] Processing CreateTablet for tablet 29dbaf9149134f548a974a5779f7da67 (DEFAULT_TABLE table=default.Table_c43ac1adc76d4ccf819411ec76775932 [id=106e3728b0b846178eb48ccbf076e24e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.972796 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29dbaf9149134f548a974a5779f7da67. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.973294 753 tablet_service.cc:1505] Processing CreateTablet for tablet e8402fd33ba842e981c6c68787c7c548 (DEFAULT_TABLE table=default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.973897 756 tablet_service.cc:1505] Processing CreateTablet for tablet 6a33cfb2f2de48d4a5e9e39730d4900e (DEFAULT_TABLE table=default.Table_4d3dff3c6fda405f84600e93d3818380 [id=86fcd91f390d4661bcbb6d43190e6f2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.976183 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 287) took 1647 ms. Trace:
I20251025 07:52:30.977648 1410 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" instance_seqno: 1761378743122862) as {username='slave'} at 127.0.104.1:58535; Asking this server to re-register.
I20251025 07:52:30.974246 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85e27fc52e9a4d62a8622645594f59c6" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:30.980837 745 tablet_service.cc:1505] Processing CreateTablet for tablet fce535a0c0bd45c6bf17f81add297f40 (DEFAULT_TABLE table=default.Table_1de0b151cf4c4c10aca35e45623caf2c [id=a947f21398254a53bc6668c4f8d39e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.981369 744 tablet_service.cc:1552] Processing DeleteTablet for tablet f51ec427dd9349a48eb221db04cf242b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:30.981588 890 tablet_service.cc:1505] Processing CreateTablet for tablet d77704b4fe6c4456a2ba40ffec0c60fc (DEFAULT_TABLE table=default.Table_e42a1f7287a34753b7e4daf36893d9c2 [id=ff115adf26ec4e499d44caa9b0366a49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.981869 750 tablet_service.cc:1505] Processing CreateTablet for tablet a7ab3150a35b4caca9119a1374d3f2f7 (DEFAULT_TABLE table=default.Table_8129e15bf5c549bd853c04b227a7ef26 [id=e9e8547061ba4ac987d0aa7616c7f3ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.982035 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d77704b4fe6c4456a2ba40ffec0c60fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.984553 744 tablet_service.cc:1505] Processing CreateTablet for tablet 645263a61b654f04985c40d04506c087 (DEFAULT_TABLE table=default.Table_3b25f9f08fa04ed5a0d2a758b95e8bbc [id=ff4ac044250d489d9826b92531214409]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:30.985669 847 leader_election.cc:343] T 85e27fc52e9a4d62a8622645594f59c6 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:30.985879 847 leader_election.cc:304] T 85e27fc52e9a4d62a8622645594f59c6 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:30.986187 1218 raft_consensus.cc:2749] T 85e27fc52e9a4d62a8622645594f59c6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:30.987602 891 tablet_service.cc:1505] Processing CreateTablet for tablet 645263a61b654f04985c40d04506c087 (DEFAULT_TABLE table=default.Table_3b25f9f08fa04ed5a0d2a758b95e8bbc [id=ff4ac044250d489d9826b92531214409]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.987946 882 tablet_service.cc:1505] Processing CreateTablet for tablet e8402fd33ba842e981c6c68787c7c548 (DEFAULT_TABLE table=default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.988616 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8402fd33ba842e981c6c68787c7c548. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.987315 1172 rpcz_store.cc:276] 1025 07:52:29.320405 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:29.320720 (+ 315us) service_pool.cc:225] Handling call
1025 07:52:30.618558 (+1297838us) ts_tablet_manager.cc:1151] Deleting tablet f51ec427dd9349a48eb221db04cf242b
1025 07:52:30.618566 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:30.849082 (+230516us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:30.943685 (+ 94603us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:30.967575 (+ 23890us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1283492}
I20251025 07:52:30.991829 889 tablet_service.cc:1505] Processing CreateTablet for tablet fce535a0c0bd45c6bf17f81add297f40 (DEFAULT_TABLE table=default.Table_1de0b151cf4c4c10aca35e45623caf2c [id=a947f21398254a53bc6668c4f8d39e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.993044 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 645263a61b654f04985c40d04506c087. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.994585 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fce535a0c0bd45c6bf17f81add297f40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.998335 878 tablet_service.cc:1505] Processing CreateTablet for tablet a7ab3150a35b4caca9119a1374d3f2f7 (DEFAULT_TABLE table=default.Table_8129e15bf5c549bd853c04b227a7ef26 [id=e9e8547061ba4ac987d0aa7616c7f3ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.999737 1441 raft_consensus.cc:493] T 85fba8b37f2c4540aa81701052bf222a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:30.999924 1441 raft_consensus.cc:515] T 85fba8b37f2c4540aa81701052bf222a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.000533 1441 leader_election.cc:290] T 85fba8b37f2c4540aa81701052bf222a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:31.002192 742 tablet_service.cc:1505] Processing CreateTablet for tablet d77704b4fe6c4456a2ba40ffec0c60fc (DEFAULT_TABLE table=default.Table_e42a1f7287a34753b7e4daf36893d9c2 [id=ff115adf26ec4e499d44caa9b0366a49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.008764 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 796413da731245459cc5dffacc3deb2f
I20251025 07:52:31.009496 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 50ffbcdd4a9247bf99b8e2361bc02ef5 with delete_type TABLET_DATA_DELETED (replaced by tablet e47c4f741f354a19b49777bd06e54764) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:31.011004 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 288) took 1688 ms. Trace:
I20251025 07:52:31.011188 1172 rpcz_store.cc:276] 1025 07:52:29.322092 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:29.322213 (+ 121us) service_pool.cc:225] Handling call
1025 07:52:30.997367 (+1675154us) ts_tablet_manager.cc:1151] Deleting tablet 80fe3255ed614270bd8352a2853fbc43
1025 07:52:30.997376 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:31.010598 (+ 13222us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1659427}
I20251025 07:52:31.012385 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7ab3150a35b4caca9119a1374d3f2f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.012497 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet f51ec427dd9349a48eb221db04cf242b (table default.Table_cc49d933a4b343aeb2220c3939495834 [id=e5e538e59868418e9805492aa80f3240]) successfully deleted
W20251025 07:52:31.012785 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 80fe3255ed614270bd8352a2853fbc43 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80fe3255ed614270bd8352a2853fbc43 already in progress: creating tablet
I20251025 07:52:30.974581 1167 tablet_bootstrap.cc:492] T 1483c40eee834f20a92e6b4cd89775b6 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:30.975790 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8402fd33ba842e981c6c68787c7c548. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.013237 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 645263a61b654f04985c40d04506c087. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.978747 821 heartbeater.cc:461] Registering TS with master...
I20251025 07:52:31.013748 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 301) took 1212 ms. Trace:
I20251025 07:52:31.013865 1172 rpcz_store.cc:276] 1025 07:52:29.799004 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:29.799293 (+ 289us) service_pool.cc:225] Handling call
1025 07:52:31.011579 (+1212286us) ts_tablet_manager.cc:1151] Deleting tablet 8956ad28b35d45f28c683ac055bba441
1025 07:52:31.011590 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:31.011694 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1211635}
I20251025 07:52:31.014525 519 master_service.cc:954] ReplaceTablet: received request to replace tablet d66ad049d5484228a268b0facb00ecf1 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:31.018846 1411 ts_manager.cc:194] Registered new tserver with Master: 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:31.018841 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 302) took 1215 ms. Trace:
I20251025 07:52:31.018944 1172 rpcz_store.cc:276] 1025 07:52:29.799151 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:29.799391 (+ 240us) service_pool.cc:225] Handling call
1025 07:52:31.014091 (+1214700us) ts_tablet_manager.cc:1151] Deleting tablet bef0ef23d90d480d9c99aef21d7271fd
1025 07:52:31.014103 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:31.014214 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1213027}
I20251025 07:52:31.024098 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fce535a0c0bd45c6bf17f81add297f40. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:31.028877 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 8956ad28b35d45f28c683ac055bba441 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8956ad28b35d45f28c683ac055bba441 already in progress: creating tablet
I20251025 07:52:31.029203 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29dbaf9149134f548a974a5779f7da67. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:31.029372 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet bef0ef23d90d480d9c99aef21d7271fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bef0ef23d90d480d9c99aef21d7271fd already in progress: creating tablet
I20251025 07:52:31.029490 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85fba8b37f2c4540aa81701052bf222a" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:31.030495 876 tablet_service.cc:1552] Processing DeleteTablet for tablet 796413da731245459cc5dffacc3deb2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:31.031754 1441 raft_consensus.cc:493] T 1b31177a062b467187d68ed2034f5690 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:31.031899 1441 raft_consensus.cc:515] T 1b31177a062b467187d68ed2034f5690 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.032552 1441 leader_election.cc:290] T 1b31177a062b467187d68ed2034f5690 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:31.033905 712 leader_election.cc:343] T 1b31177a062b467187d68ed2034f5690 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 1b31177a062b467187d68ed2034f5690
W20251025 07:52:31.034276 710 leader_election.cc:343] T 85fba8b37f2c4540aa81701052bf222a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:30.953797 1021 tablet_service.cc:1505] Processing CreateTablet for tablet a7ab3150a35b4caca9119a1374d3f2f7 (DEFAULT_TABLE table=default.Table_8129e15bf5c549bd853c04b227a7ef26 [id=e9e8547061ba4ac987d0aa7616c7f3ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.958333 1169 tablet_bootstrap.cc:492] T 742b3ec3f6604647b9e397532f1673dd P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:31.034672 1170 log.cc:1199] T 47284f96928749e6a5201103a8ee5f9f P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/47284f96928749e6a5201103a8ee5f9f
I20251025 07:52:31.034775 1169 ts_tablet_manager.cc:1403] T 742b3ec3f6604647b9e397532f1673dd P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.151s user 0.004s sys 0.000s
I20251025 07:52:31.034957 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7ab3150a35b4caca9119a1374d3f2f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.035163 1170 ts_tablet_manager.cc:1950] T 47284f96928749e6a5201103a8ee5f9f P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:31.035765 1169 raft_consensus.cc:359] T 742b3ec3f6604647b9e397532f1673dd P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.035956 1169 raft_consensus.cc:385] T 742b3ec3f6604647b9e397532f1673dd P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:31.036106 1169 raft_consensus.cc:740] T 742b3ec3f6604647b9e397532f1673dd P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:31.036228 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 47284f96928749e6a5201103a8ee5f9f (table default.Table_373f40b4ac044088a071a6d26d7a6da0 [id=05672e3da0704022b93cc96d61278433]) successfully deleted
I20251025 07:52:31.036335 1169 consensus_queue.cc:260] T 742b3ec3f6604647b9e397532f1673dd P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.036831 1169 ts_tablet_manager.cc:1434] T 742b3ec3f6604647b9e397532f1673dd P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:31.037138 1169 tablet_bootstrap.cc:492] T 85fba8b37f2c4540aa81701052bf222a P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:31.037525 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 300) took 1221 ms. Trace:
I20251025 07:52:30.959472 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 6a33cfb2f2de48d4a5e9e39730d4900e (DEFAULT_TABLE table=default.Table_4d3dff3c6fda405f84600e93d3818380 [id=86fcd91f390d4661bcbb6d43190e6f2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.960755 1023 tablet_service.cc:1505] Processing CreateTablet for tablet d77704b4fe6c4456a2ba40ffec0c60fc (DEFAULT_TABLE table=default.Table_e42a1f7287a34753b7e4daf36893d9c2 [id=ff115adf26ec4e499d44caa9b0366a49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.967808 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 29dbaf9149134f548a974a5779f7da67 (DEFAULT_TABLE table=default.Table_c43ac1adc76d4ccf819411ec76775932 [id=106e3728b0b846178eb48ccbf076e24e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.038897 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a33cfb2f2de48d4a5e9e39730d4900e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:30.973919 1018 tablet_service.cc:1505] Processing CreateTablet for tablet e8402fd33ba842e981c6c68787c7c548 (DEFAULT_TABLE table=default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.976778 1026 tablet_service.cc:1505] Processing CreateTablet for tablet fce535a0c0bd45c6bf17f81add297f40 (DEFAULT_TABLE table=default.Table_1de0b151cf4c4c10aca35e45623caf2c [id=a947f21398254a53bc6668c4f8d39e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:30.979800 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 645263a61b654f04985c40d04506c087 (DEFAULT_TABLE table=default.Table_3b25f9f08fa04ed5a0d2a758b95e8bbc [id=ff4ac044250d489d9826b92531214409]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.039563 1441 raft_consensus.cc:493] T 5caac9c943bf42a5b2ae522d87fdfe8e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:31.039724 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29dbaf9149134f548a974a5779f7da67. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.039746 1441 raft_consensus.cc:515] T 5caac9c943bf42a5b2ae522d87fdfe8e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.040357 1441 leader_election.cc:290] T 5caac9c943bf42a5b2ae522d87fdfe8e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:31.043443 710 leader_election.cc:343] T 1b31177a062b467187d68ed2034f5690 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 1b31177a062b467187d68ed2034f5690
I20251025 07:52:31.043568 710 leader_election.cc:304] T 1b31177a062b467187d68ed2034f5690 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:31.045452 1441 raft_consensus.cc:2749] T 1b31177a062b467187d68ed2034f5690 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:30.982460 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 50ffbcdd4a9247bf99b8e2361bc02ef5 with delete_type TABLET_DATA_DELETED (replaced by tablet e47c4f741f354a19b49777bd06e54764) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:31.001217 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85fba8b37f2c4540aa81701052bf222a" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:31.011247 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 796413da731245459cc5dffacc3deb2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:31.046391 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a33cfb2f2de48d4a5e9e39730d4900e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.046442 1166 tablet_bootstrap.cc:654] T 79bc2678df8042e3994b0471afa2c304 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:31.047171 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_6a95b96e977244b6b28688afee389b84"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:31.047354 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d77704b4fe6c4456a2ba40ffec0c60fc. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:31.047760 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a95b96e977244b6b28688afee389b84 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:31.048646 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8402fd33ba842e981c6c68787c7c548. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.049007 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_4b3d7b3a731942189cdc214938a45820" } modify_external_catalogs: true
I20251025 07:52:31.049160 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_4b3d7b3a731942189cdc214938a45820" } modify_external_catalogs: true
W20251025 07:52:31.051987 712 leader_election.cc:343] T 85fba8b37f2c4540aa81701052bf222a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:31.052117 712 leader_election.cc:304] T 85fba8b37f2c4540aa81701052bf222a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:31.052568 1441 raft_consensus.cc:2749] T 85fba8b37f2c4540aa81701052bf222a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:31.054328 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7ab3150a35b4caca9119a1374d3f2f7. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:31.057917 710 leader_election.cc:343] T 5caac9c943bf42a5b2ae522d87fdfe8e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 5caac9c943bf42a5b2ae522d87fdfe8e
I20251025 07:52:31.059002 1169 tablet_bootstrap.cc:654] T 85fba8b37f2c4540aa81701052bf222a P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:31.059334 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 645263a61b654f04985c40d04506c087. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.060902 1247 raft_consensus.cc:493] T 627a10a0714f4be8bdbced503fd6388d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 07:52:31.061955 712 leader_election.cc:343] T 5caac9c943bf42a5b2ae522d87fdfe8e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 5caac9c943bf42a5b2ae522d87fdfe8e
I20251025 07:52:31.062078 712 leader_election.cc:304] T 5caac9c943bf42a5b2ae522d87fdfe8e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:31.062381 1247 raft_consensus.cc:515] T 627a10a0714f4be8bdbced503fd6388d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.062729 1441 raft_consensus.cc:2749] T 5caac9c943bf42a5b2ae522d87fdfe8e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:31.063333 1247 leader_election.cc:290] T 627a10a0714f4be8bdbced503fd6388d P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:31.063441 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fce535a0c0bd45c6bf17f81add297f40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.063699 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627a10a0714f4be8bdbced503fd6388d" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:31.064316 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d77704b4fe6c4456a2ba40ffec0c60fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.065511 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627a10a0714f4be8bdbced503fd6388d" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:31.067742 1441 raft_consensus.cc:493] T f51ec427dd9349a48eb221db04cf242b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:31.067939 1441 raft_consensus.cc:515] T f51ec427dd9349a48eb221db04cf242b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:31.068207 982 leader_election.cc:343] T 627a10a0714f4be8bdbced503fd6388d P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:31.068370 983 leader_election.cc:343] T 627a10a0714f4be8bdbced503fd6388d P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:31.068861 1441 leader_election.cc:290] T f51ec427dd9349a48eb221db04cf242b P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:31.069432 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 796413da731245459cc5dffacc3deb2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:31.072656 1171 ts_tablet_manager.cc:1929] T d0fd00baa02d48278afaa5b8c19182f0 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:31.072829 1171 log.cc:1199] T d0fd00baa02d48278afaa5b8c19182f0 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/d0fd00baa02d48278afaa5b8c19182f0
I20251025 07:52:31.073263 1171 ts_tablet_manager.cc:1950] T d0fd00baa02d48278afaa5b8c19182f0 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:31.074011 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 271) took 2334 ms. Trace:
I20251025 07:52:31.075013 1172 rpcz_store.cc:276] 1025 07:52:29.797866 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:29.798000 (+ 134us) service_pool.cc:225] Handling call
1025 07:52:31.019174 (+1221174us) ts_tablet_manager.cc:1151] Deleting tablet 5fca357d7e24494fb6af4bd73d98833c
1025 07:52:31.019194 (+ 20us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:31.019284 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1218012}
I20251025 07:52:31.083199 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 303) took 1283 ms. Trace:
I20251025 07:52:31.083374 1172 rpcz_store.cc:276] 1025 07:52:29.799252 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:29.799425 (+ 173us) service_pool.cc:225] Handling call
1025 07:52:31.075382 (+1275957us) ts_tablet_manager.cc:1151] Deleting tablet 4037dee385d04e85bb78c8e77cb00379
1025 07:52:31.075392 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:31.082854 (+ 7462us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1261249}
I20251025 07:52:31.085093 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 316) took 1160 ms. Trace:
I20251025 07:52:31.085263 1172 rpcz_store.cc:276] 1025 07:52:29.923246 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:29.924849 (+ 1603us) service_pool.cc:225] Handling call
1025 07:52:31.083918 (+1159069us) ts_tablet_manager.cc:1151] Deleting tablet f97acc9a638b4d5a968dcc3412a384bc
1025 07:52:31.083929 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:31.084019 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1158276}
I20251025 07:52:31.086588 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 313) took 1163 ms. Trace:
I20251025 07:52:31.086731 1172 rpcz_store.cc:276] 1025 07:52:29.922910 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:29.924914 (+ 2004us) service_pool.cc:225] Handling call
1025 07:52:31.085566 (+1160652us) ts_tablet_manager.cc:1151] Deleting tablet 53198283718d4b45b30df65b5bde96a6
1025 07:52:31.085574 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:31.086428 (+ 854us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1154322}
I20251025 07:52:31.087386 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 314) took 1164 ms. Trace:
I20251025 07:52:31.087536 1172 rpcz_store.cc:276] 1025 07:52:29.923037 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:29.925357 (+ 2320us) service_pool.cc:225] Handling call
1025 07:52:31.087029 (+1161672us) ts_tablet_manager.cc:1151] Deleting tablet be5ab0db6ac34c2c9a370445e251eb46
1025 07:52:31.087040 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:31.087120 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1154601}
I20251025 07:52:31.088157 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 315) took 1164 ms. Trace:
I20251025 07:52:31.088303 1172 rpcz_store.cc:276] 1025 07:52:29.923139 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:29.925027 (+ 1888us) service_pool.cc:225] Handling call
1025 07:52:31.087866 (+1162839us) ts_tablet_manager.cc:1151] Deleting tablet db99a1e68daf4e948a3d66ad276cc5f9
1025 07:52:31.087874 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:31.087957 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1154513}
I20251025 07:52:31.088869 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 317) took 1134 ms. Trace:
I20251025 07:52:31.089016 1172 rpcz_store.cc:276] 1025 07:52:29.954516 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:29.975924 (+ 21408us) service_pool.cc:225] Handling call
1025 07:52:31.088572 (+1112648us) ts_tablet_manager.cc:1151] Deleting tablet 91728987303b41fd840eb56f3a311c5f
1025 07:52:31.088580 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:31.088649 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1112320}
W20251025 07:52:31.089296 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 5fca357d7e24494fb6af4bd73d98833c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5fca357d7e24494fb6af4bd73d98833c already in progress: creating tablet
I20251025 07:52:31.089478 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 319) took 1045 ms. Trace:
I20251025 07:52:31.089612 1172 rpcz_store.cc:276] 1025 07:52:30.043850 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:30.043913 (+ 63us) service_pool.cc:225] Handling call
1025 07:52:31.089315 (+1045402us) ts_tablet_manager.cc:1151] Deleting tablet a6ddefa1738f4ff98586900c344359d2
1025 07:52:31.089322 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:31.089396 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1045163}
W20251025 07:52:31.089741 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 4037dee385d04e85bb78c8e77cb00379 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4037dee385d04e85bb78c8e77cb00379 already in progress: creating tablet
W20251025 07:52:31.089938 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet f97acc9a638b4d5a968dcc3412a384bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f97acc9a638b4d5a968dcc3412a384bc already in progress: creating tablet
I20251025 07:52:31.090058 1172 tablet_replica.cc:333] T 2aa522477e934c89834026dc480fe7c0 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
W20251025 07:52:31.090101 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 53198283718d4b45b30df65b5bde96a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53198283718d4b45b30df65b5bde96a6 already in progress: creating tablet
I20251025 07:52:31.090337 1172 raft_consensus.cc:2243] T 2aa522477e934c89834026dc480fe7c0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251025 07:52:31.090883 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet be5ab0db6ac34c2c9a370445e251eb46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be5ab0db6ac34c2c9a370445e251eb46 already in progress: creating tablet
W20251025 07:52:31.091107 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet db99a1e68daf4e948a3d66ad276cc5f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db99a1e68daf4e948a3d66ad276cc5f9 already in progress: creating tablet
W20251025 07:52:31.092559 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ef1854d35a3047e3a9fbe0166ab1295a consensus state has no leader
W20251025 07:52:31.093263 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 91728987303b41fd840eb56f3a311c5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91728987303b41fd840eb56f3a311c5f already in progress: creating tablet
W20251025 07:52:31.093487 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet a6ddefa1738f4ff98586900c344359d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6ddefa1738f4ff98586900c344359d2 already in progress: creating tablet
I20251025 07:52:31.095588 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_6588890a31184cc2ab6ccb9b72e60429" } new_table_name: "default.Table_9fa69b10b5e842639fc0b0a58cd9b6a5" modify_external_catalogs: true
I20251025 07:52:31.098731 1170 tablet_replica.cc:333] T 2aa522477e934c89834026dc480fe7c0 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:31.099052 1170 raft_consensus.cc:2243] T 2aa522477e934c89834026dc480fe7c0 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:31.099313 1170 raft_consensus.cc:2272] T 2aa522477e934c89834026dc480fe7c0 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:31.101085 1171 rpcz_store.cc:276] 1025 07:52:28.739547 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:28.739624 (+ 77us) service_pool.cc:225] Handling call
1025 07:52:30.791892 (+2052268us) ts_tablet_manager.cc:1151] Deleting tablet d0fd00baa02d48278afaa5b8c19182f0
1025 07:52:30.791902 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:30.984505 (+192603us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:31.072530 (+ 88025us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:31.073979 (+ 1449us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2051967}
W20251025 07:52:31.085877 710 leader_election.cc:343] T f51ec427dd9349a48eb221db04cf242b P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: f51ec427dd9349a48eb221db04cf242b
I20251025 07:52:31.082805 1167 tablet_bootstrap.cc:654] T 1483c40eee834f20a92e6b4cd89775b6 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:31.104797 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 279) took 2220 ms. Trace:
I20251025 07:52:31.104982 1171 rpcz_store.cc:276] 1025 07:52:28.884251 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:28.890190 (+ 5939us) service_pool.cc:225] Handling call
1025 07:52:31.104626 (+2214436us) ts_tablet_manager.cc:1151] Deleting tablet 943ec703e65f4fd3b0c690e08c0bd226
1025 07:52:31.104635 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:31.104763 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2214206}
I20251025 07:52:31.104470 983 leader_election.cc:304] T 627a10a0714f4be8bdbced503fd6388d P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:31.106344 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_51f2b43545db41029568a0321a37e139"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:31.106940 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_51f2b43545db41029568a0321a37e139 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:31.107504 1172 raft_consensus.cc:2272] T 2aa522477e934c89834026dc480fe7c0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:31.108124 1166 tablet_bootstrap.cc:492] T 79bc2678df8042e3994b0471afa2c304 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:31.108311 1166 ts_tablet_manager.cc:1403] T 79bc2678df8042e3994b0471afa2c304 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.205s user 0.008s sys 0.001s
I20251025 07:52:31.108386 1247 raft_consensus.cc:2749] T 627a10a0714f4be8bdbced503fd6388d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:31.108472 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_ac3211ef7b0941d2ad4db2fab62231b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:31.108913 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac3211ef7b0941d2ad4db2fab62231b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:31.109390 1166 raft_consensus.cc:359] T 79bc2678df8042e3994b0471afa2c304 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.109786 1166 raft_consensus.cc:385] T 79bc2678df8042e3994b0471afa2c304 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:31.109964 1166 raft_consensus.cc:740] T 79bc2678df8042e3994b0471afa2c304 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:31.109675 1172 ts_tablet_manager.cc:1916] T 2aa522477e934c89834026dc480fe7c0 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:31.111907 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 292) took 2115 ms. Trace:
I20251025 07:52:31.112037 1171 rpcz_store.cc:276] 1025 07:52:28.996921 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:29.229267 (+232346us) service_pool.cc:225] Handling call
1025 07:52:31.111738 (+1882471us) ts_tablet_manager.cc:1151] Deleting tablet 80ee75935d7e461d9d4066fac9819bef
1025 07:52:31.111750 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:31.111866 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1875579}
I20251025 07:52:31.113173 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 294) took 2083 ms. Trace:
I20251025 07:52:31.113298 1171 rpcz_store.cc:276] 1025 07:52:29.029438 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:29.247276 (+217838us) service_pool.cc:225] Handling call
1025 07:52:31.113017 (+1865741us) ts_tablet_manager.cc:1151] Deleting tablet b3a0af893a8947848e12093d0a7f0229
1025 07:52:31.113027 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:31.113135 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1862791}
I20251025 07:52:31.113790 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 295) took 2084 ms. Trace:
I20251025 07:52:31.113675 1166 consensus_queue.cc:260] T 79bc2678df8042e3994b0471afa2c304 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.113905 1171 rpcz_store.cc:276] 1025 07:52:29.029550 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:29.250249 (+220699us) service_pool.cc:225] Handling call
1025 07:52:31.113669 (+1863420us) ts_tablet_manager.cc:1151] Deleting tablet aaef24ccdfba45fe8204327723df834e
1025 07:52:31.113678 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:31.113759 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1863251}
I20251025 07:52:31.114408 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 300) took 1882 ms. Trace:
I20251025 07:52:31.114524 1171 rpcz_store.cc:276] 1025 07:52:29.232377 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:29.262526 (+ 30149us) service_pool.cc:225] Handling call
1025 07:52:31.114285 (+1851759us) ts_tablet_manager.cc:1151] Deleting tablet 80fe3255ed614270bd8352a2853fbc43
1025 07:52:31.114294 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:31.114374 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1845440}
I20251025 07:52:31.114910 1171 tablet_replica.cc:333] T b00153c97104470883f6089d71d23708 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:31.115111 1171 raft_consensus.cc:2243] T b00153c97104470883f6089d71d23708 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:31.115265 1171 raft_consensus.cc:2272] T b00153c97104470883f6089d71d23708 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:31.115844 1171 ts_tablet_manager.cc:1916] T b00153c97104470883f6089d71d23708 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:31.116724 712 leader_election.cc:343] T f51ec427dd9349a48eb221db04cf242b P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: f51ec427dd9349a48eb221db04cf242b
I20251025 07:52:31.116842 712 leader_election.cc:304] T f51ec427dd9349a48eb221db04cf242b P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:31.116904 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_56d474131b4047f49432e55467b20cdd" } new_table_name: "default.Table_7b8e9726fe804f38a47ee1f428f4b666" modify_external_catalogs: true
I20251025 07:52:31.117272 1441 raft_consensus.cc:2749] T f51ec427dd9349a48eb221db04cf242b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:31.118625 1411 ts_manager.cc:194] Registered new tserver with Master: 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:31.118631 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_23cacc46bec647adbc62b457f73487cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:31.119167 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23cacc46bec647adbc62b457f73487cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:31.126387 1170 ts_tablet_manager.cc:1916] T 2aa522477e934c89834026dc480fe7c0 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:31.128454 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet d0fd00baa02d48278afaa5b8c19182f0 (table default.Table_1d4029fc57dd4ecb84b8621c33d27958 [id=ff352eacb0614dbbbb6ab6369b9192e8]) successfully deleted
I20251025 07:52:31.129172 1171 ts_tablet_manager.cc:1929] T b00153c97104470883f6089d71d23708 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:31.129320 1171 log.cc:1199] T b00153c97104470883f6089d71d23708 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/b00153c97104470883f6089d71d23708
I20251025 07:52:31.129652 1171 ts_tablet_manager.cc:1950] T b00153c97104470883f6089d71d23708 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
W20251025 07:52:31.128767 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 943ec703e65f4fd3b0c690e08c0bd226 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 943ec703e65f4fd3b0c690e08c0bd226 already in progress: creating tablet
W20251025 07:52:31.130368 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 80ee75935d7e461d9d4066fac9819bef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80ee75935d7e461d9d4066fac9819bef already in progress: creating tablet
I20251025 07:52:31.130514 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 301) took 1869 ms. Trace:
W20251025 07:52:31.130733 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet b3a0af893a8947848e12093d0a7f0229 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3a0af893a8947848e12093d0a7f0229 already in progress: creating tablet
I20251025 07:52:31.130642 1171 rpcz_store.cc:276] 1025 07:52:29.261385 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:29.268872 (+ 7487us) service_pool.cc:225] Handling call
1025 07:52:31.114852 (+1845980us) ts_tablet_manager.cc:1151] Deleting tablet b00153c97104470883f6089d71d23708
1025 07:52:31.114860 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:31.125899 (+ 11039us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:31.129061 (+ 3162us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:31.130460 (+ 1399us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1845817}
W20251025 07:52:31.131068 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet aaef24ccdfba45fe8204327723df834e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aaef24ccdfba45fe8204327723df834e already in progress: creating tablet
W20251025 07:52:31.131301 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 80fe3255ed614270bd8352a2853fbc43 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80fe3255ed614270bd8352a2853fbc43 already in progress: creating tablet
I20251025 07:52:31.131500 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet b00153c97104470883f6089d71d23708 (table default.Table_b330f1a41da2427995e699545d5fc49b [id=7ec5f704dc254c11b865c660ffabc4fe]) successfully deleted
I20251025 07:52:31.139937 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 293) took 2117 ms. Trace:
I20251025 07:52:31.140060 1171 rpcz_store.cc:276] 1025 07:52:29.022004 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:29.236173 (+214169us) service_pool.cc:225] Handling call
1025 07:52:31.139750 (+1903577us) ts_tablet_manager.cc:1151] Deleting tablet 71432a6848944a0da06ef9b81f160dc4
1025 07:52:31.139761 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:31.139879 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1870547}
I20251025 07:52:31.140677 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 299) took 1911 ms. Trace:
I20251025 07:52:31.140779 1171 rpcz_store.cc:276] 1025 07:52:29.228923 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:29.261899 (+ 32976us) service_pool.cc:225] Handling call
1025 07:52:31.140441 (+1878542us) ts_tablet_manager.cc:1151] Deleting tablet b947ecc352e54f4caa73c6ca97372256
1025 07:52:31.140452 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:31.140631 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1867872}
I20251025 07:52:31.141247 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 312) took 1706 ms. Trace:
I20251025 07:52:31.141368 1171 rpcz_store.cc:276] 1025 07:52:29.434300 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:29.575671 (+141371us) service_pool.cc:225] Handling call
1025 07:52:31.141108 (+1565437us) ts_tablet_manager.cc:1151] Deleting tablet 5fca357d7e24494fb6af4bd73d98833c
1025 07:52:31.141116 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:31.141203 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1565147}
I20251025 07:52:31.148248 1166 ts_tablet_manager.cc:1434] T 79bc2678df8042e3994b0471afa2c304 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.040s user 0.001s sys 0.001s
I20251025 07:52:31.148260 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 742b3ec3f6604647b9e397532f1673dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:31.142194 1167 tablet_bootstrap.cc:492] T 1483c40eee834f20a92e6b4cd89775b6 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:31.148604 1167 ts_tablet_manager.cc:1403] T 1483c40eee834f20a92e6b4cd89775b6 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.174s user 0.001s sys 0.004s
I20251025 07:52:31.149475 1167 raft_consensus.cc:359] T 1483c40eee834f20a92e6b4cd89775b6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.149685 1167 raft_consensus.cc:385] T 1483c40eee834f20a92e6b4cd89775b6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:31.149749 1167 raft_consensus.cc:740] T 1483c40eee834f20a92e6b4cd89775b6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:31.150013 1167 consensus_queue.cc:260] T 1483c40eee834f20a92e6b4cd89775b6 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.150535 1167 ts_tablet_manager.cc:1434] T 1483c40eee834f20a92e6b4cd89775b6 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:31.150848 1167 tablet_bootstrap.cc:492] T 47284f96928749e6a5201103a8ee5f9f P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:31.152537 1167 tablet_bootstrap.cc:654] T 47284f96928749e6a5201103a8ee5f9f P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:31.154361 1166 tablet_bootstrap.cc:492] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:31.159606 1166 tablet_bootstrap.cc:654] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:31.160938 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 313) took 1670 ms. Trace:
I20251025 07:52:31.161413 1167 tablet_bootstrap.cc:492] T 47284f96928749e6a5201103a8ee5f9f P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:31.161211 1171 rpcz_store.cc:276] 1025 07:52:29.490216 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:29.575981 (+ 85765us) service_pool.cc:225] Handling call
1025 07:52:31.160676 (+1584695us) ts_tablet_manager.cc:1151] Deleting tablet 8956ad28b35d45f28c683ac055bba441
1025 07:52:31.160686 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:31.160793 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1584460}
I20251025 07:52:31.161641 1167 ts_tablet_manager.cc:1403] T 47284f96928749e6a5201103a8ee5f9f P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20251025 07:52:31.162303 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 314) took 1669 ms. Trace:
I20251025 07:52:31.162431 1167 raft_consensus.cc:359] T 47284f96928749e6a5201103a8ee5f9f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.162621 1167 raft_consensus.cc:385] T 47284f96928749e6a5201103a8ee5f9f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:31.162719 1167 raft_consensus.cc:740] T 47284f96928749e6a5201103a8ee5f9f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:31.162947 1167 consensus_queue.cc:260] T 47284f96928749e6a5201103a8ee5f9f P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.163448 1167 ts_tablet_manager.cc:1434] T 47284f96928749e6a5201103a8ee5f9f P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:31.163724 1167 tablet_bootstrap.cc:492] T dd898450f2604987b08639cf87d0e402 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:31.165385 1167 tablet_bootstrap.cc:654] T dd898450f2604987b08639cf87d0e402 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:31.167855 1172 ts_tablet_manager.cc:1929] T 2aa522477e934c89834026dc480fe7c0 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:31.167976 1172 log.cc:1199] T 2aa522477e934c89834026dc480fe7c0 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/2aa522477e934c89834026dc480fe7c0
I20251025 07:52:31.168339 1172 ts_tablet_manager.cc:1950] T 2aa522477e934c89834026dc480fe7c0 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:31.172039 1500 raft_consensus.cc:493] T 79bc2678df8042e3994b0471afa2c304 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:31.172240 1500 raft_consensus.cc:515] T 79bc2678df8042e3994b0471afa2c304 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.172467 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet c59f6e6eb9594a8e83f109a09b034045
I20251025 07:52:31.162524 1171 rpcz_store.cc:276] 1025 07:52:29.493965 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:29.576249 (+ 82284us) service_pool.cc:225] Handling call
1025 07:52:31.161999 (+1585750us) ts_tablet_manager.cc:1151] Deleting tablet bef0ef23d90d480d9c99aef21d7271fd
1025 07:52:31.162034 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:31.162221 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1585503}
I20251025 07:52:31.173377 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 315) took 1638 ms. Trace:
I20251025 07:52:31.173549 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79bc2678df8042e3994b0471afa2c304" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:31.173758 775 raft_consensus.cc:2468] T 79bc2678df8042e3994b0471afa2c304 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:31.173561 1171 rpcz_store.cc:276] 1025 07:52:29.534418 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:29.576505 (+ 42087us) service_pool.cc:225] Handling call
1025 07:52:31.173091 (+1596586us) ts_tablet_manager.cc:1151] Deleting tablet 4037dee385d04e85bb78c8e77cb00379
1025 07:52:31.173104 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:31.173262 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1596357}
I20251025 07:52:31.174033 526 catalog_manager.cc:5649] T 8c6ff4c73e624af3bf2de36c4493b462 P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } }
I20251025 07:52:31.174307 847 leader_election.cc:304] T 79bc2678df8042e3994b0471afa2c304 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:31.174310 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 326) took 1331 ms. Trace:
I20251025 07:52:31.174557 1218 raft_consensus.cc:2804] T 79bc2678df8042e3994b0471afa2c304 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:31.174444 1171 rpcz_store.cc:276] 1025 07:52:29.842375 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:29.843558 (+ 1183us) service_pool.cc:225] Handling call
1025 07:52:31.174013 (+1330455us) ts_tablet_manager.cc:1151] Deleting tablet be5ab0db6ac34c2c9a370445e251eb46
1025 07:52:31.174049 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:31.174238 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1328900}
I20251025 07:52:31.174646 1218 raft_consensus.cc:493] T 79bc2678df8042e3994b0471afa2c304 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:31.174748 1218 raft_consensus.cc:3060] T 79bc2678df8042e3994b0471afa2c304 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:31.175233 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 4f0c4f0e8f014e59be9e46d2afe0be9d
I20251025 07:52:31.175743 1171 tablet_replica.cc:333] T cf2cab1552bd4e7394797c93b149f450 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:31.176071 1171 raft_consensus.cc:2243] T cf2cab1552bd4e7394797c93b149f450 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:31.176352 1171 raft_consensus.cc:2272] T cf2cab1552bd4e7394797c93b149f450 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:31.177249 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_9c39679c7a244c7f9dd7d0f97f16ccb3" } new_table_name: "default.Table_0ae70257430b4a1ea853e49f9720cbea" modify_external_catalogs: true
I20251025 07:52:31.177328 1171 ts_tablet_manager.cc:1916] T cf2cab1552bd4e7394797c93b149f450 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:31.177567 1218 raft_consensus.cc:515] T 79bc2678df8042e3994b0471afa2c304 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.178331 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet d66ad049d5484228a268b0facb00ecf1
I20251025 07:52:31.178460 1218 leader_election.cc:290] T 79bc2678df8042e3994b0471afa2c304 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:31.178737 519 catalog_manager.cc:6719] ReplaceTablet: tablet d66ad049d5484228a268b0facb00ecf1 deleted and replaced by tablet 2e098c9c4613459295b6043048cba587
I20251025 07:52:31.179083 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79bc2678df8042e3994b0471afa2c304" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:31.179132 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_064797535a2145bf93bd38b25ddaec83"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:31.179265 775 raft_consensus.cc:3060] T 79bc2678df8042e3994b0471afa2c304 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:31.179661 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_064797535a2145bf93bd38b25ddaec83 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:31.181097 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_ea269532aff34ca1803fc1c5d5ad3641"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:31.181527 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea269532aff34ca1803fc1c5d5ad3641 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:31.182919 775 raft_consensus.cc:2468] T 79bc2678df8042e3994b0471afa2c304 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:31.183380 847 leader_election.cc:304] T 79bc2678df8042e3994b0471afa2c304 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:31.184221 1218 raft_consensus.cc:2804] T 79bc2678df8042e3994b0471afa2c304 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:31.184635 1500 leader_election.cc:290] T 79bc2678df8042e3994b0471afa2c304 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:31.186574 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 2aa522477e934c89834026dc480fe7c0 (table default.Table_7ce70fea0fe04c52a62033afb8eef977 [id=a8b2a16fd037435893c8f60e92fd1627]) successfully deleted
I20251025 07:52:31.187260 1166 tablet_bootstrap.cc:492] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:31.187465 1166 ts_tablet_manager.cc:1403] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.033s user 0.001s sys 0.002s
I20251025 07:52:31.190230 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f0c4f0e8f014e59be9e46d2afe0be9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:31.190549 890 tablet_service.cc:1552] Processing DeleteTablet for tablet c59f6e6eb9594a8e83f109a09b034045 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:31.188275 1166 raft_consensus.cc:359] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:31.191406 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:43563, user_credentials={real_user=slave}} blocked reactor thread for 49125.9us
I20251025 07:52:31.191551 1166 raft_consensus.cc:385] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:31.191645 1166 raft_consensus.cc:740] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:31.194716 890 tablet_service.cc:1552] Processing DeleteTablet for tablet d66ad049d5484228a268b0facb00ecf1 with delete_type TABLET_DATA_DELETED (replaced by tablet 2e098c9c4613459295b6043048cba587) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:31.195071 1166 consensus_queue.cc:260] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:31.196075 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 71432a6848944a0da06ef9b81f160dc4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71432a6848944a0da06ef9b81f160dc4 already in progress: creating tablet
I20251025 07:52:31.196069 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f0c4f0e8f014e59be9e46d2afe0be9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:31.196302 742 tablet_service.cc:1552] Processing DeleteTablet for tablet c59f6e6eb9594a8e83f109a09b034045 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:31.196341 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 1483c40eee834f20a92e6b4cd89775b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:31.196566 1218 raft_consensus.cc:697] T 79bc2678df8042e3994b0471afa2c304 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:31.196597 745 tablet_service.cc:1552] Processing DeleteTablet for tablet d66ad049d5484228a268b0facb00ecf1 with delete_type TABLET_DATA_DELETED (replaced by tablet 2e098c9c4613459295b6043048cba587) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:31.196946 1218 consensus_queue.cc:237] T 79bc2678df8042e3994b0471afa2c304 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.198596 1166 ts_tablet_manager.cc:1434] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
W20251025 07:52:31.198621 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet b947ecc352e54f4caa73c6ca97372256 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b947ecc352e54f4caa73c6ca97372256 already in progress: creating tablet
I20251025 07:52:31.199054 1166 tablet_bootstrap.cc:492] T 742b3ec3f6604647b9e397532f1673dd P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:31.200140 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79bc2678df8042e3994b0471afa2c304" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:31.200780 850 leader_election.cc:343] T 79bc2678df8042e3994b0471afa2c304 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:31.201629 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79bc2678df8042e3994b0471afa2c304" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:31.202062 850 leader_election.cc:343] T 79bc2678df8042e3994b0471afa2c304 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:31.202252 1171 ts_tablet_manager.cc:1929] T cf2cab1552bd4e7394797c93b149f450 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:31.202519 1171 log.cc:1199] T cf2cab1552bd4e7394797c93b149f450 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/cf2cab1552bd4e7394797c93b149f450
I20251025 07:52:31.203009 1171 ts_tablet_manager.cc:1950] T cf2cab1552bd4e7394797c93b149f450 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:31.203868 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 324) took 1381 ms. Trace:
I20251025 07:52:31.206004 1166 tablet_bootstrap.cc:654] T 742b3ec3f6604647b9e397532f1673dd P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:31.206372 1171 rpcz_store.cc:276] 1025 07:52:29.822533 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:29.822683 (+ 150us) service_pool.cc:225] Handling call
1025 07:52:31.174913 (+1352230us) ts_tablet_manager.cc:1151] Deleting tablet cf2cab1552bd4e7394797c93b149f450
1025 07:52:31.174921 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:31.189041 (+ 14120us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:31.202077 (+ 13036us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:31.203830 (+ 1753us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1319905}
I20251025 07:52:31.207024 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 325) took 1364 ms. Trace:
I20251025 07:52:31.207214 1171 rpcz_store.cc:276] 1025 07:52:29.842247 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:29.872755 (+ 30508us) service_pool.cc:225] Handling call
1025 07:52:31.206867 (+1334112us) ts_tablet_manager.cc:1151] Deleting tablet 53198283718d4b45b30df65b5bde96a6
1025 07:52:31.206877 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:31.206977 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1322617}
I20251025 07:52:31.208072 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 328) took 1302 ms. Trace:
I20251025 07:52:31.208163 1167 tablet_bootstrap.cc:492] T dd898450f2604987b08639cf87d0e402 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:31.208478 1171 rpcz_store.cc:276] 1025 07:52:29.905377 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:29.914062 (+ 8685us) service_pool.cc:225] Handling call
1025 07:52:31.207640 (+1293578us) ts_tablet_manager.cc:1151] Deleting tablet f97acc9a638b4d5a968dcc3412a384bc
1025 07:52:31.207648 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:31.207729 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1292833}
I20251025 07:52:31.208643 1167 ts_tablet_manager.cc:1403] T dd898450f2604987b08639cf87d0e402 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.007s
I20251025 07:52:31.209638 1167 raft_consensus.cc:359] T dd898450f2604987b08639cf87d0e402 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.209892 1167 raft_consensus.cc:385] T dd898450f2604987b08639cf87d0e402 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:31.210009 1167 raft_consensus.cc:740] T dd898450f2604987b08639cf87d0e402 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:31.210420 1167 consensus_queue.cc:260] T dd898450f2604987b08639cf87d0e402 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:31.212965 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 551d07a9242740dcb9366bb2d707a4a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 551d07a9242740dcb9366bb2d707a4a8 already in progress: creating tablet
I20251025 07:52:31.213704 1172 tablet_replica.cc:333] T 567a1f5c216a4a519249da66a2a03ea4 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:31.213930 1172 raft_consensus.cc:2243] T 567a1f5c216a4a519249da66a2a03ea4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:31.214102 1172 raft_consensus.cc:2272] T 567a1f5c216a4a519249da66a2a03ea4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:31.214732 1172 ts_tablet_manager.cc:1916] T 567a1f5c216a4a519249da66a2a03ea4 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:31.215107 1167 ts_tablet_manager.cc:1434] T dd898450f2604987b08639cf87d0e402 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
W20251025 07:52:31.215229 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet f661d5909657455581dcb5e6eadd3c0b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f661d5909657455581dcb5e6eadd3c0b already in progress: creating tablet
W20251025 07:52:31.215523 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 467324e2432f4caca148951329caa292 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 467324e2432f4caca148951329caa292 already in progress: creating tablet
W20251025 07:52:31.215929 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 74e2d78c024849b0af5c0bde1f48c36f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74e2d78c024849b0af5c0bde1f48c36f already in progress: creating tablet
W20251025 07:52:31.218441 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet d9bfd7c0c3ad4e5b98b6708793d61d41 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9bfd7c0c3ad4e5b98b6708793d61d41 already in progress: creating tablet
I20251025 07:52:31.219776 1218 raft_consensus.cc:493] T 90764275825c4ffb8712cb883d37a551 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:31.219990 1218 raft_consensus.cc:515] T 90764275825c4ffb8712cb883d37a551 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.220256 1167 tablet_bootstrap.cc:492] T 85e27fc52e9a4d62a8622645594f59c6 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:31.220808 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f0c4f0e8f014e59be9e46d2afe0be9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:31.220952 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 327) took 1315 ms. Trace:
I20251025 07:52:31.221040 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90764275825c4ffb8712cb883d37a551" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:31.221078 1171 rpcz_store.cc:276] 1025 07:52:29.905269 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:29.913220 (+ 7951us) service_pool.cc:225] Handling call
1025 07:52:31.220804 (+1307584us) ts_tablet_manager.cc:1151] Deleting tablet db99a1e68daf4e948a3d66ad276cc5f9
1025 07:52:31.220812 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:31.220910 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1256077}
I20251025 07:52:31.221479 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 329) took 1272 ms. Trace:
I20251025 07:52:31.221583 1171 rpcz_store.cc:276] 1025 07:52:29.948813 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:29.951603 (+ 2790us) service_pool.cc:225] Handling call
1025 07:52:31.221331 (+1269728us) ts_tablet_manager.cc:1151] Deleting tablet 91728987303b41fd840eb56f3a311c5f
1025 07:52:31.221339 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:31.221446 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1255763}
W20251025 07:52:31.221640 847 leader_election.cc:343] T 90764275825c4ffb8712cb883d37a551 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:31.222246 1218 leader_election.cc:290] T 90764275825c4ffb8712cb883d37a551 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:31.222991 914 raft_consensus.cc:1275] T cd23b8094d7e472eba6bd170248858a9 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Refusing update from remote peer db42be7ad3c741a0aa427f27e245993b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251025 07:52:31.222988 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet c59f6e6eb9594a8e83f109a09b034045 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:31.222998 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet d66ad049d5484228a268b0facb00ecf1 with delete_type TABLET_DATA_DELETED (replaced by tablet 2e098c9c4613459295b6043048cba587) from {username='slave'} at 127.0.0.1:47408
W20251025 07:52:31.226202 982 consensus_peers.cc:597] T cd23b8094d7e472eba6bd170248858a9 P db42be7ad3c741a0aa427f27e245993b -> Peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Couldn't send request to peer 23eeffd3de254ce18db5febfde971c6b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:31.226377 1247 consensus_queue.cc:1048] T cd23b8094d7e472eba6bd170248858a9 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 07:52:31.227681 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90764275825c4ffb8712cb883d37a551" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:31.228183 850 leader_election.cc:343] T 90764275825c4ffb8712cb883d37a551 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:31.228322 850 leader_election.cc:304] T 90764275825c4ffb8712cb883d37a551 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:31.229029 1218 raft_consensus.cc:2749] T 90764275825c4ffb8712cb883d37a551 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:31.231948 1167 tablet_bootstrap.cc:654] T 85e27fc52e9a4d62a8622645594f59c6 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:31.232407 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 330) took 1191 ms. Trace:
I20251025 07:52:31.233425 1171 rpcz_store.cc:276] 1025 07:52:30.041121 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:30.043066 (+ 1945us) service_pool.cc:225] Handling call
1025 07:52:31.232264 (+1189198us) ts_tablet_manager.cc:1151] Deleting tablet a6ddefa1738f4ff98586900c344359d2
1025 07:52:31.232274 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:31.232374 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1188954}
I20251025 07:52:31.233927 1171 tablet_replica.cc:333] T 198db9a45257435f86df2605992ccb34 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:31.234299 1171 raft_consensus.cc:2243] T 198db9a45257435f86df2605992ccb34 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:31.234508 1171 raft_consensus.cc:2272] T 198db9a45257435f86df2605992ccb34 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:31.235093 1171 ts_tablet_manager.cc:1916] T 198db9a45257435f86df2605992ccb34 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:31.235320 1170 ts_tablet_manager.cc:1929] T 2aa522477e934c89834026dc480fe7c0 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:31.235463 1170 log.cc:1199] T 2aa522477e934c89834026dc480fe7c0 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/2aa522477e934c89834026dc480fe7c0
I20251025 07:52:31.235721 1166 tablet_bootstrap.cc:492] T 742b3ec3f6604647b9e397532f1673dd P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:31.235837 1170 ts_tablet_manager.cc:1950] T 2aa522477e934c89834026dc480fe7c0 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:31.235889 1166 ts_tablet_manager.cc:1403] T 742b3ec3f6604647b9e397532f1673dd P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s
I20251025 07:52:31.236673 1166 raft_consensus.cc:359] T 742b3ec3f6604647b9e397532f1673dd P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.236874 1166 raft_consensus.cc:385] T 742b3ec3f6604647b9e397532f1673dd P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:31.236852 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 2aa522477e934c89834026dc480fe7c0 (table default.Table_7ce70fea0fe04c52a62033afb8eef977 [id=a8b2a16fd037435893c8f60e92fd1627]) successfully deleted
I20251025 07:52:31.236977 1166 raft_consensus.cc:740] T 742b3ec3f6604647b9e397532f1673dd P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:31.237282 1170 tablet_replica.cc:333] T 796413da731245459cc5dffacc3deb2f P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:31.237497 1170 raft_consensus.cc:2243] T 796413da731245459cc5dffacc3deb2f P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:31.237835 1170 raft_consensus.cc:2272] T 796413da731245459cc5dffacc3deb2f P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:31.238494 1170 ts_tablet_manager.cc:1916] T 796413da731245459cc5dffacc3deb2f P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:31.240291 1166 consensus_queue.cc:260] T 742b3ec3f6604647b9e397532f1673dd P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.241369 1166 ts_tablet_manager.cc:1434] T 742b3ec3f6604647b9e397532f1673dd P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20251025 07:52:31.241670 1166 tablet_bootstrap.cc:492] T 85fba8b37f2c4540aa81701052bf222a P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:31.244060 1171 ts_tablet_manager.cc:1929] T 198db9a45257435f86df2605992ccb34 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251025 07:52:31.218070 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 5fca357d7e24494fb6af4bd73d98833c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5fca357d7e24494fb6af4bd73d98833c already in progress: creating tablet
W20251025 07:52:31.247886 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 8956ad28b35d45f28c683ac055bba441 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8956ad28b35d45f28c683ac055bba441 already in progress: creating tablet
W20251025 07:52:31.248250 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet bef0ef23d90d480d9c99aef21d7271fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bef0ef23d90d480d9c99aef21d7271fd already in progress: creating tablet
I20251025 07:52:31.248485 1172 ts_tablet_manager.cc:1929] T 567a1f5c216a4a519249da66a2a03ea4 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251025 07:52:31.248512 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 4037dee385d04e85bb78c8e77cb00379 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4037dee385d04e85bb78c8e77cb00379 already in progress: creating tablet
I20251025 07:52:31.248643 1172 log.cc:1199] T 567a1f5c216a4a519249da66a2a03ea4 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/567a1f5c216a4a519249da66a2a03ea4
W20251025 07:52:31.248852 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet be5ab0db6ac34c2c9a370445e251eb46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be5ab0db6ac34c2c9a370445e251eb46 already in progress: creating tablet
I20251025 07:52:31.249024 1172 ts_tablet_manager.cc:1950] T 567a1f5c216a4a519249da66a2a03ea4 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:31.249130 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet cf2cab1552bd4e7394797c93b149f450 (table default.Table_4c12bad86d7946b9ac019c8e3856dcba [id=7de1e93439f94b2ebd4db3ed27f95872]) successfully deleted
W20251025 07:52:31.249366 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 53198283718d4b45b30df65b5bde96a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53198283718d4b45b30df65b5bde96a6 already in progress: creating tablet
W20251025 07:52:31.249594 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet f97acc9a638b4d5a968dcc3412a384bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f97acc9a638b4d5a968dcc3412a384bc already in progress: creating tablet
W20251025 07:52:31.249922 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet db99a1e68daf4e948a3d66ad276cc5f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db99a1e68daf4e948a3d66ad276cc5f9 already in progress: creating tablet
I20251025 07:52:31.250948 1169 tablet_bootstrap.cc:492] T 85fba8b37f2c4540aa81701052bf222a P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:31.251165 1169 ts_tablet_manager.cc:1403] T 85fba8b37f2c4540aa81701052bf222a P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.214s user 0.003s sys 0.007s
I20251025 07:52:31.251930 1169 raft_consensus.cc:359] T 85fba8b37f2c4540aa81701052bf222a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.252197 1169 raft_consensus.cc:385] T 85fba8b37f2c4540aa81701052bf222a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:31.252300 1169 raft_consensus.cc:740] T 85fba8b37f2c4540aa81701052bf222a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:31.252596 1169 consensus_queue.cc:260] T 85fba8b37f2c4540aa81701052bf222a P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.253028 1166 tablet_bootstrap.cc:654] T 85fba8b37f2c4540aa81701052bf222a P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:31.253401 1169 ts_tablet_manager.cc:1434] T 85fba8b37f2c4540aa81701052bf222a P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:31.245294 1167 tablet_bootstrap.cc:492] T 85e27fc52e9a4d62a8622645594f59c6 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:31.255705 1167 ts_tablet_manager.cc:1403] T 85e27fc52e9a4d62a8622645594f59c6 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.002s
I20251025 07:52:31.256546 1167 raft_consensus.cc:359] T 85e27fc52e9a4d62a8622645594f59c6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.256810 1167 raft_consensus.cc:385] T 85e27fc52e9a4d62a8622645594f59c6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:31.256950 1167 raft_consensus.cc:740] T 85e27fc52e9a4d62a8622645594f59c6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:31.257256 1167 consensus_queue.cc:260] T 85e27fc52e9a4d62a8622645594f59c6 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.260936 1167 ts_tablet_manager.cc:1434] T 85e27fc52e9a4d62a8622645594f59c6 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251025 07:52:31.261314 1167 tablet_bootstrap.cc:492] T 627a10a0714f4be8bdbced503fd6388d P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:31.264143 1167 tablet_bootstrap.cc:654] T 627a10a0714f4be8bdbced503fd6388d P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:31.265156 1171 log.cc:1199] T 198db9a45257435f86df2605992ccb34 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/198db9a45257435f86df2605992ccb34
I20251025 07:52:31.265612 1171 ts_tablet_manager.cc:1950] T 198db9a45257435f86df2605992ccb34 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:31.266531 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 331) took 1131 ms. Trace:
I20251025 07:52:31.266705 1171 rpcz_store.cc:276] 1025 07:52:30.135022 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:30.138116 (+ 3094us) service_pool.cc:225] Handling call
1025 07:52:31.233858 (+1095742us) ts_tablet_manager.cc:1151] Deleting tablet 198db9a45257435f86df2605992ccb34
1025 07:52:31.233869 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:31.239325 (+ 5456us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:31.243947 (+ 4622us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:31.266478 (+ 22531us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1095489}
I20251025 07:52:31.267858 1171 tablet_replica.cc:333] T 1b31177a062b467187d68ed2034f5690 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:31.268110 1171 raft_consensus.cc:2243] T 1b31177a062b467187d68ed2034f5690 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:31.268345 1171 raft_consensus.cc:2272] T 1b31177a062b467187d68ed2034f5690 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:31.269012 1171 ts_tablet_manager.cc:1916] T 1b31177a062b467187d68ed2034f5690 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:31.275885 1166 tablet_bootstrap.cc:492] T 85fba8b37f2c4540aa81701052bf222a P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:31.276063 1166 ts_tablet_manager.cc:1403] T 85fba8b37f2c4540aa81701052bf222a P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.034s user 0.006s sys 0.002s
I20251025 07:52:31.277060 1166 raft_consensus.cc:359] T 85fba8b37f2c4540aa81701052bf222a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.277285 1166 raft_consensus.cc:385] T 85fba8b37f2c4540aa81701052bf222a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:31.277379 1166 raft_consensus.cc:740] T 85fba8b37f2c4540aa81701052bf222a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:31.277702 1166 consensus_queue.cc:260] T 85fba8b37f2c4540aa81701052bf222a P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.279318 1172 tablet_replica.cc:333] T 1aeaad5c5d01478bb997254ac372d9a6 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:31.279525 1172 raft_consensus.cc:2243] T 1aeaad5c5d01478bb997254ac372d9a6 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:31.279731 1172 raft_consensus.cc:2272] T 1aeaad5c5d01478bb997254ac372d9a6 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:31.280313 1172 ts_tablet_manager.cc:1916] T 1aeaad5c5d01478bb997254ac372d9a6 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:31.280822 1166 ts_tablet_manager.cc:1434] T 85fba8b37f2c4540aa81701052bf222a P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251025 07:52:31.283491 1166 tablet_bootstrap.cc:492] T 47284f96928749e6a5201103a8ee5f9f P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:31.286019 1166 tablet_bootstrap.cc:654] T 47284f96928749e6a5201103a8ee5f9f P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:31.292375 1171 ts_tablet_manager.cc:1929] T 1b31177a062b467187d68ed2034f5690 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:31.293774 1169 tablet_bootstrap.cc:492] T 90764275825c4ffb8712cb883d37a551 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:31.293915 1166 tablet_bootstrap.cc:492] T 47284f96928749e6a5201103a8ee5f9f P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:31.294075 1166 ts_tablet_manager.cc:1403] T 47284f96928749e6a5201103a8ee5f9f P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.001s
I20251025 07:52:31.294874 1166 raft_consensus.cc:359] T 47284f96928749e6a5201103a8ee5f9f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.295073 1166 raft_consensus.cc:385] T 47284f96928749e6a5201103a8ee5f9f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:31.295174 1166 raft_consensus.cc:740] T 47284f96928749e6a5201103a8ee5f9f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:31.296376 1166 consensus_queue.cc:260] T 47284f96928749e6a5201103a8ee5f9f P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.301128 1166 ts_tablet_manager.cc:1434] T 47284f96928749e6a5201103a8ee5f9f P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20251025 07:52:31.305218 1171 log.cc:1199] T 1b31177a062b467187d68ed2034f5690 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/1b31177a062b467187d68ed2034f5690
I20251025 07:52:31.305581 1166 tablet_bootstrap.cc:492] T 8e6d76b1a97945cfb6c3055e1b622f8a P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:31.307358 1171 ts_tablet_manager.cc:1950] T 1b31177a062b467187d68ed2034f5690 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:31.307822 1166 tablet_bootstrap.cc:654] T 8e6d76b1a97945cfb6c3055e1b622f8a P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:31.308728 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 332) took 1173 ms. Trace:
I20251025 07:52:31.311538 1171 rpcz_store.cc:276] 1025 07:52:30.135711 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:30.139022 (+ 3311us) service_pool.cc:225] Handling call
1025 07:52:31.267194 (+1128172us) ts_tablet_manager.cc:1151] Deleting tablet 1b31177a062b467187d68ed2034f5690
1025 07:52:31.267203 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:31.283319 (+ 16116us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:31.292201 (+ 8882us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:31.308692 (+ 16491us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1121325}
I20251025 07:52:31.312773 1171 tablet_replica.cc:333] T 79bc2678df8042e3994b0471afa2c304 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:31.313282 1171 raft_consensus.cc:2243] T 79bc2678df8042e3994b0471afa2c304 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:31.314894 1169 tablet_bootstrap.cc:654] T 90764275825c4ffb8712cb883d37a551 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:31.316045 1166 tablet_bootstrap.cc:492] T 8e6d76b1a97945cfb6c3055e1b622f8a P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:31.316219 1166 ts_tablet_manager.cc:1403] T 8e6d76b1a97945cfb6c3055e1b622f8a P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20251025 07:52:31.317041 1166 raft_consensus.cc:359] T 8e6d76b1a97945cfb6c3055e1b622f8a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.317251 1166 raft_consensus.cc:385] T 8e6d76b1a97945cfb6c3055e1b622f8a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:31.317324 1166 raft_consensus.cc:740] T 8e6d76b1a97945cfb6c3055e1b622f8a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:31.317600 1166 consensus_queue.cc:260] T 8e6d76b1a97945cfb6c3055e1b622f8a P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.318365 1171 raft_consensus.cc:2272] T 79bc2678df8042e3994b0471afa2c304 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:31.319237 1171 ts_tablet_manager.cc:1916] T 79bc2678df8042e3994b0471afa2c304 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:31.322748 1561 raft_consensus.cc:493] T 85fba8b37f2c4540aa81701052bf222a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:31.322908 1561 raft_consensus.cc:515] T 85fba8b37f2c4540aa81701052bf222a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.323633 1561 leader_election.cc:290] T 85fba8b37f2c4540aa81701052bf222a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:31.324443 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85fba8b37f2c4540aa81701052bf222a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:31.324676 775 raft_consensus.cc:2468] T 85fba8b37f2c4540aa81701052bf222a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:31.325189 847 leader_election.cc:304] T 85fba8b37f2c4540aa81701052bf222a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:31.325400 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_23d4d4a3c90c47ea8230c39a51138418"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:31.325894 1561 raft_consensus.cc:2804] T 85fba8b37f2c4540aa81701052bf222a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:31.325999 1561 raft_consensus.cc:493] T 85fba8b37f2c4540aa81701052bf222a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:31.326090 1561 raft_consensus.cc:3060] T 85fba8b37f2c4540aa81701052bf222a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:31.327363 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 292fe04055954afd8c08ac2636b68191 consensus state has no leader
I20251025 07:52:31.327766 519 master_service.cc:954] ReplaceTablet: received request to replace tablet 5cba9c05dd424cc788febc614a34819a from {username='slave'} at 127.0.0.1:45978
W20251025 07:52:31.328908 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c501ef5d70e742598c1a902efe47feb1 consensus state has no leader
I20251025 07:52:31.328961 1561 raft_consensus.cc:515] T 85fba8b37f2c4540aa81701052bf222a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.329979 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85fba8b37f2c4540aa81701052bf222a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:31.330194 775 raft_consensus.cc:3060] T 85fba8b37f2c4540aa81701052bf222a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:31.330528 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2e098c9c4613459295b6043048cba587 has no consensus state
I20251025 07:52:31.330737 522 master_service.cc:954] ReplaceTablet: received request to replace tablet 0f5f6ca6c3cd4d369690e425387a4344 from {username='slave'} at 127.0.0.1:45978
W20251025 07:52:31.330945 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23d4d4a3c90c47ea8230c39a51138418 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:31.332041 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_8d39f75f88854f528f5cac09c3b0423a" } modify_external_catalogs: true
I20251025 07:52:31.332203 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_8d39f75f88854f528f5cac09c3b0423a" } modify_external_catalogs: true
I20251025 07:52:31.332585 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_3193d4af1aaa42a28d4eda999726f033" } new_table_name: "default.Table_eb2e66417bc9444085fb5ba50ee3853e" modify_external_catalogs: true
I20251025 07:52:31.333637 775 raft_consensus.cc:2468] T 85fba8b37f2c4540aa81701052bf222a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:31.334233 847 leader_election.cc:304] T 85fba8b37f2c4540aa81701052bf222a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:31.334578 1218 raft_consensus.cc:2804] T 85fba8b37f2c4540aa81701052bf222a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:31.334756 1218 raft_consensus.cc:697] T 85fba8b37f2c4540aa81701052bf222a P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:31.335026 1218 consensus_queue.cc:237] T 85fba8b37f2c4540aa81701052bf222a P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.336503 1172 ts_tablet_manager.cc:1929] T 1aeaad5c5d01478bb997254ac372d9a6 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:31.336642 1172 log.cc:1199] T 1aeaad5c5d01478bb997254ac372d9a6 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/1aeaad5c5d01478bb997254ac372d9a6
I20251025 07:52:31.337009 1172 ts_tablet_manager.cc:1950] T 1aeaad5c5d01478bb997254ac372d9a6 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:31.337929 1561 leader_election.cc:290] T 85fba8b37f2c4540aa81701052bf222a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:31.338891 1171 ts_tablet_manager.cc:1929] T 79bc2678df8042e3994b0471afa2c304 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:31.339694 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_fd1c5aa76b5c48caad4824c7a5d52347"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:31.340395 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd1c5aa76b5c48caad4824c7a5d52347 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:31.340549 1171 log.cc:1199] T 79bc2678df8042e3994b0471afa2c304 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/79bc2678df8042e3994b0471afa2c304
I20251025 07:52:31.340960 1171 ts_tablet_manager.cc:1950] T 79bc2678df8042e3994b0471afa2c304 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:31.341439 1166 ts_tablet_manager.cc:1434] T 8e6d76b1a97945cfb6c3055e1b622f8a P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.025s user 0.001s sys 0.001s
I20251025 07:52:31.341784 1166 tablet_bootstrap.cc:492] T 08da9d2687704725aacd11cd92c6a9f9 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:31.341826 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_d647b0fa975743998d7f49c36abe9c96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:31.342367 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_208daf030a0d4efdb6ef8250b2f48896"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:31.342876 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_208daf030a0d4efdb6ef8250b2f48896 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:31.343528 1166 tablet_bootstrap.cc:654] T 08da9d2687704725aacd11cd92c6a9f9 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:31.344830 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d647b0fa975743998d7f49c36abe9c96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:31.345343 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 333) took 1141 ms. Trace:
I20251025 07:52:31.345438 1171 rpcz_store.cc:276] 1025 07:52:30.203776 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:30.203973 (+ 197us) service_pool.cc:225] Handling call
1025 07:52:31.312665 (+1108692us) ts_tablet_manager.cc:1151] Deleting tablet 79bc2678df8042e3994b0471afa2c304
1025 07:52:31.312677 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:31.325143 (+ 12466us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:31.338763 (+ 13620us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:31.345271 (+ 6508us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1108148}
I20251025 07:52:31.345952 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_dac528b081844b7c86027169c6234c27" } modify_external_catalogs: true
I20251025 07:52:31.346220 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_dac528b081844b7c86027169c6234c27" } modify_external_catalogs: true
I20251025 07:52:31.347654 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_bef94ac5c2bb4f3e8cbbc68037145d6a" } new_table_name: "default.Table_e90d7f5ff13b40878e5b625c97792c7d" modify_external_catalogs: true
I20251025 07:52:31.348897 1172 tablet_replica.cc:333] T 90764275825c4ffb8712cb883d37a551 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:31.349184 1172 raft_consensus.cc:2243] T 90764275825c4ffb8712cb883d37a551 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:31.349387 1172 raft_consensus.cc:2272] T 90764275825c4ffb8712cb883d37a551 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:31.350116 1172 ts_tablet_manager.cc:1916] T 90764275825c4ffb8712cb883d37a551 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:31.351122 1166 tablet_bootstrap.cc:492] T 08da9d2687704725aacd11cd92c6a9f9 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:31.351272 1166 ts_tablet_manager.cc:1403] T 08da9d2687704725aacd11cd92c6a9f9 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20251025 07:52:31.352159 1166 raft_consensus.cc:359] T 08da9d2687704725aacd11cd92c6a9f9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.352350 1166 raft_consensus.cc:385] T 08da9d2687704725aacd11cd92c6a9f9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:31.352459 1166 raft_consensus.cc:740] T 08da9d2687704725aacd11cd92c6a9f9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:31.352703 1166 consensus_queue.cc:260] T 08da9d2687704725aacd11cd92c6a9f9 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:31.353075 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet f656945454fe4921ab819c772a089eb3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f656945454fe4921ab819c772a089eb3 already in progress: creating tablet
I20251025 07:52:31.353238 1166 ts_tablet_manager.cc:1434] T 08da9d2687704725aacd11cd92c6a9f9 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20251025 07:52:31.353379 511 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:43563, user_credentials={real_user=slave}} blocked reactor thread for 133530us
I20251025 07:52:31.353562 1166 tablet_bootstrap.cc:492] T 5cba9c05dd424cc788febc614a34819a P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
W20251025 07:52:31.353718 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 896bd7870d87408e8adfdb7a78998111 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 896bd7870d87408e8adfdb7a78998111 already in progress: creating tablet
W20251025 07:52:31.353994 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet ebc824ec4c60411dab1a2ea4a385148e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ebc824ec4c60411dab1a2ea4a385148e already in progress: creating tablet
I20251025 07:52:31.354287 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 567a1f5c216a4a519249da66a2a03ea4 (table default.Table_b6cf4ff8abe642088cedf32609854bdd [id=e55f743deb3c45f9a90c2f3a0ab68a3c]) successfully deleted
I20251025 07:52:31.354516 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 1aeaad5c5d01478bb997254ac372d9a6 (table default.Table_ebcfacbbec3147a189ff08cec113fc55 [id=8383812e49d24d98bf0ef5f985376d98]) successfully deleted
I20251025 07:52:31.361349 1171 tablet_replica.cc:333] T 85fba8b37f2c4540aa81701052bf222a P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:31.361588 1171 raft_consensus.cc:2243] T 85fba8b37f2c4540aa81701052bf222a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:31.361781 1171 raft_consensus.cc:2272] T 85fba8b37f2c4540aa81701052bf222a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:31.362351 1171 ts_tablet_manager.cc:1916] T 85fba8b37f2c4540aa81701052bf222a P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:31.370566 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9a8e2658a81e436298fe2190726080d4 consensus state has no leader
I20251025 07:52:31.371899 1166 tablet_bootstrap.cc:654] T 5cba9c05dd424cc788febc614a34819a P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:31.373724 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet a4e4b99be0434baaa75d988281797846
I20251025 07:52:31.381768 1166 tablet_bootstrap.cc:492] T 5cba9c05dd424cc788febc614a34819a P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
W20251025 07:52:31.381999 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet df0d86ad301245c1a6bcb44805936d25 has no consensus state
I20251025 07:52:31.382519 1166 ts_tablet_manager.cc:1403] T 5cba9c05dd424cc788febc614a34819a P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.000s
I20251025 07:52:31.383309 1166 raft_consensus.cc:359] T 5cba9c05dd424cc788febc614a34819a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.383502 1166 raft_consensus.cc:385] T 5cba9c05dd424cc788febc614a34819a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:31.383572 1166 raft_consensus.cc:740] T 5cba9c05dd424cc788febc614a34819a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:31.383976 1171 ts_tablet_manager.cc:1929] T 85fba8b37f2c4540aa81701052bf222a P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:31.385236 1166 consensus_queue.cc:260] T 5cba9c05dd424cc788febc614a34819a P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.385752 1166 ts_tablet_manager.cc:1434] T 5cba9c05dd424cc788febc614a34819a P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251025 07:52:31.386082 1166 tablet_bootstrap.cc:492] T 21e52928580c4762b0ae7152490ef955 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:31.390275 1171 log.cc:1199] T 85fba8b37f2c4540aa81701052bf222a P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/85fba8b37f2c4540aa81701052bf222a
I20251025 07:52:31.390578 890 tablet_service.cc:1552] Processing DeleteTablet for tablet a4e4b99be0434baaa75d988281797846 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:31.391145 1171 ts_tablet_manager.cc:1950] T 85fba8b37f2c4540aa81701052bf222a P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:31.391149 1172 ts_tablet_manager.cc:1929] T 90764275825c4ffb8712cb883d37a551 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:31.391312 1172 log.cc:1199] T 90764275825c4ffb8712cb883d37a551 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/90764275825c4ffb8712cb883d37a551
I20251025 07:52:31.391695 1172 ts_tablet_manager.cc:1950] T 90764275825c4ffb8712cb883d37a551 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:31.392120 1166 tablet_bootstrap.cc:654] T 21e52928580c4762b0ae7152490ef955 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:31.394840 1171 tablet_replica.cc:333] T 5caac9c943bf42a5b2ae522d87fdfe8e P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:31.395308 1171 raft_consensus.cc:2243] T 5caac9c943bf42a5b2ae522d87fdfe8e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:31.395795 1171 raft_consensus.cc:2272] T 5caac9c943bf42a5b2ae522d87fdfe8e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:31.396634 1171 ts_tablet_manager.cc:1916] T 5caac9c943bf42a5b2ae522d87fdfe8e P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:31.396808 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_49beae553d03426aa395b7cdb6dfec9a" } modify_external_catalogs: true
I20251025 07:52:31.397068 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_49beae553d03426aa395b7cdb6dfec9a" } modify_external_catalogs: true
I20251025 07:52:31.329907 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85fba8b37f2c4540aa81701052bf222a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:31.339440 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85fba8b37f2c4540aa81701052bf222a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:31.402421 1049 raft_consensus.cc:3060] T 85fba8b37f2c4540aa81701052bf222a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:31.405404 1049 raft_consensus.cc:2468] T 85fba8b37f2c4540aa81701052bf222a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:31.405814 1048 raft_consensus.cc:2376] T 85fba8b37f2c4540aa81701052bf222a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1. Re-sending same reply.
I20251025 07:52:31.381516 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet a4e4b99be0434baaa75d988281797846 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:31.409282 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 90764275825c4ffb8712cb883d37a551 (table default.Table_072e0092adb64ef0ac6834dc18688ccf [id=1d4cbe4a4f38445297ceeebb8ffdaa8e]) successfully deleted
I20251025 07:52:31.412580 1172 tablet_replica.cc:333] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:31.412885 1172 raft_consensus.cc:2243] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:31.413368 1172 raft_consensus.cc:2272] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:31.414090 1172 ts_tablet_manager.cc:1916] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:31.415858 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:43563, user_credentials={real_user=slave}} blocked reactor thread for 116707us
I20251025 07:52:31.409699 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_6a95b96e977244b6b28688afee389b84" } modify_external_catalogs: true
I20251025 07:52:31.416818 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_6a95b96e977244b6b28688afee389b84" } modify_external_catalogs: true
I20251025 07:52:31.414239 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_26b3fa566ddf4e5aadd68de388cb61ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:31.418783 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26b3fa566ddf4e5aadd68de388cb61ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:31.415675 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet f0bdaa8cfd764f1b83f47881353d26b6
I20251025 07:52:31.421305 1166 tablet_bootstrap.cc:492] T 21e52928580c4762b0ae7152490ef955 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:31.421459 1166 ts_tablet_manager.cc:1403] T 21e52928580c4762b0ae7152490ef955 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.006s
W20251025 07:52:31.421667 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 7e6544202d4f489d82f22d6073ef240d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e6544202d4f489d82f22d6073ef240d already in progress: creating tablet
W20251025 07:52:31.421934 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 5b6383c99e9e4c90a374da9ccd830a6f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b6383c99e9e4c90a374da9ccd830a6f already in progress: creating tablet
I20251025 07:52:31.422204 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet f0bdaa8cfd764f1b83f47881353d26b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:31.422268 1166 raft_consensus.cc:359] T 21e52928580c4762b0ae7152490ef955 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.422446 1166 raft_consensus.cc:385] T 21e52928580c4762b0ae7152490ef955 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:31.422562 1166 raft_consensus.cc:740] T 21e52928580c4762b0ae7152490ef955 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:31.422840 1166 consensus_queue.cc:260] T 21e52928580c4762b0ae7152490ef955 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.422852 1577 tablet.cc:1722] T cd23b8094d7e472eba6bd170248858a9 P db42be7ad3c741a0aa427f27e245993b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251025 07:52:31.423610 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 91728987303b41fd840eb56f3a311c5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91728987303b41fd840eb56f3a311c5f already in progress: creating tablet
I20251025 07:52:31.423626 1171 ts_tablet_manager.cc:1929] T 5caac9c943bf42a5b2ae522d87fdfe8e P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:31.423790 1171 log.cc:1199] T 5caac9c943bf42a5b2ae522d87fdfe8e P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/5caac9c943bf42a5b2ae522d87fdfe8e
W20251025 07:52:31.423861 510 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:46231, user_credentials={real_user=slave}} blocked reactor thread for 173721us
I20251025 07:52:31.424263 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 198db9a45257435f86df2605992ccb34 (table default.Table_79969e4ace6f4227b2748b59136027d6 [id=a1135611169142ba9ef987e3e9f95ce8]) successfully deleted
I20251025 07:52:31.424587 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 1b31177a062b467187d68ed2034f5690 (table default.Table_e50bf47b78d94bbc811e6f3fb3551edb [id=0827603890c8466082bc6c306c9c66af]) successfully deleted
I20251025 07:52:31.425027 1171 ts_tablet_manager.cc:1950] T 5caac9c943bf42a5b2ae522d87fdfe8e P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:31.426698 1166 ts_tablet_manager.cc:1434] T 21e52928580c4762b0ae7152490ef955 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:31.427065 1166 tablet_bootstrap.cc:492] T ac0ea298bc114c56a8093e12def21ce1 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:31.429540 1166 tablet_bootstrap.cc:654] T ac0ea298bc114c56a8093e12def21ce1 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:31.430495 1576 tablet.cc:1722] T cd23b8094d7e472eba6bd170248858a9 P 3aea2869ff814cc7b142012faa274382: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251025 07:52:31.437880 1171 tablet_replica.cc:333] T f51ec427dd9349a48eb221db04cf242b P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:31.438383 1171 raft_consensus.cc:2243] T f51ec427dd9349a48eb221db04cf242b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:31.438895 1171 raft_consensus.cc:2272] T f51ec427dd9349a48eb221db04cf242b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:31.439905 1171 ts_tablet_manager.cc:1916] T f51ec427dd9349a48eb221db04cf242b P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:31.448004 1561 raft_consensus.cc:493] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:31.448170 1561 raft_consensus.cc:515] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.448794 1561 leader_election.cc:290] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:31.449944 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627a10a0714f4be8bdbced503fd6388d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
W20251025 07:52:31.450665 847 leader_election.cc:343] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:31.451193 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627a10a0714f4be8bdbced503fd6388d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:31.451476 1048 raft_consensus.cc:2468] T 627a10a0714f4be8bdbced503fd6388d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:31.452126 850 leader_election.cc:304] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b
I20251025 07:52:31.452744 1561 raft_consensus.cc:2804] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:31.452881 1561 raft_consensus.cc:493] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:31.452979 1561 raft_consensus.cc:3060] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:31.453688 1166 tablet_bootstrap.cc:492] T ac0ea298bc114c56a8093e12def21ce1 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:31.453871 1166 ts_tablet_manager.cc:1403] T ac0ea298bc114c56a8093e12def21ce1 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.011s
I20251025 07:52:31.455572 1166 raft_consensus.cc:359] T ac0ea298bc114c56a8093e12def21ce1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.455782 1166 raft_consensus.cc:385] T ac0ea298bc114c56a8093e12def21ce1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:31.455893 1166 raft_consensus.cc:740] T ac0ea298bc114c56a8093e12def21ce1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:31.456243 1166 consensus_queue.cc:260] T ac0ea298bc114c56a8093e12def21ce1 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.456756 1166 ts_tablet_manager.cc:1434] T ac0ea298bc114c56a8093e12def21ce1 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251025 07:52:31.457073 1166 tablet_bootstrap.cc:492] T fd9b4be139d347f4a6a60767029c342a P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:31.457793 1561 raft_consensus.cc:515] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.458418 1171 ts_tablet_manager.cc:1929] T f51ec427dd9349a48eb221db04cf242b P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:31.458560 1171 log.cc:1199] T f51ec427dd9349a48eb221db04cf242b P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/f51ec427dd9349a48eb221db04cf242b
I20251025 07:52:31.458930 1166 tablet_bootstrap.cc:654] T fd9b4be139d347f4a6a60767029c342a P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:31.458943 1171 ts_tablet_manager.cc:1950] T f51ec427dd9349a48eb221db04cf242b P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:31.459232 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627a10a0714f4be8bdbced503fd6388d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:31.459869 1171 tablet_replica.cc:333] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
W20251025 07:52:31.459826 847 leader_election.cc:343] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:31.460088 1171 raft_consensus.cc:2243] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:31.460289 1171 raft_consensus.cc:2272] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:31.460850 1171 ts_tablet_manager.cc:1916] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:31.460969 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627a10a0714f4be8bdbced503fd6388d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:31.461162 1048 raft_consensus.cc:3060] T 627a10a0714f4be8bdbced503fd6388d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:31.463068 1561 leader_election.cc:290] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:31.464800 1048 raft_consensus.cc:2468] T 627a10a0714f4be8bdbced503fd6388d P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:31.465358 850 leader_election.cc:304] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b
I20251025 07:52:31.465917 1561 raft_consensus.cc:2804] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:31.466040 1172 ts_tablet_manager.cc:1929] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20251025 07:52:31.475848 1170 ts_tablet_manager.cc:1929] T 796413da731245459cc5dffacc3deb2f P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 07:52:31.484459 1172 log.cc:1199] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/50ffbcdd4a9247bf99b8e2361bc02ef5
I20251025 07:52:31.484768 1171 ts_tablet_manager.cc:1929] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20251025 07:52:31.484910 1171 log.cc:1199] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/50ffbcdd4a9247bf99b8e2361bc02ef5
I20251025 07:52:31.485077 1172 ts_tablet_manager.cc:1950] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:31.485280 1171 ts_tablet_manager.cc:1950] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:31.466464 1561 raft_consensus.cc:697] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:31.488148 1561 consensus_queue.cc:237] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:31.442728 513 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 2c50ec95e48e49b59a76fac4fc862d3d (table default.Table_ad729c05dcae46f19df251a219aeccc4 [id=c28b0b6acd61490caecfdaab803a9d92], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2c50ec95e48e49b59a76fac4fc862d3d consensus state has no leader
W20251025 07:52:31.489394 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 410ef150a7014c6c867a021b544a2f4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 410ef150a7014c6c867a021b544a2f4e already in progress: creating tablet
W20251025 07:52:31.496536 510 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:46231, user_credentials={real_user=slave}} blocked reactor thread for 70147.4us
I20251025 07:52:31.496906 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 79bc2678df8042e3994b0471afa2c304 (table default.Table_ca46a7a783e547998d1be8476801a9e0 [id=525e8a793cf14cd492ca2ee986231311]) successfully deleted
I20251025 07:52:31.497013 1172 tablet_replica.cc:333] T 85e27fc52e9a4d62a8622645594f59c6 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:31.497260 1172 raft_consensus.cc:2243] T 85e27fc52e9a4d62a8622645594f59c6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:31.497440 1172 raft_consensus.cc:2272] T 85e27fc52e9a4d62a8622645594f59c6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:31.498024 1172 ts_tablet_manager.cc:1916] T 85e27fc52e9a4d62a8622645594f59c6 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:31.500970 521 master_service.cc:954] ReplaceTablet: received request to replace tablet f43ef51ee45d47a99ec732b49d50d3b4 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:31.501520 1171 tablet_replica.cc:333] T 796413da731245459cc5dffacc3deb2f P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:31.501940 1171 raft_consensus.cc:2243] T 796413da731245459cc5dffacc3deb2f P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:31.502489 1171 raft_consensus.cc:2272] T 796413da731245459cc5dffacc3deb2f P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:31.503391 1171 ts_tablet_manager.cc:1916] T 796413da731245459cc5dffacc3deb2f P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:31.513554 1166 tablet_bootstrap.cc:492] T fd9b4be139d347f4a6a60767029c342a P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
W20251025 07:52:31.512907 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 551d07a9242740dcb9366bb2d707a4a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 551d07a9242740dcb9366bb2d707a4a8 already in progress: creating tablet
I20251025 07:52:31.514087 1166 ts_tablet_manager.cc:1403] T fd9b4be139d347f4a6a60767029c342a P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.057s user 0.027s sys 0.019s
I20251025 07:52:31.514935 1166 raft_consensus.cc:359] T fd9b4be139d347f4a6a60767029c342a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.515244 1166 raft_consensus.cc:385] T fd9b4be139d347f4a6a60767029c342a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:31.515369 1166 raft_consensus.cc:740] T fd9b4be139d347f4a6a60767029c342a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:31.515658 1166 consensus_queue.cc:260] T fd9b4be139d347f4a6a60767029c342a P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.518689 1166 ts_tablet_manager.cc:1434] T fd9b4be139d347f4a6a60767029c342a P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251025 07:52:31.519219 1166 tablet_bootstrap.cc:492] T f3c87a83082b421cb4be546ac4ce7ad8 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:31.522639 1166 tablet_bootstrap.cc:654] T f3c87a83082b421cb4be546ac4ce7ad8 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:31.523696 1167 tablet_bootstrap.cc:492] T 627a10a0714f4be8bdbced503fd6388d P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:31.523867 1167 ts_tablet_manager.cc:1403] T 627a10a0714f4be8bdbced503fd6388d P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.263s user 0.029s sys 0.042s
I20251025 07:52:31.524694 1167 raft_consensus.cc:359] T 627a10a0714f4be8bdbced503fd6388d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.524993 1167 raft_consensus.cc:385] T 627a10a0714f4be8bdbced503fd6388d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:31.525118 1167 raft_consensus.cc:740] T 627a10a0714f4be8bdbced503fd6388d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:31.525643 1167 consensus_queue.cc:260] T 627a10a0714f4be8bdbced503fd6388d P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.529263 1167 ts_tablet_manager.cc:1434] T 627a10a0714f4be8bdbced503fd6388d P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251025 07:52:31.532397 1167 tablet_bootstrap.cc:492] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:31.534492 1167 tablet_bootstrap.cc:654] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:31.538311 1170 log.cc:1199] T 796413da731245459cc5dffacc3deb2f P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/796413da731245459cc5dffacc3deb2f
I20251025 07:52:31.538800 1170 ts_tablet_manager.cc:1950] T 796413da731245459cc5dffacc3deb2f P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:31.538980 1171 ts_tablet_manager.cc:1929] T 796413da731245459cc5dffacc3deb2f P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 07:52:31.539151 1171 log.cc:1199] T 796413da731245459cc5dffacc3deb2f P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/796413da731245459cc5dffacc3deb2f
I20251025 07:52:31.539357 1169 tablet_bootstrap.cc:492] T 90764275825c4ffb8712cb883d37a551 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:31.539600 1171 ts_tablet_manager.cc:1950] T 796413da731245459cc5dffacc3deb2f P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:31.540323 1169 ts_tablet_manager.cc:1403] T 90764275825c4ffb8712cb883d37a551 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.246s user 0.002s sys 0.002s
I20251025 07:52:31.540597 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 796413da731245459cc5dffacc3deb2f (table default.Table_293596c64cca4768bc5869fd9924f856 [id=d1b9d89b22bc4431a7d59ceeb9d75050]) successfully deleted
I20251025 07:52:31.541214 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_843881ca867542619b4fff840d2879ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:31.541500 1169 raft_consensus.cc:359] T 90764275825c4ffb8712cb883d37a551 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.541744 1169 raft_consensus.cc:385] T 90764275825c4ffb8712cb883d37a551 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251025 07:52:31.541806 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_843881ca867542619b4fff840d2879ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:31.541843 1169 raft_consensus.cc:740] T 90764275825c4ffb8712cb883d37a551 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:31.543481 519 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 370e79fbb11d443a8ac60d3b3bbaa335
I20251025 07:52:31.544699 1171 tablet_replica.cc:333] T 1483c40eee834f20a92e6b4cd89775b6 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:31.544678 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_dad8e9c26a9a41e2bcf3fa1db9f5b087"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:31.545089 1171 raft_consensus.cc:2243] T 1483c40eee834f20a92e6b4cd89775b6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:31.545301 1171 raft_consensus.cc:2272] T 1483c40eee834f20a92e6b4cd89775b6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:31.545665 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dad8e9c26a9a41e2bcf3fa1db9f5b087 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:31.546278 1171 ts_tablet_manager.cc:1916] T 1483c40eee834f20a92e6b4cd89775b6 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:31.546859 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet b94634ac5077439fadfbbce569f60e8b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b94634ac5077439fadfbbce569f60e8b already in progress: creating tablet
I20251025 07:52:31.547107 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_5a8143ecfd74418f928ceb9ae9cd50ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:31.547602 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a8143ecfd74418f928ceb9ae9cd50ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:31.547847 1169 consensus_queue.cc:260] T 90764275825c4ffb8712cb883d37a551 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.548774 1169 ts_tablet_manager.cc:1434] T 90764275825c4ffb8712cb883d37a551 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251025 07:52:31.540333 1170 tablet_replica.cc:333] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
W20251025 07:52:31.549015 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 2aa522477e934c89834026dc480fe7c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2aa522477e934c89834026dc480fe7c0 already in progress: creating tablet
I20251025 07:52:31.549268 1170 raft_consensus.cc:2243] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:31.549499 1170 raft_consensus.cc:2272] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:31.549733 511 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:43563, user_credentials={real_user=slave}} blocked reactor thread for 52099.3us
W20251025 07:52:31.550033 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 48ff4ce7846b4e56a37eee9bcb17bd5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48ff4ce7846b4e56a37eee9bcb17bd5e already in progress: creating tablet
I20251025 07:52:31.550244 1170 ts_tablet_manager.cc:1916] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:31.550283 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 50ffbcdd4a9247bf99b8e2361bc02ef5 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]) successfully deleted
I20251025 07:52:31.552732 1167 tablet_bootstrap.cc:492] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:31.552920 1167 ts_tablet_manager.cc:1403] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20251025 07:52:31.553678 1167 raft_consensus.cc:359] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.553921 1167 raft_consensus.cc:385] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:31.554056 1167 raft_consensus.cc:740] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:31.554355 1167 consensus_queue.cc:260] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.555742 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_3d995c9c75174000b6cb98779cc0bb2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:31.556289 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d995c9c75174000b6cb98779cc0bb2f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:31.558530 1218 raft_consensus.cc:493] T 47284f96928749e6a5201103a8ee5f9f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:31.558995 1218 raft_consensus.cc:515] T 47284f96928749e6a5201103a8ee5f9f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.559664 1218 leader_election.cc:290] T 47284f96928749e6a5201103a8ee5f9f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:31.560065 1167 ts_tablet_manager.cc:1434] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20251025 07:52:31.560254 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47284f96928749e6a5201103a8ee5f9f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:31.561022 775 raft_consensus.cc:2468] T 47284f96928749e6a5201103a8ee5f9f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:31.561630 890 tablet_service.cc:1552] Processing DeleteTablet for tablet f0bdaa8cfd764f1b83f47881353d26b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:31.562335 1167 tablet_bootstrap.cc:492] T 742b3ec3f6604647b9e397532f1673dd P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:31.562460 847 leader_election.cc:304] T 47284f96928749e6a5201103a8ee5f9f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:31.563728 1218 raft_consensus.cc:2804] T 47284f96928749e6a5201103a8ee5f9f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:31.563845 1218 raft_consensus.cc:493] T 47284f96928749e6a5201103a8ee5f9f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:31.563942 1218 raft_consensus.cc:3060] T 47284f96928749e6a5201103a8ee5f9f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:31.565002 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 467324e2432f4caca148951329caa292 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 467324e2432f4caca148951329caa292 already in progress: creating tablet
W20251025 07:52:31.565354 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 74e2d78c024849b0af5c0bde1f48c36f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74e2d78c024849b0af5c0bde1f48c36f already in progress: creating tablet
W20251025 07:52:31.565608 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet d9bfd7c0c3ad4e5b98b6708793d61d41 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9bfd7c0c3ad4e5b98b6708793d61d41 already in progress: creating tablet
W20251025 07:52:31.565793 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet f661d5909657455581dcb5e6eadd3c0b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f661d5909657455581dcb5e6eadd3c0b already in progress: creating tablet
W20251025 07:52:31.565964 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet f656945454fe4921ab819c772a089eb3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f656945454fe4921ab819c772a089eb3 already in progress: creating tablet
I20251025 07:52:31.566092 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 85fba8b37f2c4540aa81701052bf222a (table default.Table_5237c35dbfa7429e803d5008a8b3487c [id=0d597ecab6f54d1ba6cb5bd3436a9db8]) successfully deleted
W20251025 07:52:31.566293 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 896bd7870d87408e8adfdb7a78998111 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 896bd7870d87408e8adfdb7a78998111 already in progress: creating tablet
W20251025 07:52:31.566488 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet ebc824ec4c60411dab1a2ea4a385148e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ebc824ec4c60411dab1a2ea4a385148e already in progress: creating tablet
I20251025 07:52:31.566668 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 5caac9c943bf42a5b2ae522d87fdfe8e (table default.Table_60bfcc345b8649e9bc73fe3d077f46c7 [id=ed06dbbb96a24ab39619f4186ba93191]) successfully deleted
W20251025 07:52:31.566855 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 7e6544202d4f489d82f22d6073ef240d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e6544202d4f489d82f22d6073ef240d already in progress: creating tablet
I20251025 07:52:31.567070 1167 tablet_bootstrap.cc:654] T 742b3ec3f6604647b9e397532f1673dd P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:31.567075 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet b94634ac5077439fadfbbce569f60e8b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b94634ac5077439fadfbbce569f60e8b already in progress: creating tablet
I20251025 07:52:31.567061 1218 raft_consensus.cc:515] T 47284f96928749e6a5201103a8ee5f9f P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:31.567241 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 5b6383c99e9e4c90a374da9ccd830a6f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b6383c99e9e4c90a374da9ccd830a6f already in progress: creating tablet
W20251025 07:52:31.567468 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 48ff4ce7846b4e56a37eee9bcb17bd5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48ff4ce7846b4e56a37eee9bcb17bd5e already in progress: creating tablet
W20251025 07:52:31.567651 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 410ef150a7014c6c867a021b544a2f4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 410ef150a7014c6c867a021b544a2f4e already in progress: creating tablet
I20251025 07:52:31.567821 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet f51ec427dd9349a48eb221db04cf242b (table default.Table_cc49d933a4b343aeb2220c3939495834 [id=e5e538e59868418e9805492aa80f3240]) successfully deleted
I20251025 07:52:31.568001 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 50ffbcdd4a9247bf99b8e2361bc02ef5 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]) successfully deleted
I20251025 07:52:31.567997 1172 ts_tablet_manager.cc:1929] T 85e27fc52e9a4d62a8622645594f59c6 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:31.568135 1172 log.cc:1199] T 85e27fc52e9a4d62a8622645594f59c6 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/85e27fc52e9a4d62a8622645594f59c6
I20251025 07:52:31.568240 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 796413da731245459cc5dffacc3deb2f (table default.Table_293596c64cca4768bc5869fd9924f856 [id=d1b9d89b22bc4431a7d59ceeb9d75050]) successfully deleted
W20251025 07:52:31.568475 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 4f0c4f0e8f014e59be9e46d2afe0be9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f0c4f0e8f014e59be9e46d2afe0be9d already in progress: creating tablet
I20251025 07:52:31.568506 1172 ts_tablet_manager.cc:1950] T 85e27fc52e9a4d62a8622645594f59c6 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:31.568472 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47284f96928749e6a5201103a8ee5f9f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:31.568691 775 raft_consensus.cc:3060] T 47284f96928749e6a5201103a8ee5f9f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:31.568758 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet c59f6e6eb9594a8e83f109a09b034045 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c59f6e6eb9594a8e83f109a09b034045 already in progress: creating tablet
I20251025 07:52:31.569346 744 tablet_service.cc:1552] Processing DeleteTablet for tablet 47284f96928749e6a5201103a8ee5f9f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:31.569486 1172 tablet_replica.cc:333] T 796413da731245459cc5dffacc3deb2f P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:31.569697 1172 raft_consensus.cc:2243] T 796413da731245459cc5dffacc3deb2f P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:31.569948 744 tablet_service.cc:1552] Processing DeleteTablet for tablet a4e4b99be0434baaa75d988281797846 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:31.570030 1172 raft_consensus.cc:2272] T 796413da731245459cc5dffacc3deb2f P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:31.570786 744 tablet_service.cc:1552] Processing DeleteTablet for tablet dd898450f2604987b08639cf87d0e402 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:48830
W20251025 07:52:31.571682 850 leader_election.cc:343] T 47284f96928749e6a5201103a8ee5f9f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 47284f96928749e6a5201103a8ee5f9f
I20251025 07:52:31.572692 744 tablet_service.cc:1552] Processing DeleteTablet for tablet 85e27fc52e9a4d62a8622645594f59c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:31.573485 1441 raft_consensus.cc:493] T dd898450f2604987b08639cf87d0e402 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 07:52:31.573938 850 leader_election.cc:343] T 47284f96928749e6a5201103a8ee5f9f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 47284f96928749e6a5201103a8ee5f9f
I20251025 07:52:31.574311 1441 raft_consensus.cc:515] T dd898450f2604987b08639cf87d0e402 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.575093 1441 leader_election.cc:290] T dd898450f2604987b08639cf87d0e402 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:31.575352 775 raft_consensus.cc:2468] T 47284f96928749e6a5201103a8ee5f9f P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:31.575799 847 leader_election.cc:304] T 47284f96928749e6a5201103a8ee5f9f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:31.576050 1218 leader_election.cc:290] T 47284f96928749e6a5201103a8ee5f9f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:31.576339 1218 raft_consensus.cc:2804] T 47284f96928749e6a5201103a8ee5f9f P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
W20251025 07:52:31.576745 710 leader_election.cc:343] T dd898450f2604987b08639cf87d0e402 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: dd898450f2604987b08639cf87d0e402
I20251025 07:52:31.577021 1218 raft_consensus.cc:697] T 47284f96928749e6a5201103a8ee5f9f P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
W20251025 07:52:31.577064 712 leader_election.cc:343] T dd898450f2604987b08639cf87d0e402 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: dd898450f2604987b08639cf87d0e402
I20251025 07:52:31.577183 712 leader_election.cc:304] T dd898450f2604987b08639cf87d0e402 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:31.577373 1218 consensus_queue.cc:237] T 47284f96928749e6a5201103a8ee5f9f P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.577656 1441 raft_consensus.cc:2749] T dd898450f2604987b08639cf87d0e402 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:31.578500 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 85e27fc52e9a4d62a8622645594f59c6 (table default.Table_4bafafcc9e8641b687d438725186f639 [id=73f69c9574ee4019ada5af76bd902107]) successfully deleted
I20251025 07:52:31.579377 1166 tablet_bootstrap.cc:492] T f3c87a83082b421cb4be546ac4ce7ad8 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:31.579584 1166 ts_tablet_manager.cc:1403] T f3c87a83082b421cb4be546ac4ce7ad8 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.060s user 0.023s sys 0.014s
I20251025 07:52:31.587107 1171 ts_tablet_manager.cc:1929] T 1483c40eee834f20a92e6b4cd89775b6 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:31.587620 1171 log.cc:1199] T 1483c40eee834f20a92e6b4cd89775b6 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/1483c40eee834f20a92e6b4cd89775b6
W20251025 07:52:31.587767 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:43563, user_credentials={real_user=slave}} blocked reactor thread for 105855us
I20251025 07:52:31.588824 1166 raft_consensus.cc:359] T f3c87a83082b421cb4be546ac4ce7ad8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.589039 1166 raft_consensus.cc:385] T f3c87a83082b421cb4be546ac4ce7ad8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:31.589120 1166 raft_consensus.cc:740] T f3c87a83082b421cb4be546ac4ce7ad8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:31.589380 1166 consensus_queue.cc:260] T f3c87a83082b421cb4be546ac4ce7ad8 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.589893 1166 ts_tablet_manager.cc:1434] T f3c87a83082b421cb4be546ac4ce7ad8 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251025 07:52:31.589962 1171 ts_tablet_manager.cc:1950] T 1483c40eee834f20a92e6b4cd89775b6 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:31.590621 1166 tablet_bootstrap.cc:492] T 7e869c65771f4339a546dfeebb227c62 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:31.589589 1167 tablet_bootstrap.cc:492] T 742b3ec3f6604647b9e397532f1673dd P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:31.591045 1167 ts_tablet_manager.cc:1403] T 742b3ec3f6604647b9e397532f1673dd P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.001s
I20251025 07:52:31.592442 1166 tablet_bootstrap.cc:654] T 7e869c65771f4339a546dfeebb227c62 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:31.592453 1167 raft_consensus.cc:359] T 742b3ec3f6604647b9e397532f1673dd P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.592885 1167 raft_consensus.cc:385] T 742b3ec3f6604647b9e397532f1673dd P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:31.593009 1167 raft_consensus.cc:740] T 742b3ec3f6604647b9e397532f1673dd P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:31.593317 1167 consensus_queue.cc:260] T 742b3ec3f6604647b9e397532f1673dd P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.593808 1172 ts_tablet_manager.cc:1916] T 796413da731245459cc5dffacc3deb2f P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:31.599272 1167 ts_tablet_manager.cc:1434] T 742b3ec3f6604647b9e397532f1673dd P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251025 07:52:31.605742 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_2ddc7ef45af34002a5db135b33ec855e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:31.606343 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ddc7ef45af34002a5db135b33ec855e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:31.608130 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_fd1c5aa76b5c48caad4824c7a5d52347" } new_table_name: "default.Table_41c7a8a9dce040e790987150ecf0c1c1" modify_external_catalogs: true
I20251025 07:52:31.609115 1172 ts_tablet_manager.cc:1929] T 796413da731245459cc5dffacc3deb2f P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 07:52:31.609252 1172 log.cc:1199] T 796413da731245459cc5dffacc3deb2f P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/796413da731245459cc5dffacc3deb2f
I20251025 07:52:31.609678 1172 ts_tablet_manager.cc:1950] T 796413da731245459cc5dffacc3deb2f P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:31.609678 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_ad729c05dcae46f19df251a219aeccc4" } new_table_name: "default.Table_32b54e82cbc84d39a0020826f8737d62" modify_external_catalogs: true
I20251025 07:52:31.611392 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_c0432016895e4094b7ff582abe612bbd" } new_table_name: "default.Table_4791672658f043f6b97b9320a531ebb5" modify_external_catalogs: true
I20251025 07:52:31.615412 1167 tablet_bootstrap.cc:492] T 567a1f5c216a4a519249da66a2a03ea4 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:31.621042 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet f43ef51ee45d47a99ec732b49d50d3b4
I20251025 07:52:31.621961 890 tablet_service.cc:1552] Processing DeleteTablet for tablet f43ef51ee45d47a99ec732b49d50d3b4 with delete_type TABLET_DATA_DELETED (replaced by tablet e429b94326cc4827a03968fd92de38e0) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:31.622668 521 catalog_manager.cc:6719] ReplaceTablet: tablet f43ef51ee45d47a99ec732b49d50d3b4 deleted and replaced by tablet e429b94326cc4827a03968fd92de38e0
I20251025 07:52:31.622996 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 796413da731245459cc5dffacc3deb2f (table default.Table_293596c64cca4768bc5869fd9924f856 [id=d1b9d89b22bc4431a7d59ceeb9d75050]) successfully deleted
W20251025 07:52:31.623849 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 4f0c4f0e8f014e59be9e46d2afe0be9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f0c4f0e8f014e59be9e46d2afe0be9d already in progress: creating tablet
I20251025 07:52:31.623901 1167 tablet_bootstrap.cc:654] T 567a1f5c216a4a519249da66a2a03ea4 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:31.623466 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_da4ed35a8ed24427b6f4499fd1cc203c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:31.624377 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet c59f6e6eb9594a8e83f109a09b034045 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c59f6e6eb9594a8e83f109a09b034045 already in progress: creating tablet
W20251025 07:52:31.625030 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet d66ad049d5484228a268b0facb00ecf1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d66ad049d5484228a268b0facb00ecf1 already in progress: creating tablet
W20251025 07:52:31.625370 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet a4e4b99be0434baaa75d988281797846 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a4e4b99be0434baaa75d988281797846 already in progress: creating tablet
I20251025 07:52:31.625909 1171 tablet_replica.cc:333] T 47284f96928749e6a5201103a8ee5f9f P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
W20251025 07:52:31.626118 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet f0bdaa8cfd764f1b83f47881353d26b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0bdaa8cfd764f1b83f47881353d26b6 already in progress: creating tablet
I20251025 07:52:31.626190 1171 raft_consensus.cc:2243] T 47284f96928749e6a5201103a8ee5f9f P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:31.626375 1171 raft_consensus.cc:2272] T 47284f96928749e6a5201103a8ee5f9f P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:31.626623 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da4ed35a8ed24427b6f4499fd1cc203c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:31.627229 1171 ts_tablet_manager.cc:1916] T 47284f96928749e6a5201103a8ee5f9f P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:31.627723 1167 tablet_bootstrap.cc:492] T 567a1f5c216a4a519249da66a2a03ea4 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:31.630125 1167 ts_tablet_manager.cc:1403] T 567a1f5c216a4a519249da66a2a03ea4 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.002s
W20251025 07:52:31.631026 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet f43ef51ee45d47a99ec732b49d50d3b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f43ef51ee45d47a99ec732b49d50d3b4 already in progress: creating tablet
I20251025 07:52:31.631106 1167 raft_consensus.cc:359] T 567a1f5c216a4a519249da66a2a03ea4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.631335 1167 raft_consensus.cc:385] T 567a1f5c216a4a519249da66a2a03ea4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:31.631495 1167 raft_consensus.cc:740] T 567a1f5c216a4a519249da66a2a03ea4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:31.631855 1167 consensus_queue.cc:260] T 567a1f5c216a4a519249da66a2a03ea4 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:31.632874 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d0240d20f9904b4bbc8bf7f0f05e8850 has no consensus state
I20251025 07:52:31.633318 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_c8e45b61c6d4479a99fede45d02dbc1d" } new_table_name: "default.Table_06053a14dcc84e79917273c10b06b751" modify_external_catalogs: true
W20251025 07:52:31.634783 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2c50ec95e48e49b59a76fac4fc862d3d consensus state has no leader
I20251025 07:52:31.634822 1171 ts_tablet_manager.cc:1929] T 47284f96928749e6a5201103a8ee5f9f P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:31.635257 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_d57e7ea8e59641f781c564e5b01479bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:31.635972 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d57e7ea8e59641f781c564e5b01479bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:31.637208 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aa9d9ab8073945d28d52fb02213f201a consensus state has no leader
I20251025 07:52:31.637604 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_ea39b0e2e2204f38b1c9b1d3346a1e36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:31.638332 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea39b0e2e2204f38b1c9b1d3346a1e36 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:31.640413 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_309913c03a4d46019d35478363f149e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:31.640981 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_309913c03a4d46019d35478363f149e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:31.649492 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_c1bf6bb7c62c4648853b9a5659c73da0" } modify_external_catalogs: true
I20251025 07:52:31.638844 1167 ts_tablet_manager.cc:1434] T 567a1f5c216a4a519249da66a2a03ea4 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251025 07:52:31.650738 1167 tablet_bootstrap.cc:492] T 1aeaad5c5d01478bb997254ac372d9a6 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:31.653298 1167 tablet_bootstrap.cc:654] T 1aeaad5c5d01478bb997254ac372d9a6 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:31.654278 1171 log.cc:1199] T 47284f96928749e6a5201103a8ee5f9f P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/47284f96928749e6a5201103a8ee5f9f
I20251025 07:52:31.656256 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_064797535a2145bf93bd38b25ddaec83" } new_table_name: "default.Table_6597694b643c4ef2b46ca84afc10f4b0" modify_external_catalogs: true
I20251025 07:52:31.659624 1171 ts_tablet_manager.cc:1950] T 47284f96928749e6a5201103a8ee5f9f P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:31.661018 1171 tablet_replica.cc:333] T dd898450f2604987b08639cf87d0e402 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:31.661230 1171 raft_consensus.cc:2243] T dd898450f2604987b08639cf87d0e402 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:31.661585 1171 raft_consensus.cc:2272] T dd898450f2604987b08639cf87d0e402 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:31.662465 1171 ts_tablet_manager.cc:1916] T dd898450f2604987b08639cf87d0e402 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:31.656181 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_c1bf6bb7c62c4648853b9a5659c73da0" } modify_external_catalogs: true
W20251025 07:52:31.669373 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 062c0f3e51a3487880cea373f8cf1f54 has no consensus state
W20251025 07:52:31.672209 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4f7c15529be54a11961f385a953b0b03 has no consensus state
I20251025 07:52:31.676617 1171 ts_tablet_manager.cc:1929] T dd898450f2604987b08639cf87d0e402 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:31.680919 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_c4716673eec04f3baa6fc852c668fed8" } modify_external_catalogs: true
I20251025 07:52:31.681207 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_c4716673eec04f3baa6fc852c668fed8" } modify_external_catalogs: true
I20251025 07:52:31.689182 744 tablet_service.cc:1552] Processing DeleteTablet for tablet f0bdaa8cfd764f1b83f47881353d26b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:31.690232 1171 log.cc:1199] T dd898450f2604987b08639cf87d0e402 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/dd898450f2604987b08639cf87d0e402
I20251025 07:52:31.691005 1171 ts_tablet_manager.cc:1950] T dd898450f2604987b08639cf87d0e402 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:31.695356 1171 tablet_replica.cc:333] T 85e27fc52e9a4d62a8622645594f59c6 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:31.699111 1218 raft_consensus.cc:493] T 742b3ec3f6604647b9e397532f1673dd P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:31.699288 1218 raft_consensus.cc:515] T 742b3ec3f6604647b9e397532f1673dd P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.699478 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 1483c40eee834f20a92e6b4cd89775b6 (table default.Table_e0bf55e24b084fdbb62f09cce80b50c5 [id=084833e021c24b4a84dd1b1a3bb08dfc]) successfully deleted
W20251025 07:52:31.699781 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet d66ad049d5484228a268b0facb00ecf1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d66ad049d5484228a268b0facb00ecf1 already in progress: creating tablet
I20251025 07:52:31.699960 1218 leader_election.cc:290] T 742b3ec3f6604647b9e397532f1673dd P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:31.698990 744 tablet_service.cc:1552] Processing DeleteTablet for tablet f43ef51ee45d47a99ec732b49d50d3b4 with delete_type TABLET_DATA_DELETED (replaced by tablet e429b94326cc4827a03968fd92de38e0) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:31.700894 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "742b3ec3f6604647b9e397532f1673dd" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:31.702025 1171 raft_consensus.cc:2243] T 85e27fc52e9a4d62a8622645594f59c6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:31.702242 1171 raft_consensus.cc:2272] T 85e27fc52e9a4d62a8622645594f59c6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:31.702188 775 raft_consensus.cc:2468] T 742b3ec3f6604647b9e397532f1673dd P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:31.702664 847 leader_election.cc:304] T 742b3ec3f6604647b9e397532f1673dd P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:31.702993 1218 raft_consensus.cc:2804] T 742b3ec3f6604647b9e397532f1673dd P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:31.703094 1218 raft_consensus.cc:493] T 742b3ec3f6604647b9e397532f1673dd P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:31.703164 1218 raft_consensus.cc:3060] T 742b3ec3f6604647b9e397532f1673dd P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:31.699998 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 47284f96928749e6a5201103a8ee5f9f (table default.Table_373f40b4ac044088a071a6d26d7a6da0 [id=05672e3da0704022b93cc96d61278433]) successfully deleted
W20251025 07:52:31.703418 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet a4e4b99be0434baaa75d988281797846 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a4e4b99be0434baaa75d988281797846 already in progress: creating tablet
I20251025 07:52:31.703605 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet dd898450f2604987b08639cf87d0e402 (table default.Table_56f5f8c6d70645309e7ea14dcf847a43 [id=2dd014b2a3a94681967660e0399d334a]) successfully deleted
I20251025 07:52:31.704383 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 1912e59548874e0e8e989050ec0064c5
I20251025 07:52:31.700390 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 0f5f6ca6c3cd4d369690e425387a4344
I20251025 07:52:31.705675 522 catalog_manager.cc:6719] ReplaceTablet: tablet 0f5f6ca6c3cd4d369690e425387a4344 deleted and replaced by tablet 22fc90c704f64d2f9f4b8c54dee45576
I20251025 07:52:31.705682 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 1912e59548874e0e8e989050ec0064c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:31.702536 523 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 8e6d76b1a97945cfb6c3055e1b622f8a (table default.Table_10e57b42f97544208675708b1e028846 [id=f5c1be203e2b4831a025871cf8a2ef17]). Expected version 2 got 0
I20251025 07:52:31.706048 1218 raft_consensus.cc:515] T 742b3ec3f6604647b9e397532f1673dd P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.706248 523 catalog_manager.cc:5649] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } }
I20251025 07:52:31.706676 523 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 627a10a0714f4be8bdbced503fd6388d (table default.Table_6b937442c9614efeb8cc1562ac306174 [id=7fc07b0c33f74604be526c4a0df949e9]). Expected version 1 got 0
W20251025 07:52:31.706975 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 1912e59548874e0e8e989050ec0064c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1912e59548874e0e8e989050ec0064c5 already in progress: creating tablet
I20251025 07:52:31.707304 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 0f5f6ca6c3cd4d369690e425387a4344 with delete_type TABLET_DATA_DELETED (replaced by tablet 22fc90c704f64d2f9f4b8c54dee45576) from {username='slave'} at 127.0.0.1:56590
W20251025 07:52:31.707846 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 0f5f6ca6c3cd4d369690e425387a4344 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f5f6ca6c3cd4d369690e425387a4344 already in progress: creating tablet
I20251025 07:52:31.710381 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 34315f64b89148efbf1f20a43d5c5efb
I20251025 07:52:31.711939 1218 leader_election.cc:290] T 742b3ec3f6604647b9e397532f1673dd P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:31.712466 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 34315f64b89148efbf1f20a43d5c5efb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:31.713572 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "742b3ec3f6604647b9e397532f1673dd" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:31.713781 775 raft_consensus.cc:3060] T 742b3ec3f6604647b9e397532f1673dd P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:31.714679 1171 ts_tablet_manager.cc:1916] T 85e27fc52e9a4d62a8622645594f59c6 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:31.716042 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 0f5f6ca6c3cd4d369690e425387a4344 with delete_type TABLET_DATA_DELETED (replaced by tablet 22fc90c704f64d2f9f4b8c54dee45576) from {username='slave'} at 127.0.0.1:48830
W20251025 07:52:31.717895 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 34315f64b89148efbf1f20a43d5c5efb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34315f64b89148efbf1f20a43d5c5efb already in progress: creating tablet
I20251025 07:52:31.718070 878 tablet_service.cc:1505] Processing CreateTablet for tablet dc4a788dca9d4a8ba40f8fdf1a4b915e (DEFAULT_TABLE table=default.Table_d5ce3dfc91174e268c17c2d5b3f93b86 [id=607c24e5c581467db1eac04c5165450d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.718374 890 tablet_service.cc:1505] Processing CreateTablet for tablet 437da5ded9b648abb668a8cb4896452c (DEFAULT_TABLE table=default.Table_0ebd49dea97c4b0790902d42d65e749f [id=68d232ecc08c4f8db462fc0ad620e9be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.718531 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc4a788dca9d4a8ba40f8fdf1a4b915e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.718765 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 437da5ded9b648abb668a8cb4896452c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.721128 882 tablet_service.cc:1505] Processing CreateTablet for tablet 5dab439bc50d4e869548154e1d222ad6 (DEFAULT_TABLE table=default.Table_262a9d20d20840aa85bb3dc06aa5dde6 [id=45325c188eac4d34bb016d01de091f87]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.721494 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5dab439bc50d4e869548154e1d222ad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.724038 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_3cd469e1b1414bc7ad4a122483cb7755"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:31.724623 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3cd469e1b1414bc7ad4a122483cb7755 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:31.726001 524 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 627a10a0714f4be8bdbced503fd6388d (table default.Table_6b937442c9614efeb8cc1562ac306174 [id=7fc07b0c33f74604be526c4a0df949e9]). Expected version 1 got 0
I20251025 07:52:31.726799 891 tablet_service.cc:1505] Processing CreateTablet for tablet 4f7c15529be54a11961f385a953b0b03 (DEFAULT_TABLE table=default.Table_06053a14dcc84e79917273c10b06b751 [id=c42ad6bf27aa4744b66b4462ac97e57d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.727145 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f7c15529be54a11961f385a953b0b03. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.737236 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet dda0b26e4f1a44928bc4b84f333901c0
I20251025 07:52:31.738488 889 tablet_service.cc:1505] Processing CreateTablet for tablet 72618756f756484e895e885ef12545c8 (DEFAULT_TABLE table=default.Table_dbe39538a228480a9ff9cd45159e39f7 [id=2965a45ad79e4523ab42d65823f19c8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.738927 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72618756f756484e895e885ef12545c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.739696 523 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: default.Table_dfc7f1d4bf8b48bebbb00313419c3fd8 [id=d82b3e1317434cf08a0d51e8a9427224] alter complete (version 2)
I20251025 07:52:31.741745 886 tablet_service.cc:1552] Processing DeleteTablet for tablet dda0b26e4f1a44928bc4b84f333901c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:31.742210 894 tablet_service.cc:1505] Processing CreateTablet for tablet d46673a283fe42b78ddeb99fbd2678eb (DEFAULT_TABLE table=default.Table_241aaedabd0245dc85e8692b83d51965 [id=bf77b5ff6f1048dd996d948437f9ff5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.742552 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d46673a283fe42b78ddeb99fbd2678eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.742648 1441 raft_consensus.cc:493] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:31.742822 1441 raft_consensus.cc:515] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.743594 1441 leader_election.cc:290] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:31.744902 710 leader_election.cc:343] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 90764275825c4ffb8712cb883d37a551
I20251025 07:52:31.748553 881 tablet_service.cc:1505] Processing CreateTablet for tablet e47c4f741f354a19b49777bd06e54764 (DEFAULT_TABLE table=default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.748931 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e47c4f741f354a19b49777bd06e54764. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.750521 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 95efd553493e4966bc48a01cac6e1e10
I20251025 07:52:31.751750 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 79bc2678df8042e3994b0471afa2c304 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:31.751945 876 tablet_service.cc:1552] Processing DeleteTablet for tablet 47284f96928749e6a5201103a8ee5f9f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:31.752101 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 85fba8b37f2c4540aa81701052bf222a with delete_type TABLET_DATA_DELETED (replaced by tablet 6a58931e7cd7454fb583f1c70cd2ad15) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:31.754395 893 tablet_service.cc:1505] Processing CreateTablet for tablet 95efd553493e4966bc48a01cac6e1e10 (DEFAULT_TABLE table=default.Table_c4716673eec04f3baa6fc852c668fed8 [id=1d09f9e02b4a4a4d8d38ee522fea841b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.754813 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95efd553493e4966bc48a01cac6e1e10. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.755462 525 master_service.cc:954] ReplaceTablet: received request to replace tablet f3001314b56345f5a70314777cae0125 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:31.759443 1218 raft_consensus.cc:493] T 8e6d76b1a97945cfb6c3055e1b622f8a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:31.759578 1218 raft_consensus.cc:515] T 8e6d76b1a97945cfb6c3055e1b622f8a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.760548 1218 leader_election.cc:290] T 8e6d76b1a97945cfb6c3055e1b622f8a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:31.761866 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 95efd553493e4966bc48a01cac6e1e10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:31.763336 1167 tablet_bootstrap.cc:492] T 1aeaad5c5d01478bb997254ac372d9a6 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:31.763518 1167 ts_tablet_manager.cc:1403] T 1aeaad5c5d01478bb997254ac372d9a6 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.113s user 0.013s sys 0.007s
I20251025 07:52:31.716199 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 34315f64b89148efbf1f20a43d5c5efb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:31.764322 1167 raft_consensus.cc:359] T 1aeaad5c5d01478bb997254ac372d9a6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.764657 1167 raft_consensus.cc:385] T 1aeaad5c5d01478bb997254ac372d9a6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:31.764663 886 tablet_service.cc:1552] Processing DeleteTablet for tablet 742b3ec3f6604647b9e397532f1673dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:31.764737 1167 raft_consensus.cc:740] T 1aeaad5c5d01478bb997254ac372d9a6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:31.716562 742 tablet_service.cc:1505] Processing CreateTablet for tablet 437da5ded9b648abb668a8cb4896452c (DEFAULT_TABLE table=default.Table_0ebd49dea97c4b0790902d42d65e749f [id=68d232ecc08c4f8db462fc0ad620e9be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.765161 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 437da5ded9b648abb668a8cb4896452c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.765043 1167 consensus_queue.cc:260] T 1aeaad5c5d01478bb997254ac372d9a6 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.765651 1167 ts_tablet_manager.cc:1434] T 1aeaad5c5d01478bb997254ac372d9a6 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:31.766168 1167 tablet_bootstrap.cc:492] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:31.716931 750 tablet_service.cc:1505] Processing CreateTablet for tablet dc4a788dca9d4a8ba40f8fdf1a4b915e (DEFAULT_TABLE table=default.Table_d5ce3dfc91174e268c17c2d5b3f93b86 [id=607c24e5c581467db1eac04c5165450d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.717108 744 tablet_service.cc:1552] Processing DeleteTablet for tablet 1912e59548874e0e8e989050ec0064c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:31.768405 1569 raft_consensus.cc:493] T 742b3ec3f6604647b9e397532f1673dd P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:31.768754 1169 tablet_bootstrap.cc:492] T 8c6ff4c73e624af3bf2de36c4493b462 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:31.769629 1172 tablet_replica.cc:333] T 79bc2678df8042e3994b0471afa2c304 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:31.769856 1172 raft_consensus.cc:2243] T 79bc2678df8042e3994b0471afa2c304 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:31.770108 1172 pending_rounds.cc:70] T 79bc2678df8042e3994b0471afa2c304 P 3aea2869ff814cc7b142012faa274382: Trying to abort 1 pending ops.
I20251025 07:52:31.770236 1172 pending_rounds.cc:77] T 79bc2678df8042e3994b0471afa2c304 P 3aea2869ff814cc7b142012faa274382: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214607364905873408 op_type: NO_OP noop_request { }
I20251025 07:52:31.770335 1172 raft_consensus.cc:2889] T 79bc2678df8042e3994b0471afa2c304 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 07:52:31.770452 1172 raft_consensus.cc:2272] T 79bc2678df8042e3994b0471afa2c304 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:31.771271 511 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet dda0b26e4f1a44928bc4b84f333901c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dda0b26e4f1a44928bc4b84f333901c0 already in progress: creating tablet
I20251025 07:52:31.774521 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "742b3ec3f6604647b9e397532f1673dd" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:31.716858 775 raft_consensus.cc:2468] T 742b3ec3f6604647b9e397532f1673dd P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:31.782485 1172 ts_tablet_manager.cc:1916] T 79bc2678df8042e3994b0471afa2c304 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:31.790009 1166 tablet_bootstrap.cc:492] T 7e869c65771f4339a546dfeebb227c62 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:31.791590 1166 ts_tablet_manager.cc:1403] T 7e869c65771f4339a546dfeebb227c62 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.201s user 0.020s sys 0.011s
I20251025 07:52:31.794384 1166 raft_consensus.cc:359] T 7e869c65771f4339a546dfeebb227c62 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.796241 1166 raft_consensus.cc:385] T 7e869c65771f4339a546dfeebb227c62 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:31.796409 1166 raft_consensus.cc:740] T 7e869c65771f4339a546dfeebb227c62 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:31.796775 1166 consensus_queue.cc:260] T 7e869c65771f4339a546dfeebb227c62 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.797443 1166 ts_tablet_manager.cc:1434] T 7e869c65771f4339a546dfeebb227c62 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251025 07:52:31.799151 1166 tablet_bootstrap.cc:492] T 2784bbfb97d5466f9ec856add7d0b29a P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:31.802634 1166 tablet_bootstrap.cc:654] T 2784bbfb97d5466f9ec856add7d0b29a P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:31.826182 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_d57e7ea8e59641f781c564e5b01479bc" } modify_external_catalogs: true
I20251025 07:52:31.826295 525 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 627a10a0714f4be8bdbced503fd6388d (table default.Table_6b937442c9614efeb8cc1562ac306174 [id=7fc07b0c33f74604be526c4a0df949e9]). Expected version 1 got 0
I20251025 07:52:31.826463 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_d57e7ea8e59641f781c564e5b01479bc" } modify_external_catalogs: true
I20251025 07:52:31.830683 523 master_service.cc:954] ReplaceTablet: received request to replace tablet f4578ce69a0d4322944e5ce7c8a59831 from {username='slave'} at 127.0.0.1:45978
I20251025 07:52:31.834010 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e6d76b1a97945cfb6c3055e1b622f8a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:31.834455 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc4a788dca9d4a8ba40f8fdf1a4b915e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.768607 1569 raft_consensus.cc:515] T 742b3ec3f6604647b9e397532f1673dd P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.838304 1569 leader_election.cc:290] T 742b3ec3f6604647b9e397532f1673dd P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:31.838526 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 3b940fa5ae26427dafbcfd26372754b3
I20251025 07:52:31.841198 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet f43ef51ee45d47a99ec732b49d50d3b4 with delete_type TABLET_DATA_DELETED (replaced by tablet e429b94326cc4827a03968fd92de38e0) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:31.834213 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e6d76b1a97945cfb6c3055e1b622f8a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:31.841948 1048 raft_consensus.cc:2468] T 742b3ec3f6604647b9e397532f1673dd P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:31.842180 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "742b3ec3f6604647b9e397532f1673dd" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:31.842227 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 85fba8b37f2c4540aa81701052bf222a with delete_type TABLET_DATA_DELETED (replaced by tablet 6a58931e7cd7454fb583f1c70cd2ad15) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:31.845131 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90764275825c4ffb8712cb883d37a551" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:31.845623 1046 raft_consensus.cc:2468] T 90764275825c4ffb8712cb883d37a551 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:31.849675 1170 ts_tablet_manager.cc:1929] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20251025 07:52:31.850709 1170 log.cc:1199] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/50ffbcdd4a9247bf99b8e2361bc02ef5
I20251025 07:52:31.851245 1170 ts_tablet_manager.cc:1950] T 50ffbcdd4a9247bf99b8e2361bc02ef5 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:31.851285 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet f4578ce69a0d4322944e5ce7c8a59831
I20251025 07:52:31.851679 523 catalog_manager.cc:6719] ReplaceTablet: tablet f4578ce69a0d4322944e5ce7c8a59831 deleted and replaced by tablet 48b73e0c0dce4be686d05f991a21bd19
I20251025 07:52:31.852157 1169 tablet_bootstrap.cc:654] T 8c6ff4c73e624af3bf2de36c4493b462 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:31.853228 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 1912e59548874e0e8e989050ec0064c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:31.854947 1025 tablet_service.cc:1505] Processing CreateTablet for tablet e47c4f741f354a19b49777bd06e54764 (DEFAULT_TABLE table=default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.855371 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e47c4f741f354a19b49777bd06e54764. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.855554 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 34315f64b89148efbf1f20a43d5c5efb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:31.855975 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 437da5ded9b648abb668a8cb4896452c (DEFAULT_TABLE table=default.Table_0ebd49dea97c4b0790902d42d65e749f [id=68d232ecc08c4f8db462fc0ad620e9be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.856360 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 437da5ded9b648abb668a8cb4896452c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.858266 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 95efd553493e4966bc48a01cac6e1e10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:31.858410 1027 tablet_service.cc:1505] Processing CreateTablet for tablet dc4a788dca9d4a8ba40f8fdf1a4b915e (DEFAULT_TABLE table=default.Table_d5ce3dfc91174e268c17c2d5b3f93b86 [id=607c24e5c581467db1eac04c5165450d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.858739 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc4a788dca9d4a8ba40f8fdf1a4b915e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.861709 1016 tablet_service.cc:1505] Processing CreateTablet for tablet 67cf96a7dd3f42e19b86746c87e689f6 (DEFAULT_TABLE table=default.Table_ea39b0e2e2204f38b1c9b1d3346a1e36 [id=9170ed658b754ac8ad50f060c442f27c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.862121 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67cf96a7dd3f42e19b86746c87e689f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.862342 1019 tablet_service.cc:1505] Processing CreateTablet for tablet 1c40b2e06d894a06af9e58285892ea8d (DEFAULT_TABLE table=default.Table_3cd469e1b1414bc7ad4a122483cb7755 [id=7b4db2984274499c98f6189d753f7856]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.862789 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet dda0b26e4f1a44928bc4b84f333901c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:31.863209 1017 tablet_service.cc:1505] Processing CreateTablet for tablet 95efd553493e4966bc48a01cac6e1e10 (DEFAULT_TABLE table=default.Table_c4716673eec04f3baa6fc852c668fed8 [id=1d09f9e02b4a4a4d8d38ee522fea841b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.864101 1018 tablet_service.cc:1505] Processing CreateTablet for tablet d46673a283fe42b78ddeb99fbd2678eb (DEFAULT_TABLE table=default.Table_241aaedabd0245dc85e8692b83d51965 [id=bf77b5ff6f1048dd996d948437f9ff5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.864655 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 72618756f756484e895e885ef12545c8 (DEFAULT_TABLE table=default.Table_dbe39538a228480a9ff9cd45159e39f7 [id=2965a45ad79e4523ab42d65823f19c8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.865135 1023 tablet_service.cc:1505] Processing CreateTablet for tablet 5dab439bc50d4e869548154e1d222ad6 (DEFAULT_TABLE table=default.Table_262a9d20d20840aa85bb3dc06aa5dde6 [id=45325c188eac4d34bb016d01de091f87]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.865643 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 4f7c15529be54a11961f385a953b0b03 (DEFAULT_TABLE table=default.Table_06053a14dcc84e79917273c10b06b751 [id=c42ad6bf27aa4744b66b4462ac97e57d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.866348 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c40b2e06d894a06af9e58285892ea8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.870850 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5dab439bc50d4e869548154e1d222ad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.874348 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72618756f756484e895e885ef12545c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.878266 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95efd553493e4966bc48a01cac6e1e10. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.882813 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f7c15529be54a11961f385a953b0b03. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.886386 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 0f5f6ca6c3cd4d369690e425387a4344 with delete_type TABLET_DATA_DELETED (replaced by tablet 22fc90c704f64d2f9f4b8c54dee45576) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:31.887100 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d46673a283fe42b78ddeb99fbd2678eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.890705 1169 tablet_bootstrap.cc:492] T 8c6ff4c73e624af3bf2de36c4493b462 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:31.890880 1169 ts_tablet_manager.cc:1403] T 8c6ff4c73e624af3bf2de36c4493b462 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.305s user 0.002s sys 0.002s
I20251025 07:52:31.891718 1169 raft_consensus.cc:359] T 8c6ff4c73e624af3bf2de36c4493b462 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.891929 1169 raft_consensus.cc:385] T 8c6ff4c73e624af3bf2de36c4493b462 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:31.892005 1169 raft_consensus.cc:740] T 8c6ff4c73e624af3bf2de36c4493b462 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:31.892313 1016 tablet_service.cc:1505] Processing CreateTablet for tablet 243b8828fc344bacab46d7bab6bd2dec (DEFAULT_TABLE table=default.Table_d647b0fa975743998d7f49c36abe9c96 [id=5b435831d8254ae8b126218a39e3a362]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.892508 1020 tablet_service.cc:1505] Processing CreateTablet for tablet ed243540f2d746b980c25687fbca7414 (DEFAULT_TABLE table=default.Table_208daf030a0d4efdb6ef8250b2f48896 [id=7507e4bd03ca4dd2a93655b138e8d49c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.892518 1169 consensus_queue.cc:260] T 8c6ff4c73e624af3bf2de36c4493b462 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.892700 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 243b8828fc344bacab46d7bab6bd2dec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.892920 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed243540f2d746b980c25687fbca7414. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.893116 1169 ts_tablet_manager.cc:1434] T 8c6ff4c73e624af3bf2de36c4493b462 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:31.893553 1169 tablet_bootstrap.cc:492] T 79bc2678df8042e3994b0471afa2c304 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:31.895017 1021 tablet_service.cc:1505] Processing CreateTablet for tablet 075f0a151e024220b51ba7fcaecdcc3d (DEFAULT_TABLE table=default.Table_26b3fa566ddf4e5aadd68de388cb61ba [id=b064a22cfdc04a0793ffd858b3fdc45e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.895390 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 075f0a151e024220b51ba7fcaecdcc3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.895932 1169 tablet_bootstrap.cc:654] T 79bc2678df8042e3994b0471afa2c304 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:31.901773 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 288e543bb279410c91c01dd1623e110f (DEFAULT_TABLE table=default.Table_ea269532aff34ca1803fc1c5d5ad3641 [id=4c34b64b299f4162a53866cf01c5298f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.902098 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 288e543bb279410c91c01dd1623e110f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.903965 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_3d4495f2005e498a928c68b14b8017a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:31.904839 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d4495f2005e498a928c68b14b8017a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:31.906962 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 90764275825c4ffb8712cb883d37a551 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:31.908397 1170 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47408 (request call id 385) took 1001 ms. Trace:
I20251025 07:52:31.908634 1170 rpcz_store.cc:276] 1025 07:52:30.900233 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:30.982371 (+ 82138us) service_pool.cc:225] Handling call
1025 07:52:31.540059 (+557688us) ts_tablet_manager.cc:1151] Deleting tablet 50ffbcdd4a9247bf99b8e2361bc02ef5
1025 07:52:31.540069 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:31.582724 (+ 42655us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:31.849484 (+266760us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:31.901592 (+ 52108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":493893}
I20251025 07:52:31.910228 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 792f32cbd66449ca98813c669b2a261d (DEFAULT_TABLE table=default.Table_23cacc46bec647adbc62b457f73487cf [id=a8b081495f564c10aa6a05e5eed80488]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.918498 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b940fa5ae26427dafbcfd26372754b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:31.923127 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_c26454d3c928434ab0dae167abcc2ff2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:31.923707 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c26454d3c928434ab0dae167abcc2ff2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:31.925599 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_20b987167f0e4863824e7bb7c17e5c03" } new_table_name: "default.Table_cd9190d6cb714bdbb763dd934b33c10a" modify_external_catalogs: true
I20251025 07:52:31.926745 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_974f1ee455d7488593e5c496cc012099"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:31.927142 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_2070d67c804b4eceb6d29bfb97a42604" } new_table_name: "default.Table_b936c4aedc5340daa5e60cff2e2ba46e" modify_external_catalogs: true
W20251025 07:52:31.927304 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_974f1ee455d7488593e5c496cc012099 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:31.928373 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_e386bc02f89041418ff3ddf8ac224eea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:31.928735 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_3d163f45f94f450ebf0447897bdf41a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:31.928872 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e386bc02f89041418ff3ddf8ac224eea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:31.929178 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d163f45f94f450ebf0447897bdf41a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:31.929939 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_241aaedabd0245dc85e8692b83d51965" } new_table_name: "default.Table_3822b091ab6541efb91252e18b0a2a8b" modify_external_catalogs: true
I20251025 07:52:31.930994 1015 tablet_service.cc:1505] Processing CreateTablet for tablet 3b940fa5ae26427dafbcfd26372754b3 (DEFAULT_TABLE table=default.Table_d57e7ea8e59641f781c564e5b01479bc [id=fbaf826daa7c448eb92c7f93826260c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.931470 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_fd5720d69b084df1b36485b7890ac6c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:31.931891 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd5720d69b084df1b36485b7890ac6c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:31.931964 751 tablet_service.cc:1552] Processing DeleteTablet for tablet dda0b26e4f1a44928bc4b84f333901c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:31.912632 1014 tablet_service.cc:1505] Processing CreateTablet for tablet 2d24e3e7a9e4431db609331ad1fd57c9 (DEFAULT_TABLE table=default.Table_2ddc7ef45af34002a5db135b33ec855e [id=7e34aeb0c3ea4fc5ba6a9077a79335d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.932971 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_6aaf0d6ed7554bd4afa602cdca717ce2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:31.933367 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6aaf0d6ed7554bd4afa602cdca717ce2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:31.933810 890 tablet_service.cc:1505] Processing CreateTablet for tablet 67cf96a7dd3f42e19b86746c87e689f6 (DEFAULT_TABLE table=default.Table_ea39b0e2e2204f38b1c9b1d3346a1e36 [id=9170ed658b754ac8ad50f060c442f27c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.935581 1218 raft_consensus.cc:493] T fd9b4be139d347f4a6a60767029c342a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:31.934592 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 792f32cbd66449ca98813c669b2a261d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:31.935777 850 leader_election.cc:343] T 8e6d76b1a97945cfb6c3055e1b622f8a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:31.935925 1561 raft_consensus.cc:493] T 08da9d2687704725aacd11cd92c6a9f9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:31.934958 1027 tablet_service.cc:1505] Processing CreateTablet for tablet a6ae1315b28940d49b667e3c80347021 (DEFAULT_TABLE table=default.Table_ac3211ef7b0941d2ad4db2fab62231b2 [id=f721af9327cd4e00a20fc1bfebd9e17b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.936729 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67cf96a7dd3f42e19b86746c87e689f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.937140 1218 raft_consensus.cc:515] T fd9b4be139d347f4a6a60767029c342a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.937353 1013 tablet_service.cc:1505] Processing CreateTablet for tablet 6ce62847e80f48e8807fdbbbdc50803c (DEFAULT_TABLE table=default.Table_dad8e9c26a9a41e2bcf3fa1db9f5b087 [id=579250c3c8fb4e9783e388a53ba35a6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.937812 1218 leader_election.cc:290] T fd9b4be139d347f4a6a60767029c342a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
W20251025 07:52:31.937976 850 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:45417, user_credentials={real_user=slave}} blocked reactor thread for 90743.4us
I20251025 07:52:31.938969 1561 raft_consensus.cc:515] T 08da9d2687704725aacd11cd92c6a9f9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.939543 1561 leader_election.cc:290] T 08da9d2687704725aacd11cd92c6a9f9 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:31.939582 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6ae1315b28940d49b667e3c80347021. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.940130 878 tablet_service.cc:1505] Processing CreateTablet for tablet 3b940fa5ae26427dafbcfd26372754b3 (DEFAULT_TABLE table=default.Table_d57e7ea8e59641f781c564e5b01479bc [id=fbaf826daa7c448eb92c7f93826260c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.940979 1012 tablet_service.cc:1505] Processing CreateTablet for tablet e4da89f863cc43f3bf6df908bb9dc6bf (DEFAULT_TABLE table=default.Table_3d995c9c75174000b6cb98779cc0bb2f [id=6c1ffab0043545c2b7cce93a641eaa1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.941788 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b940fa5ae26427dafbcfd26372754b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.942178 889 tablet_service.cc:1552] Processing DeleteTablet for tablet ac0ea298bc114c56a8093e12def21ce1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:31.936424 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 5cba9c05dd424cc788febc614a34819a with delete_type TABLET_DATA_DELETED (replaced by tablet 4b28215f61574cf48c447dd01955ac77) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:31.942744 1170 tablet_replica.cc:333] T 742b3ec3f6604647b9e397532f1673dd P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:31.942826 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd9b4be139d347f4a6a60767029c342a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:31.943132 1170 raft_consensus.cc:2243] T 742b3ec3f6604647b9e397532f1673dd P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:31.942767 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "08da9d2687704725aacd11cd92c6a9f9" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:31.943487 1170 raft_consensus.cc:2272] T 742b3ec3f6604647b9e397532f1673dd P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:31.943632 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 8917a289bd9b4428a4603429edd94581 (DEFAULT_TABLE table=default.Table_45faec266a9b4b1b9f5ef3ac915e3bbc [id=bc377ac4bb164a80b9ebce289a790326]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.944664 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 21e52928580c4762b0ae7152490ef955 with delete_type TABLET_DATA_DELETED (replaced by tablet 9d6a9937d10a4cc2813671dd1bfa1664) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:31.944849 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b940fa5ae26427dafbcfd26372754b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.945520 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 08da9d2687704725aacd11cd92c6a9f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:31.945696 881 tablet_service.cc:1505] Processing CreateTablet for tablet 1c40b2e06d894a06af9e58285892ea8d (DEFAULT_TABLE table=default.Table_3cd469e1b1414bc7ad4a122483cb7755 [id=7b4db2984274499c98f6189d753f7856]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.944705 1010 tablet_service.cc:1505] Processing CreateTablet for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (DEFAULT_TABLE table=default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.947058 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c40b2e06d894a06af9e58285892ea8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.950505 893 tablet_service.cc:1552] Processing DeleteTablet for tablet f3c87a83082b421cb4be546ac4ce7ad8 with delete_type TABLET_DATA_DELETED (replaced by tablet b17d2a9ad6be49d29f2413666fba7e69) from {username='slave'} at 127.0.0.1:56590
W20251025 07:52:31.951177 850 leader_election.cc:343] T fd9b4be139d347f4a6a60767029c342a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:31.952037 850 leader_election.cc:343] T 08da9d2687704725aacd11cd92c6a9f9 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:31.953886 893 tablet_service.cc:1505] Processing CreateTablet for tablet 2d24e3e7a9e4431db609331ad1fd57c9 (DEFAULT_TABLE table=default.Table_2ddc7ef45af34002a5db135b33ec855e [id=7e34aeb0c3ea4fc5ba6a9077a79335d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.954481 1025 tablet_service.cc:1505] Processing CreateTablet for tablet df9db9dd31d840e0b1394db04d662a62 (DEFAULT_TABLE table=default.Table_da4ed35a8ed24427b6f4499fd1cc203c [id=cdbb76e674fc41cfb0f86d678a3f4f0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.954964 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4da89f863cc43f3bf6df908bb9dc6bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.955060 891 tablet_service.cc:1505] Processing CreateTablet for tablet 6ce62847e80f48e8807fdbbbdc50803c (DEFAULT_TABLE table=default.Table_dad8e9c26a9a41e2bcf3fa1db9f5b087 [id=579250c3c8fb4e9783e388a53ba35a6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.956202 1011 tablet_service.cc:1505] Processing CreateTablet for tablet 062c0f3e51a3487880cea373f8cf1f54 (DEFAULT_TABLE table=default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.956277 894 tablet_service.cc:1505] Processing CreateTablet for tablet 075f0a151e024220b51ba7fcaecdcc3d (DEFAULT_TABLE table=default.Table_26b3fa566ddf4e5aadd68de388cb61ba [id=b064a22cfdc04a0793ffd858b3fdc45e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.956847 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d24e3e7a9e4431db609331ad1fd57c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.960404 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ce62847e80f48e8807fdbbbdc50803c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.961913 889 tablet_service.cc:1505] Processing CreateTablet for tablet 243b8828fc344bacab46d7bab6bd2dec (DEFAULT_TABLE table=default.Table_d647b0fa975743998d7f49c36abe9c96 [id=5b435831d8254ae8b126218a39e3a362]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.962220 886 tablet_service.cc:1505] Processing CreateTablet for tablet 288e543bb279410c91c01dd1623e110f (DEFAULT_TABLE table=default.Table_ea269532aff34ca1803fc1c5d5ad3641 [id=4c34b64b299f4162a53866cf01c5298f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.962437 882 tablet_service.cc:1505] Processing CreateTablet for tablet ed243540f2d746b980c25687fbca7414 (DEFAULT_TABLE table=default.Table_208daf030a0d4efdb6ef8250b2f48896 [id=7507e4bd03ca4dd2a93655b138e8d49c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.962729 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 075f0a151e024220b51ba7fcaecdcc3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.963280 1016 tablet_service.cc:1505] Processing CreateTablet for tablet 4b278ce7768349e49dcd1131e404ab34 (DEFAULT_TABLE table=default.Table_309913c03a4d46019d35478363f149e8 [id=bad99c3045fc4b938e20d2dcfdd0b1bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.964186 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 062c0f3e51a3487880cea373f8cf1f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.964951 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d24e3e7a9e4431db609331ad1fd57c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.965526 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0240d20f9904b4bbc8bf7f0f05e8850. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.966233 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed243540f2d746b980c25687fbca7414. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.967583 746 tablet_service.cc:1505] Processing CreateTablet for tablet 5dab439bc50d4e869548154e1d222ad6 (DEFAULT_TABLE table=default.Table_262a9d20d20840aa85bb3dc06aa5dde6 [id=45325c188eac4d34bb016d01de091f87]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.969424 737 tablet_service.cc:1505] Processing CreateTablet for tablet d46673a283fe42b78ddeb99fbd2678eb (DEFAULT_TABLE table=default.Table_241aaedabd0245dc85e8692b83d51965 [id=bf77b5ff6f1048dd996d948437f9ff5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.969843 756 tablet_service.cc:1505] Processing CreateTablet for tablet e47c4f741f354a19b49777bd06e54764 (DEFAULT_TABLE table=default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.970183 753 tablet_service.cc:1505] Processing CreateTablet for tablet 95efd553493e4966bc48a01cac6e1e10 (DEFAULT_TABLE table=default.Table_c4716673eec04f3baa6fc852c668fed8 [id=1d09f9e02b4a4a4d8d38ee522fea841b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.970500 744 tablet_service.cc:1505] Processing CreateTablet for tablet 4f7c15529be54a11961f385a953b0b03 (DEFAULT_TABLE table=default.Table_06053a14dcc84e79917273c10b06b751 [id=c42ad6bf27aa4744b66b4462ac97e57d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.970819 751 tablet_service.cc:1505] Processing CreateTablet for tablet 67cf96a7dd3f42e19b86746c87e689f6 (DEFAULT_TABLE table=default.Table_ea39b0e2e2204f38b1c9b1d3346a1e36 [id=9170ed658b754ac8ad50f060c442f27c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.971130 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5dab439bc50d4e869548154e1d222ad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.971686 739 tablet_service.cc:1505] Processing CreateTablet for tablet 1c40b2e06d894a06af9e58285892ea8d (DEFAULT_TABLE table=default.Table_3cd469e1b1414bc7ad4a122483cb7755 [id=7b4db2984274499c98f6189d753f7856]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.971982 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c40b2e06d894a06af9e58285892ea8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.972517 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f7c15529be54a11961f385a953b0b03. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.973039 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e47c4f741f354a19b49777bd06e54764. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.973783 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95efd553493e4966bc48a01cac6e1e10. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.974274 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d46673a283fe42b78ddeb99fbd2678eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.974460 892 tablet_service.cc:1505] Processing CreateTablet for tablet 792f32cbd66449ca98813c669b2a261d (DEFAULT_TABLE table=default.Table_23cacc46bec647adbc62b457f73487cf [id=a8b081495f564c10aa6a05e5eed80488]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.975482 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67cf96a7dd3f42e19b86746c87e689f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.975824 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 288e543bb279410c91c01dd1623e110f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:31.975870 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6d3c9954693d4e8782ee0128ef066aa9 consensus state has no leader
I20251025 07:52:31.976428 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 792f32cbd66449ca98813c669b2a261d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.979022 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b940fa5ae26427dafbcfd26372754b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:31.979278 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 243b8828fc344bacab46d7bab6bd2dec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.979475 1170 ts_tablet_manager.cc:1916] T 742b3ec3f6604647b9e397532f1673dd P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:31.980057 712 leader_election.cc:304] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters: 3aea2869ff814cc7b142012faa274382
I20251025 07:52:31.981047 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "742b3ec3f6604647b9e397532f1673dd" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:31.981402 1441 raft_consensus.cc:2804] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:31.981506 1441 raft_consensus.cc:493] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:31.981626 1441 raft_consensus.cc:3060] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:31.982214 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df9db9dd31d840e0b1394db04d662a62. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.984149 876 tablet_service.cc:1505] Processing CreateTablet for tablet a6ae1315b28940d49b667e3c80347021 (DEFAULT_TABLE table=default.Table_ac3211ef7b0941d2ad4db2fab62231b2 [id=f721af9327cd4e00a20fc1bfebd9e17b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.985484 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b278ce7768349e49dcd1131e404ab34. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.985730 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8917a289bd9b4428a4603429edd94581. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.986308 1020 tablet_service.cc:1505] Processing CreateTablet for tablet 22fc90c704f64d2f9f4b8c54dee45576 (DEFAULT_TABLE table=default.Table_e317f8d728da4c76886f67d0995ab7f4 [id=4d2121dea4904850aecb65da5dd2d8f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.986464 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 50ffbcdd4a9247bf99b8e2361bc02ef5 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]) successfully deleted
I20251025 07:52:31.986932 1172 ts_tablet_manager.cc:1929] T 79bc2678df8042e3994b0471afa2c304 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:31.987207 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6ae1315b28940d49b667e3c80347021. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.987227 1017 tablet_service.cc:1505] Processing CreateTablet for tablet df0d86ad301245c1a6bcb44805936d25 (DEFAULT_TABLE table=default.Table_e90d7f5ff13b40878e5b625c97792c7d [id=fc8180c09d7b4e74b2e161f7acb8fe0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.988755 1167 tablet_bootstrap.cc:654] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:31.989120 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22fc90c704f64d2f9f4b8c54dee45576. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.990042 755 tablet_service.cc:1505] Processing CreateTablet for tablet 3b940fa5ae26427dafbcfd26372754b3 (DEFAULT_TABLE table=default.Table_d57e7ea8e59641f781c564e5b01479bc [id=fbaf826daa7c448eb92c7f93826260c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.990460 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b940fa5ae26427dafbcfd26372754b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.996678 1026 tablet_service.cc:1505] Processing CreateTablet for tablet e429b94326cc4827a03968fd92de38e0 (DEFAULT_TABLE table=default.Table_0b0bd6b71b92454fb8cdf967ffd8349e [id=e9b21659de7d4276aa1f9effb7076eb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.997501 1218 raft_consensus.cc:493] T f3c87a83082b421cb4be546ac4ce7ad8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:31.997694 1218 raft_consensus.cc:515] T f3c87a83082b421cb4be546ac4ce7ad8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:31.997969 875 tablet_service.cc:1505] Processing CreateTablet for tablet e4da89f863cc43f3bf6df908bb9dc6bf (DEFAULT_TABLE table=default.Table_3d995c9c75174000b6cb98779cc0bb2f [id=6c1ffab0043545c2b7cce93a641eaa1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.998788 883 tablet_service.cc:1505] Processing CreateTablet for tablet df0d86ad301245c1a6bcb44805936d25 (DEFAULT_TABLE table=default.Table_e90d7f5ff13b40878e5b625c97792c7d [id=fc8180c09d7b4e74b2e161f7acb8fe0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:31.999176 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df0d86ad301245c1a6bcb44805936d25. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:31.999841 884 tablet_service.cc:1505] Processing CreateTablet for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (DEFAULT_TABLE table=default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.000974 1172 log.cc:1199] T 79bc2678df8042e3994b0471afa2c304 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/79bc2678df8042e3994b0471afa2c304
I20251025 07:52:32.005075 914 raft_consensus.cc:2393] T 742b3ec3f6604647b9e397532f1673dd P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:32.007232 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3c87a83082b421cb4be546ac4ce7ad8" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:32.007921 850 leader_election.cc:343] T f3c87a83082b421cb4be546ac4ce7ad8 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:32.008678 1021 tablet_service.cc:1505] Processing CreateTablet for tablet 3e79d69d5d004cc395b8e35569e25440 (DEFAULT_TABLE table=default.Table_23d4d4a3c90c47ea8230c39a51138418 [id=a7070a0269fc4fc99447ab671eb226be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.009327 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df0d86ad301245c1a6bcb44805936d25. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.014828 1441 raft_consensus.cc:515] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:32.018253 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0240d20f9904b4bbc8bf7f0f05e8850. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.018200 1441 leader_election.cc:290] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:32.018692 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e429b94326cc4827a03968fd92de38e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.019086 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e79d69d5d004cc395b8e35569e25440. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.019814 1169 tablet_bootstrap.cc:492] T 79bc2678df8042e3994b0471afa2c304 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:32.019860 880 tablet_service.cc:1505] Processing CreateTablet for tablet 22fc90c704f64d2f9f4b8c54dee45576 (DEFAULT_TABLE table=default.Table_e317f8d728da4c76886f67d0995ab7f4 [id=4d2121dea4904850aecb65da5dd2d8f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.019945 1169 ts_tablet_manager.cc:1403] T 79bc2678df8042e3994b0471afa2c304 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.126s user 0.024s sys 0.025s
I20251025 07:52:32.022451 1023 tablet_service.cc:1505] Processing CreateTablet for tablet af743ee344e543a0aa6da3cd7a30e467 (DEFAULT_TABLE table=default.Table_843881ca867542619b4fff840d2879ca [id=d324f310a01a4ae5abbff6ddafffca7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.022832 1166 tablet_bootstrap.cc:492] T 2784bbfb97d5466f9ec856add7d0b29a P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:32.020720 1169 raft_consensus.cc:359] T 79bc2678df8042e3994b0471afa2c304 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:32.022993 887 tablet_service.cc:1505] Processing CreateTablet for tablet 8917a289bd9b4428a4603429edd94581 (DEFAULT_TABLE table=default.Table_45faec266a9b4b1b9f5ef3ac915e3bbc [id=bc377ac4bb164a80b9ebce289a790326]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.023671 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 95efd553493e4966bc48a01cac6e1e10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:32.023142 879 tablet_service.cc:1505] Processing CreateTablet for tablet 062c0f3e51a3487880cea373f8cf1f54 (DEFAULT_TABLE table=default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.024716 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22fc90c704f64d2f9f4b8c54dee45576. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.024842 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af743ee344e543a0aa6da3cd7a30e467. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.025967 877 tablet_service.cc:1505] Processing CreateTablet for tablet e429b94326cc4827a03968fd92de38e0 (DEFAULT_TABLE table=default.Table_0b0bd6b71b92454fb8cdf967ffd8349e [id=e9b21659de7d4276aa1f9effb7076eb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.027208 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "742b3ec3f6604647b9e397532f1673dd" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:32.027499 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90764275825c4ffb8712cb883d37a551" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:32.027688 1046 raft_consensus.cc:3060] T 90764275825c4ffb8712cb883d37a551 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:32.028501 740 tablet_service.cc:1505] Processing CreateTablet for tablet ed243540f2d746b980c25687fbca7414 (DEFAULT_TABLE table=default.Table_208daf030a0d4efdb6ef8250b2f48896 [id=7507e4bd03ca4dd2a93655b138e8d49c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.025617 893 tablet_service.cc:1505] Processing CreateTablet for tablet 3e79d69d5d004cc395b8e35569e25440 (DEFAULT_TABLE table=default.Table_23d4d4a3c90c47ea8230c39a51138418 [id=a7070a0269fc4fc99447ab671eb226be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.031112 1218 leader_election.cc:290] T f3c87a83082b421cb4be546ac4ce7ad8 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:32.031286 743 tablet_service.cc:1505] Processing CreateTablet for tablet 075f0a151e024220b51ba7fcaecdcc3d (DEFAULT_TABLE table=default.Table_26b3fa566ddf4e5aadd68de388cb61ba [id=b064a22cfdc04a0793ffd858b3fdc45e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.031709 745 tablet_service.cc:1505] Processing CreateTablet for tablet 72618756f756484e895e885ef12545c8 (DEFAULT_TABLE table=default.Table_dbe39538a228480a9ff9cd45159e39f7 [id=2965a45ad79e4523ab42d65823f19c8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.031740 1169 raft_consensus.cc:385] T 79bc2678df8042e3994b0471afa2c304 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:32.032109 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b940fa5ae26427dafbcfd26372754b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:32.032182 774 raft_consensus.cc:2393] T 742b3ec3f6604647b9e397532f1673dd P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:32.032428 747 tablet_service.cc:1505] Processing CreateTablet for tablet a6ae1315b28940d49b667e3c80347021 (DEFAULT_TABLE table=default.Table_ac3211ef7b0941d2ad4db2fab62231b2 [id=f721af9327cd4e00a20fc1bfebd9e17b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.032470 748 tablet_service.cc:1505] Processing CreateTablet for tablet e4da89f863cc43f3bf6df908bb9dc6bf (DEFAULT_TABLE table=default.Table_3d995c9c75174000b6cb98779cc0bb2f [id=6c1ffab0043545c2b7cce93a641eaa1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.032725 738 tablet_service.cc:1505] Processing CreateTablet for tablet 288e543bb279410c91c01dd1623e110f (DEFAULT_TABLE table=default.Table_ea269532aff34ca1803fc1c5d5ad3641 [id=4c34b64b299f4162a53866cf01c5298f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.040243 752 tablet_service.cc:1505] Processing CreateTablet for tablet 243b8828fc344bacab46d7bab6bd2dec (DEFAULT_TABLE table=default.Table_d647b0fa975743998d7f49c36abe9c96 [id=5b435831d8254ae8b126218a39e3a362]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.040608 749 tablet_service.cc:1505] Processing CreateTablet for tablet 792f32cbd66449ca98813c669b2a261d (DEFAULT_TABLE table=default.Table_23cacc46bec647adbc62b457f73487cf [id=a8b081495f564c10aa6a05e5eed80488]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.042827 881 tablet_service.cc:1505] Processing CreateTablet for tablet af743ee344e543a0aa6da3cd7a30e467 (DEFAULT_TABLE table=default.Table_843881ca867542619b4fff840d2879ca [id=d324f310a01a4ae5abbff6ddafffca7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.043663 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8917a289bd9b4428a4603429edd94581. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.047839 522 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 627a10a0714f4be8bdbced503fd6388d (table default.Table_6b937442c9614efeb8cc1562ac306174 [id=7fc07b0c33f74604be526c4a0df949e9]). Expected version 1 got 0
I20251025 07:52:32.048797 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_f8c38604344c429295d6d0ab9cb85b13" } modify_external_catalogs: true
I20251025 07:52:32.049037 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_f8c38604344c429295d6d0ab9cb85b13" } modify_external_catalogs: true
I20251025 07:52:32.049479 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_d5abcbf8c8c741a2b40880a3467f2bcd" } modify_external_catalogs: true
I20251025 07:52:32.049669 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_d5abcbf8c8c741a2b40880a3467f2bcd" } modify_external_catalogs: true
I20251025 07:52:32.049803 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_7d2010ece331457fa18a0f13ac5cbdfa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:32.050397 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d2010ece331457fa18a0f13ac5cbdfa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:32.050711 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_208daf030a0d4efdb6ef8250b2f48896" } new_table_name: "default.Table_616a4d314c7445b0b4f4283d76c0bea8" modify_external_catalogs: true
I20251025 07:52:32.050843 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_06053a14dcc84e79917273c10b06b751" } modify_external_catalogs: true
I20251025 07:52:32.051050 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_06053a14dcc84e79917273c10b06b751" } modify_external_catalogs: true
I20251025 07:52:32.052085 1172 ts_tablet_manager.cc:1950] T 79bc2678df8042e3994b0471afa2c304 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:32.052110 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_0ae70257430b4a1ea853e49f9720cbea" } new_table_name: "default.Table_87d38832f85e4965bfab1a077990114f" modify_external_catalogs: true
I20251025 07:52:32.052265 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_4b1446ea61a9442fb3e397dfe2c5fb03"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:32.052662 1172 tablet_replica.cc:333] T 47284f96928749e6a5201103a8ee5f9f P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
W20251025 07:52:32.052749 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b1446ea61a9442fb3e397dfe2c5fb03 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:32.052886 1172 raft_consensus.cc:2243] T 47284f96928749e6a5201103a8ee5f9f P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:32.053154 1172 pending_rounds.cc:70] T 47284f96928749e6a5201103a8ee5f9f P 3aea2869ff814cc7b142012faa274382: Trying to abort 1 pending ops.
I20251025 07:52:32.053238 1172 pending_rounds.cc:77] T 47284f96928749e6a5201103a8ee5f9f P 3aea2869ff814cc7b142012faa274382: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214607366463311872 op_type: NO_OP noop_request { }
I20251025 07:52:32.053354 1172 raft_consensus.cc:2889] T 47284f96928749e6a5201103a8ee5f9f P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 07:52:32.053475 1172 raft_consensus.cc:2272] T 47284f96928749e6a5201103a8ee5f9f P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:32.053831 888 tablet_service.cc:1505] Processing CreateTablet for tablet df9db9dd31d840e0b1394db04d662a62 (DEFAULT_TABLE table=default.Table_da4ed35a8ed24427b6f4499fd1cc203c [id=cdbb76e674fc41cfb0f86d678a3f4f0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.057009 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e429b94326cc4827a03968fd92de38e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.057320 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ce62847e80f48e8807fdbbbdc50803c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.057034 1166 ts_tablet_manager.cc:1403] T 2784bbfb97d5466f9ec856add7d0b29a P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.224s user 0.026s sys 0.020s
I20251025 07:52:32.058022 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 062c0f3e51a3487880cea373f8cf1f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.058382 1166 raft_consensus.cc:359] T 2784bbfb97d5466f9ec856add7d0b29a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:32.058585 1166 raft_consensus.cc:385] T 2784bbfb97d5466f9ec856add7d0b29a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:32.058607 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df9db9dd31d840e0b1394db04d662a62. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.058663 1166 raft_consensus.cc:740] T 2784bbfb97d5466f9ec856add7d0b29a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:32.058916 1166 consensus_queue.cc:260] T 2784bbfb97d5466f9ec856add7d0b29a P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:32.059393 1166 ts_tablet_manager.cc:1434] T 2784bbfb97d5466f9ec856add7d0b29a P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:32.059716 1166 tablet_bootstrap.cc:492] T a13bf4d7eb1440f7a28b85cb1bf99667 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:32.060722 885 tablet_service.cc:1505] Processing CreateTablet for tablet 4b278ce7768349e49dcd1131e404ab34 (DEFAULT_TABLE table=default.Table_309913c03a4d46019d35478363f149e8 [id=bad99c3045fc4b938e20d2dcfdd0b1bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.066365 741 tablet_service.cc:1505] Processing CreateTablet for tablet df0d86ad301245c1a6bcb44805936d25 (DEFAULT_TABLE table=default.Table_e90d7f5ff13b40878e5b625c97792c7d [id=fc8180c09d7b4e74b2e161f7acb8fe0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.067200 1441 raft_consensus.cc:493] T 567a1f5c216a4a519249da66a2a03ea4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:32.067945 754 tablet_service.cc:1505] Processing CreateTablet for tablet 8917a289bd9b4428a4603429edd94581 (DEFAULT_TABLE table=default.Table_45faec266a9b4b1b9f5ef3ac915e3bbc [id=bc377ac4bb164a80b9ebce289a790326]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.070365 1172 ts_tablet_manager.cc:1916] T 47284f96928749e6a5201103a8ee5f9f P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:32.071920 1018 tablet_service.cc:1505] Processing CreateTablet for tablet 2e098c9c4613459295b6043048cba587 (DEFAULT_TABLE table=default.Table_7b8e9726fe804f38a47ee1f428f4b666 [id=55440328803042a28df1b6869d279f59]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.079066 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 792f32cbd66449ca98813c669b2a261d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.083561 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed243540f2d746b980c25687fbca7414. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.083824 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 288e543bb279410c91c01dd1623e110f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.085057 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4da89f863cc43f3bf6df908bb9dc6bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.085269 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b278ce7768349e49dcd1131e404ab34. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.077657 1441 raft_consensus.cc:515] T 567a1f5c216a4a519249da66a2a03ea4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:32.071492 982 leader_election.cc:304] T 742b3ec3f6604647b9e397532f1673dd P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:32.088207 742 tablet_service.cc:1505] Processing CreateTablet for tablet 062c0f3e51a3487880cea373f8cf1f54 (DEFAULT_TABLE table=default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.092700 1441 leader_election.cc:290] T 567a1f5c216a4a519249da66a2a03ea4 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:32.093379 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 075f0a151e024220b51ba7fcaecdcc3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.096415 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 79bc2678df8042e3994b0471afa2c304 (table default.Table_ca46a7a783e547998d1be8476801a9e0 [id=525e8a793cf14cd492ca2ee986231311]) successfully deleted
W20251025 07:52:32.097420 712 leader_election.cc:343] T 567a1f5c216a4a519249da66a2a03ea4 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 567a1f5c216a4a519249da66a2a03ea4
I20251025 07:52:32.099107 1561 raft_consensus.cc:493] T 21e52928580c4762b0ae7152490ef955 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:32.099261 1561 raft_consensus.cc:515] T 21e52928580c4762b0ae7152490ef955 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:32.101349 1218 raft_consensus.cc:493] T ac0ea298bc114c56a8093e12def21ce1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:32.103910 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e79d69d5d004cc395b8e35569e25440. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.104956 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_9bde97409538452a99917ba1e4ed04b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:32.105484 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9bde97409538452a99917ba1e4ed04b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:32.108839 1046 raft_consensus.cc:2468] T 90764275825c4ffb8712cb883d37a551 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:32.111598 1218 raft_consensus.cc:515] T ac0ea298bc114c56a8093e12def21ce1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:32.112277 1218 leader_election.cc:290] T ac0ea298bc114c56a8093e12def21ce1 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:32.112552 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 72dc084f8efc45478771c247e07a16f4 (DEFAULT_TABLE table=default.Table_5a8143ecfd74418f928ceb9ae9cd50ba [id=23d0aa66d6f545d4b579cfd63b7e7143]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.114775 1048 raft_consensus.cc:3060] T 8c6ff4c73e624af3bf2de36c4493b462 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:32.116796 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e098c9c4613459295b6043048cba587. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.118216 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 26c849d376d143c6bef0c8e0e1d7d03a
W20251025 07:52:32.118508 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d46673a283fe42b78ddeb99fbd2678eb consensus state has no leader
I20251025 07:52:32.120530 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_e42a1f7287a34753b7e4daf36893d9c2" } modify_external_catalogs: true
I20251025 07:52:32.120725 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_e42a1f7287a34753b7e4daf36893d9c2" } modify_external_catalogs: true
I20251025 07:52:32.123590 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_3d995c9c75174000b6cb98779cc0bb2f" } modify_external_catalogs: true
I20251025 07:52:32.123760 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_3d995c9c75174000b6cb98779cc0bb2f" } modify_external_catalogs: true
I20251025 07:52:32.124673 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "08da9d2687704725aacd11cd92c6a9f9" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:32.125032 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd9b4be139d347f4a6a60767029c342a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:32.126736 1169 raft_consensus.cc:740] T 79bc2678df8042e3994b0471afa2c304 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:32.128113 1169 consensus_queue.cc:260] T 79bc2678df8042e3994b0471afa2c304 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:32.129951 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ed243540f2d746b980c25687fbca7414 consensus state has no leader
I20251025 07:52:32.130517 1561 leader_election.cc:290] T 21e52928580c4762b0ae7152490ef955 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:32.140416 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 26c849d376d143c6bef0c8e0e1d7d03a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:47408
W20251025 07:52:32.142211 982 raft_consensus.cc:2686] T 742b3ec3f6604647b9e397532f1673dd P db42be7ad3c741a0aa427f27e245993b: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251025 07:52:32.142450 982 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:46231, user_credentials={real_user=slave}} blocked reactor thread for 108258us
I20251025 07:52:32.142918 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 48b73e0c0dce4be686d05f991a21bd19 (DEFAULT_TABLE table=default.Table_3bd421885a1b4e06b11d12780c34cad4 [id=4d9e99d6474b4d00bc66e26a33694e69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.142894 886 tablet_service.cc:1505] Processing CreateTablet for tablet 2e098c9c4613459295b6043048cba587 (DEFAULT_TABLE table=default.Table_7b8e9726fe804f38a47ee1f428f4b666 [id=55440328803042a28df1b6869d279f59]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.143787 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 4f7c15529be54a11961f385a953b0b03
W20251025 07:52:32.147737 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c501ef5d70e742598c1a902efe47feb1 consensus state has no leader
I20251025 07:52:32.149039 894 tablet_service.cc:1505] Processing CreateTablet for tablet 72dc084f8efc45478771c247e07a16f4 (DEFAULT_TABLE table=default.Table_5a8143ecfd74418f928ceb9ae9cd50ba [id=23d0aa66d6f545d4b579cfd63b7e7143]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.151265 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 3aeaa0e9dc594e21a9ddd923e314017d (DEFAULT_TABLE table=default.Table_51f2b43545db41029568a0321a37e139 [id=5a499fdacb8047dc9ce9bed92d7dd233]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.153151 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72dc084f8efc45478771c247e07a16f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.153363 1166 tablet_bootstrap.cc:654] T a13bf4d7eb1440f7a28b85cb1bf99667 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:32.147822 1561 consensus_queue.cc:1048] T 8c6ff4c73e624af3bf2de36c4493b462 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20251025 07:52:32.156299 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af743ee344e543a0aa6da3cd7a30e467. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.155915 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac0ea298bc114c56a8093e12def21ce1" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:32.157740 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48b73e0c0dce4be686d05f991a21bd19. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.158464 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3aeaa0e9dc594e21a9ddd923e314017d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.159482 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet f4578ce69a0d4322944e5ce7c8a59831 with delete_type TABLET_DATA_DELETED (replaced by tablet 48b73e0c0dce4be686d05f991a21bd19) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:32.160943 750 tablet_service.cc:1505] Processing CreateTablet for tablet 6ce62847e80f48e8807fdbbbdc50803c (DEFAULT_TABLE table=default.Table_dad8e9c26a9a41e2bcf3fa1db9f5b087 [id=579250c3c8fb4e9783e388a53ba35a6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.162798 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 243b8828fc344bacab46d7bab6bd2dec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.163033 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ce62847e80f48e8807fdbbbdc50803c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.164436 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 062c0f3e51a3487880cea373f8cf1f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.164403 755 tablet_service.cc:1505] Processing CreateTablet for tablet 2d24e3e7a9e4431db609331ad1fd57c9 (DEFAULT_TABLE table=default.Table_2ddc7ef45af34002a5db135b33ec855e [id=7e34aeb0c3ea4fc5ba6a9077a79335d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.164786 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d24e3e7a9e4431db609331ad1fd57c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.165537 712 leader_election.cc:304] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
W20251025 07:52:32.165786 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:45417, user_credentials={real_user=slave}} blocked reactor thread for 55988.3us
W20251025 07:52:32.166358 850 leader_election.cc:343] T ac0ea298bc114c56a8093e12def21ce1 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:32.170606 1441 raft_consensus.cc:2804] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:32.170845 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6ae1315b28940d49b667e3c80347021. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.171397 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet ce1ebb1ec14b4801930a72765bca1a7d
I20251025 07:52:32.172874 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3c87a83082b421cb4be546ac4ce7ad8" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:32.174414 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4da89f863cc43f3bf6df908bb9dc6bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.174898 746 tablet_service.cc:1505] Processing CreateTablet for tablet df9db9dd31d840e0b1394db04d662a62 (DEFAULT_TABLE table=default.Table_da4ed35a8ed24427b6f4499fd1cc203c [id=cdbb76e674fc41cfb0f86d678a3f4f0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.175016 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df0d86ad301245c1a6bcb44805936d25. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.176563 737 tablet_service.cc:1505] Processing CreateTablet for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (DEFAULT_TABLE table=default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.179663 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet e4da89f863cc43f3bf6df908bb9dc6bf
I20251025 07:52:32.183115 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet d77704b4fe6c4456a2ba40ffec0c60fc
I20251025 07:52:32.192360 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_6f5fbeb82b024bfe95511811dcc51406"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:32.193145 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f5fbeb82b024bfe95511811dcc51406 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:32.196000 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_e8d761a95cdf469b9cba89359c58a5a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:32.196595 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e8d761a95cdf469b9cba89359c58a5a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:32.199414 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_50fe33e9d42343ff9194780a28b1b1b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:32.199975 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50fe33e9d42343ff9194780a28b1b1b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:32.201069 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet ce1ebb1ec14b4801930a72765bca1a7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:32.202276 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_d3207dcbfb93438e9f51b211e8b1ebdc" } new_table_name: "default.Table_92c0120dd29348618040150dedb14c43" modify_external_catalogs: true
I20251025 07:52:32.204612 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_bef97bb8f0a442f3a23cb20cbf85d934" } new_table_name: "default.Table_df5799a9181a48bb84af0f3c767e7c29" modify_external_catalogs: true
I20251025 07:52:32.215526 1169 ts_tablet_manager.cc:1434] T 79bc2678df8042e3994b0471afa2c304 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.112s user 0.002s sys 0.002s
I20251025 07:52:32.215569 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e52928580c4762b0ae7152490ef955" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:32.217052 1015 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f7c15529be54a11961f385a953b0b03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:32.219537 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df9db9dd31d840e0b1394db04d662a62. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.224493 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e098c9c4613459295b6043048cba587. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:32.234656 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6897405bbc1c47a4ab0aa3af80e60e3d consensus state has no leader
W20251025 07:52:32.235097 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 37c7938e8c5e41f08506aff6747801c9 consensus state has no leader
I20251025 07:52:32.236311 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet e4da89f863cc43f3bf6df908bb9dc6bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:32.237313 1011 tablet_service.cc:1552] Processing DeleteTablet for tablet d77704b4fe6c4456a2ba40ffec0c60fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:32.237746 1017 tablet_service.cc:1505] Processing CreateTablet for tablet 5fc092c2bb2d41ac8c5b4fc07460c8ec (DEFAULT_TABLE table=default.Table_974f1ee455d7488593e5c496cc012099 [id=630c274339664b9db18ab4ef7c29c16f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.238610 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fc092c2bb2d41ac8c5b4fc07460c8ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.239279 1169 tablet_bootstrap.cc:492] T 5cba9c05dd424cc788febc614a34819a P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:32.239933 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72dc084f8efc45478771c247e07a16f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.240221 882 tablet_service.cc:1552] Processing DeleteTablet for tablet f4578ce69a0d4322944e5ce7c8a59831 with delete_type TABLET_DATA_DELETED (replaced by tablet 48b73e0c0dce4be686d05f991a21bd19) from {username='slave'} at 127.0.0.1:56590
W20251025 07:52:32.241842 850 leader_election.cc:343] T 21e52928580c4762b0ae7152490ef955 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:32.242851 1011 tablet_service.cc:1505] Processing CreateTablet for tablet 9903e80a26fd469a8744dc855534ba75 (DEFAULT_TABLE table=default.Table_3d4495f2005e498a928c68b14b8017a3 [id=f6e8d725be3e47d3a3f682510f90b236]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.246424 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet 79bc2678df8042e3994b0471afa2c304 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:32.241437 1010 tablet_service.cc:1505] Processing CreateTablet for tablet 32a143a715cb485cb4501f4e28c68252 (DEFAULT_TABLE table=default.Table_e386bc02f89041418ff3ddf8ac224eea [id=e800c96844e843b7a6fd3145907e5503]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.247162 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9903e80a26fd469a8744dc855534ba75. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.247861 1016 tablet_service.cc:1505] Processing CreateTablet for tablet aea8fcf0154a4bd4a05b19e27ab78b79 (DEFAULT_TABLE table=default.Table_c26454d3c928434ab0dae167abcc2ff2 [id=4cd0c9a010af47939a90da1132b9e6be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.249174 1014 tablet_service.cc:1505] Processing CreateTablet for tablet 22f7db660616457b83b24e947a0b0bae (DEFAULT_TABLE table=default.Table_3d163f45f94f450ebf0447897bdf41a6 [id=bb316a7e000748ca902a9bfb494802d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.251271 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_a55d3aa69b804240b042e7b94ca493d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:32.251924 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a55d3aa69b804240b042e7b94ca493d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:32.253562 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32a143a715cb485cb4501f4e28c68252. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.257418 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_590ce35533c7442390ce8991bcfa80fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:32.257997 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_590ce35533c7442390ce8991bcfa80fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:32.261269 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22f7db660616457b83b24e947a0b0bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.262563 1170 ts_tablet_manager.cc:1929] T 742b3ec3f6604647b9e397532f1673dd P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:32.262702 1170 log.cc:1199] T 742b3ec3f6604647b9e397532f1673dd P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/742b3ec3f6604647b9e397532f1673dd
I20251025 07:52:32.264884 1170 ts_tablet_manager.cc:1950] T 742b3ec3f6604647b9e397532f1673dd P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:32.267302 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aea8fcf0154a4bd4a05b19e27ab78b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.284260 1170 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47408 (request call id 397) took 1224 ms. Trace:
I20251025 07:52:32.287874 1018 tablet_service.cc:1505] Processing CreateTablet for tablet ab68e5a89b234064b27aa7cae9467e14 (DEFAULT_TABLE table=default.Table_4b1446ea61a9442fb3e397dfe2c5fb03 [id=e9652929eaab46cab42bf8d6896577ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.288190 1021 tablet_service.cc:1505] Processing CreateTablet for tablet 0582ee42ab564c07ae3facfc5935617c (DEFAULT_TABLE table=default.Table_6aaf0d6ed7554bd4afa602cdca717ce2 [id=f32a2592b45047bda447d79e36427e31]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.289232 1013 tablet_service.cc:1505] Processing CreateTablet for tablet 8707b93d491e4e4386d148087a2f0870 (DEFAULT_TABLE table=default.Table_fd5720d69b084df1b36485b7890ac6c2 [id=59840a567e5244e8a9024cbad42471e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.289408 1015 tablet_service.cc:1505] Processing CreateTablet for tablet 1b22bd172e0e4f86896e9ecdef666b42 (DEFAULT_TABLE table=default.Table_9bde97409538452a99917ba1e4ed04b3 [id=eb909ab8da1d45e28008f447066bea71]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.297814 739 tablet_service.cc:1505] Processing CreateTablet for tablet 22fc90c704f64d2f9f4b8c54dee45576 (DEFAULT_TABLE table=default.Table_e317f8d728da4c76886f67d0995ab7f4 [id=4d2121dea4904850aecb65da5dd2d8f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.304996 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_915572e49894471d8933816077123d6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:32.242825 756 tablet_service.cc:1505] Processing CreateTablet for tablet 4b278ce7768349e49dcd1131e404ab34 (DEFAULT_TABLE table=default.Table_309913c03a4d46019d35478363f149e8 [id=bad99c3045fc4b938e20d2dcfdd0b1bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.298316 1028 tablet_service.cc:1505] Processing CreateTablet for tablet d40519ed9ee746348b17be2786d60e95 (DEFAULT_TABLE table=default.Table_7d2010ece331457fa18a0f13ac5cbdfa [id=c0e60203ceb3445ca63dc5f8f81c37b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.308629 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab68e5a89b234064b27aa7cae9467e14. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.310299 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0582ee42ab564c07ae3facfc5935617c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:32.310619 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_915572e49894471d8933816077123d6c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:32.312098 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8707b93d491e4e4386d148087a2f0870. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.312122 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8917a289bd9b4428a4603429edd94581. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.316669 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b22bd172e0e4f86896e9ecdef666b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.291088 1170 rpcz_store.cc:276] 1025 07:52:31.054050 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:31.127197 (+ 73147us) service_pool.cc:225] Handling call
1025 07:52:31.910054 (+782857us) ts_tablet_manager.cc:1151] Deleting tablet 742b3ec3f6604647b9e397532f1673dd
1025 07:52:31.921960 (+ 11906us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:32.186125 (+264165us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:32.262411 (+ 76286us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:32.278968 (+ 16557us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":761627}
I20251025 07:52:32.322569 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 742b3ec3f6604647b9e397532f1673dd (table default.Table_11117cabb5ca4540a58aac25d02964b2 [id=b6e173967bad47e6b306627221b80f94]) successfully deleted
I20251025 07:52:32.322609 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_f24cd210ae3342e68c69aa6670e3e4b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:32.323202 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f24cd210ae3342e68c69aa6670e3e4b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:32.324389 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_ba46ca0cf5d941e0879dead1c3544190" } modify_external_catalogs: true
I20251025 07:52:32.324579 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_ba46ca0cf5d941e0879dead1c3544190" } modify_external_catalogs: true
I20251025 07:52:32.326040 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_e8be4237bba844c8963e8955729b4fba" } new_table_name: "default.Table_a437a2623d704e3ea3c5b8c65c88b936" modify_external_catalogs: true
I20251025 07:52:32.327467 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d40519ed9ee746348b17be2786d60e95. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.333514 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_6b937442c9614efeb8cc1562ac306174" } new_table_name: "default.Table_8773c967a71644baba838946c154aee2" modify_external_catalogs: true
I20251025 07:52:32.338177 1022 tablet_service.cc:1505] Processing CreateTablet for tablet dbb254115a574bf4976aff0ea56f3bd9 (DEFAULT_TABLE table=default.Table_a55d3aa69b804240b042e7b94ca493d5 [id=83841f9dd7b34f27a36830af24022ab9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.338857 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbb254115a574bf4976aff0ea56f3bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.342527 1169 tablet_bootstrap.cc:654] T 5cba9c05dd424cc788febc614a34819a P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:32.344892 1170 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47408 (request call id 399) took 1130 ms. Trace:
I20251025 07:52:32.347800 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_157e52b1e07f4356afc68bb3ec5e4eb0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:32.348078 1170 rpcz_store.cc:276] 1025 07:52:31.200561 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:31.200729 (+ 168us) service_pool.cc:225] Handling call
1025 07:52:32.330163 (+1129434us) ts_tablet_manager.cc:1151] Deleting tablet 4f0c4f0e8f014e59be9e46d2afe0be9d
1025 07:52:32.330174 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:32.331164 (+ 990us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1109138}
W20251025 07:52:32.348426 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_157e52b1e07f4356afc68bb3ec5e4eb0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:32.348532 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 4f0c4f0e8f014e59be9e46d2afe0be9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f0c4f0e8f014e59be9e46d2afe0be9d already in progress: creating tablet
I20251025 07:52:32.349097 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 66134661be834f7b811d7927ab629812 (DEFAULT_TABLE table=default.Table_590ce35533c7442390ce8991bcfa80fb [id=7fd21f23b9ca483b8044392285400801]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.351064 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_b210463b66db4877b9a791f7cd741492" } modify_external_catalogs: true
I20251025 07:52:32.351244 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_b210463b66db4877b9a791f7cd741492" } modify_external_catalogs: true
I20251025 07:52:32.354030 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_7b8e9726fe804f38a47ee1f428f4b666" } new_table_name: "default.Table_9f52d94037424befbdea5c711068396a" modify_external_catalogs: true
I20251025 07:52:32.355170 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66134661be834f7b811d7927ab629812. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.355919 1171 ts_tablet_manager.cc:1929] T 85e27fc52e9a4d62a8622645594f59c6 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:32.356254 1016 tablet_service.cc:1505] Processing CreateTablet for tablet ee1c8ea694284d80a30f9c05be4cbf49 (DEFAULT_TABLE table=default.Table_e8d761a95cdf469b9cba89359c58a5a4 [id=b369bec99e4d48c1847a2a327b1d6ac8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.358566 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22fc90c704f64d2f9f4b8c54dee45576. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.358896 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72618756f756484e895e885ef12545c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.362979 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 716fb61312e94a4d9eb8ab8336ca140c
I20251025 07:52:32.363037 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 88eb51447e0f4080baa49dd902deb24f
I20251025 07:52:32.376398 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_2c8bad53187a49858df0c2bde3485e61"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:32.376519 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_517752d265854c7cbd706e2742536620" } modify_external_catalogs: true
I20251025 07:52:32.376727 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_517752d265854c7cbd706e2742536620" } modify_external_catalogs: true
W20251025 07:52:32.377030 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c8bad53187a49858df0c2bde3485e61 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:32.377346 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_c78d2d5f287d45ab9318f5c8a4c1e7ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:32.377874 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c78d2d5f287d45ab9318f5c8a4c1e7ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:32.379657 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 716fb61312e94a4d9eb8ab8336ca140c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:32.380501 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 88eb51447e0f4080baa49dd902deb24f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:32.380909 1017 tablet_service.cc:1505] Processing CreateTablet for tablet cc407c7eb12e4eadb41fb21385611c94 (DEFAULT_TABLE table=default.Table_50fe33e9d42343ff9194780a28b1b1b7 [id=01b6f41ccb23455b87ed38ed96184aa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.381333 1011 tablet_service.cc:1505] Processing CreateTablet for tablet f7011f1dcad74fe4ba0356c618890e06 (DEFAULT_TABLE table=default.Table_6f5fbeb82b024bfe95511811dcc51406 [id=9c2312d7095f407d9d30f224cdce66c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.382685 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_e08a3ff70f3a411e82b4bb292d768961" } modify_external_catalogs: true
I20251025 07:52:32.382890 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_e08a3ff70f3a411e82b4bb292d768961" } modify_external_catalogs: true
I20251025 07:52:32.385299 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_434ef1ade0a14af987864563993afae8" } modify_external_catalogs: true
I20251025 07:52:32.385459 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_434ef1ade0a14af987864563993afae8" } modify_external_catalogs: true
I20251025 07:52:32.385752 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee1c8ea694284d80a30f9c05be4cbf49. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:32.389307 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2e098c9c4613459295b6043048cba587 consensus state has no leader
I20251025 07:52:32.389117 1170 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47408 (request call id 398) took 1169 ms. Trace:
I20251025 07:52:32.394629 1170 rpcz_store.cc:276] 1025 07:52:31.200456 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:31.200753 (+ 297us) service_pool.cc:225] Handling call
1025 07:52:32.356023 (+1155270us) ts_tablet_manager.cc:1151] Deleting tablet c59f6e6eb9594a8e83f109a09b034045
1025 07:52:32.356036 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:32.369959 (+ 13923us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1132901}
I20251025 07:52:32.395085 1170 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47408 (request call id 400) took 1175 ms. Trace:
I20251025 07:52:32.395202 1170 rpcz_store.cc:276] 1025 07:52:31.219878 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:31.220007 (+ 129us) service_pool.cc:225] Handling call
1025 07:52:32.394945 (+1174938us) ts_tablet_manager.cc:1151] Deleting tablet d66ad049d5484228a268b0facb00ecf1
1025 07:52:32.394954 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:32.395052 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1171725}
I20251025 07:52:32.395586 1170 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47408 (request call id 401) took 1014 ms. Trace:
I20251025 07:52:32.395697 1170 rpcz_store.cc:276] 1025 07:52:31.381347 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:31.381440 (+ 93us) service_pool.cc:225] Handling call
1025 07:52:32.395427 (+1013987us) ts_tablet_manager.cc:1151] Deleting tablet a4e4b99be0434baaa75d988281797846
1025 07:52:32.395436 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:32.395555 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":989017}
I20251025 07:52:32.396250 1170 tablet_replica.cc:333] T 85fba8b37f2c4540aa81701052bf222a P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:32.396464 1170 raft_consensus.cc:2243] T 85fba8b37f2c4540aa81701052bf222a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:32.396644 1170 raft_consensus.cc:2272] T 85fba8b37f2c4540aa81701052bf222a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:32.397171 1170 ts_tablet_manager.cc:1916] T 85fba8b37f2c4540aa81701052bf222a P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:32.397830 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet c59f6e6eb9594a8e83f109a09b034045 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c59f6e6eb9594a8e83f109a09b034045 already in progress: creating tablet
W20251025 07:52:32.398180 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet d66ad049d5484228a268b0facb00ecf1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d66ad049d5484228a268b0facb00ecf1 already in progress: creating tablet
W20251025 07:52:32.398406 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet a4e4b99be0434baaa75d988281797846 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a4e4b99be0434baaa75d988281797846 already in progress: creating tablet
W20251025 07:52:32.398665 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet f0bdaa8cfd764f1b83f47881353d26b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0bdaa8cfd764f1b83f47881353d26b6 already in progress: creating tablet
I20251025 07:52:32.398808 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7011f1dcad74fe4ba0356c618890e06. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:32.398885 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet f43ef51ee45d47a99ec732b49d50d3b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f43ef51ee45d47a99ec732b49d50d3b4 already in progress: creating tablet
I20251025 07:52:32.400262 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc407c7eb12e4eadb41fb21385611c94. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.405062 882 tablet_service.cc:1505] Processing CreateTablet for tablet 3aeaa0e9dc594e21a9ddd923e314017d (DEFAULT_TABLE table=default.Table_51f2b43545db41029568a0321a37e139 [id=5a499fdacb8047dc9ce9bed92d7dd233]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.405550 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3aeaa0e9dc594e21a9ddd923e314017d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.412894 878 tablet_service.cc:1505] Processing CreateTablet for tablet 48b73e0c0dce4be686d05f991a21bd19 (DEFAULT_TABLE table=default.Table_3bd421885a1b4e06b11d12780c34cad4 [id=4d9e99d6474b4d00bc66e26a33694e69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.417124 1010 tablet_service.cc:1505] Processing CreateTablet for tablet 3c4970c8a491460e8b4b1c29fde83e4c (DEFAULT_TABLE table=default.Table_915572e49894471d8933816077123d6c [id=e372ca4384644779a6b37f389f549dec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.417572 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c4970c8a491460e8b4b1c29fde83e4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.420907 1013 tablet_service.cc:1505] Processing CreateTablet for tablet b71bdfd09a27449d98509bb220680147 (DEFAULT_TABLE table=default.Table_f24cd210ae3342e68c69aa6670e3e4b0 [id=22cab1e94aa544fa828350f9a70f8555]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.421458 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b71bdfd09a27449d98509bb220680147. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.424746 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 2b594d4718344aafbdaec34040695589
I20251025 07:52:32.426574 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b594d4718344aafbdaec34040695589 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:32.437302 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet fd265cdbf6a8440393a2b5d8240d1872
I20251025 07:52:32.438413 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 70fe9aa2f58d4abea3b3b71066a29653
W20251025 07:52:32.442464 512 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 6897405bbc1c47a4ab0aa3af80e60e3d (table default.Table_92c0120dd29348618040150dedb14c43 [id=ea892976886644d8aa025ae2edef718d], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6897405bbc1c47a4ab0aa3af80e60e3d consensus state has no leader
I20251025 07:52:32.444608 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 70fe9aa2f58d4abea3b3b71066a29653 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:32.445097 1169 tablet_bootstrap.cc:492] T 5cba9c05dd424cc788febc614a34819a P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:32.447767 1169 ts_tablet_manager.cc:1403] T 5cba9c05dd424cc788febc614a34819a P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.229s user 0.005s sys 0.004s
I20251025 07:52:32.449026 1169 raft_consensus.cc:359] T 5cba9c05dd424cc788febc614a34819a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:32.449431 1011 tablet_service.cc:1552] Processing DeleteTablet for tablet fd265cdbf6a8440393a2b5d8240d1872 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:32.449761 1169 raft_consensus.cc:385] T 5cba9c05dd424cc788febc614a34819a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:32.454671 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48b73e0c0dce4be686d05f991a21bd19. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.457602 1169 raft_consensus.cc:740] T 5cba9c05dd424cc788febc614a34819a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:32.458774 1021 tablet_service.cc:1505] Processing CreateTablet for tablet 0b47d1b0e6a342888135d7e4e5fa1f43 (DEFAULT_TABLE table=default.Table_c78d2d5f287d45ab9318f5c8a4c1e7ae [id=4d099e6134f548b49dec1897f6660560]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.460275 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b47d1b0e6a342888135d7e4e5fa1f43. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.462098 1011 tablet_service.cc:1505] Processing CreateTablet for tablet 421db850a64048218eecb9de65e2bdbf (DEFAULT_TABLE table=default.Table_157e52b1e07f4356afc68bb3ec5e4eb0 [id=a0facec9e7314c59b2175f0196869d73]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.464205 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 421db850a64048218eecb9de65e2bdbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.472859 1169 consensus_queue.cc:260] T 5cba9c05dd424cc788febc614a34819a P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:32.473644 1169 ts_tablet_manager.cc:1434] T 5cba9c05dd424cc788febc614a34819a P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20251025 07:52:32.476308 1169 tablet_bootstrap.cc:492] T 7e869c65771f4339a546dfeebb227c62 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:32.489837 1017 tablet_service.cc:1505] Processing CreateTablet for tablet 0063a0776b2247afaa8be82f6eff26f1 (DEFAULT_TABLE table=default.Table_2c8bad53187a49858df0c2bde3485e61 [id=1d22c44df6d24fed9c7009b7c7bd5a70]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.491317 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_d67fb372b12e40d082d119e53b9ebc3e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:32.491997 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d67fb372b12e40d082d119e53b9ebc3e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:32.494903 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0063a0776b2247afaa8be82f6eff26f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.495416 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_26b3fa566ddf4e5aadd68de388cb61ba" } new_table_name: "default.Table_cb2c8357cfa2432cbc3d708c62222479" modify_external_catalogs: true
I20251025 07:52:32.495822 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_4eaf06c0720744ff8b2fc8866cf05e2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:32.496384 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4eaf06c0720744ff8b2fc8866cf05e2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:32.498557 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_563d524d00364bd2819969149a8f436c" } new_table_name: "default.Table_3bbc24e4eb334b6487c9da0142b1fe03" modify_external_catalogs: true
I20251025 07:52:32.500519 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_7955f4594e0e4f7c803f728a43c229ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:32.501022 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7955f4594e0e4f7c803f728a43c229ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:32.503870 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 075f0a151e024220b51ba7fcaecdcc3d consensus state has no leader
I20251025 07:52:32.504817 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_3f291c48b0144f0a8ddb1b51ebbe31cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:32.505316 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f291c48b0144f0a8ddb1b51ebbe31cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:32.505815 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_173708dfd0274b34a7db6fa2b239a8b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:32.505488 1170 ts_tablet_manager.cc:1929] T 85fba8b37f2c4540aa81701052bf222a P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251025 07:52:32.506436 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_173708dfd0274b34a7db6fa2b239a8b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:32.507799 1170 log.cc:1199] T 85fba8b37f2c4540aa81701052bf222a P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/85fba8b37f2c4540aa81701052bf222a
I20251025 07:52:32.508819 1169 tablet_bootstrap.cc:654] T 7e869c65771f4339a546dfeebb227c62 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:32.513732 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 050a5d3b5f934dd19bf2e221f2fe0780 consensus state has no leader
I20251025 07:52:32.513159 1170 ts_tablet_manager.cc:1950] T 85fba8b37f2c4540aa81701052bf222a P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:32.528211 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 85fba8b37f2c4540aa81701052bf222a (table default.Table_5237c35dbfa7429e803d5008a8b3487c [id=0d597ecab6f54d1ba6cb5bd3436a9db8]) successfully deleted
W20251025 07:52:32.532048 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 1912e59548874e0e8e989050ec0064c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1912e59548874e0e8e989050ec0064c5 already in progress: creating tablet
W20251025 07:52:32.535475 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 34315f64b89148efbf1f20a43d5c5efb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34315f64b89148efbf1f20a43d5c5efb already in progress: creating tablet
W20251025 07:52:32.538484 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 95efd553493e4966bc48a01cac6e1e10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95efd553493e4966bc48a01cac6e1e10 already in progress: creating tablet
W20251025 07:52:32.541808 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet dda0b26e4f1a44928bc4b84f333901c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dda0b26e4f1a44928bc4b84f333901c0 already in progress: creating tablet
W20251025 07:52:32.545114 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 0f5f6ca6c3cd4d369690e425387a4344 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f5f6ca6c3cd4d369690e425387a4344 already in progress: creating tablet
I20251025 07:52:32.546906 1170 tablet_replica.cc:333] T 90764275825c4ffb8712cb883d37a551 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:32.548974 1170 raft_consensus.cc:2243] T 90764275825c4ffb8712cb883d37a551 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:32.551090 1169 tablet_bootstrap.cc:492] T 7e869c65771f4339a546dfeebb227c62 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:32.551239 1169 ts_tablet_manager.cc:1403] T 7e869c65771f4339a546dfeebb227c62 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.075s user 0.002s sys 0.004s
I20251025 07:52:32.550837 1170 raft_consensus.cc:2272] T 90764275825c4ffb8712cb883d37a551 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:32.553895 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_f16b7a3b7d91423f8502d73e4234ab4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:32.554467 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f16b7a3b7d91423f8502d73e4234ab4a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:32.555539 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 294d7d20c2f545b8813e632aa058a60c (DEFAULT_TABLE table=default.Table_4eaf06c0720744ff8b2fc8866cf05e2b [id=2b66e5f736de408bbd711b88361e1c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.555375 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_d49fa3011bf146b5a0dcb1c2d4557138"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:32.555939 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 294d7d20c2f545b8813e632aa058a60c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:32.556087 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d49fa3011bf146b5a0dcb1c2d4557138 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:32.557418 1169 raft_consensus.cc:359] T 7e869c65771f4339a546dfeebb227c62 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:32.557668 1169 raft_consensus.cc:385] T 7e869c65771f4339a546dfeebb227c62 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:32.557797 1169 raft_consensus.cc:740] T 7e869c65771f4339a546dfeebb227c62 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:32.558087 1169 consensus_queue.cc:260] T 7e869c65771f4339a546dfeebb227c62 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:32.558782 1169 ts_tablet_manager.cc:1434] T 7e869c65771f4339a546dfeebb227c62 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20251025 07:52:32.559186 1169 tablet_bootstrap.cc:492] T 8e6d76b1a97945cfb6c3055e1b622f8a P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:32.561144 1169 tablet_bootstrap.cc:654] T 8e6d76b1a97945cfb6c3055e1b622f8a P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:32.562819 1011 tablet_service.cc:1505] Processing CreateTablet for tablet 7e18611cbc1348eca927162fec70e36f (DEFAULT_TABLE table=default.Table_d67fb372b12e40d082d119e53b9ebc3e [id=a4f956a8e567440cbb5ad5214a0aa10a]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:32.564931 850 consensus_peers.cc:597] T 85fba8b37f2c4540aa81701052bf222a P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 85fba8b37f2c4540aa81701052bf222a. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:32.564961 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e18611cbc1348eca927162fec70e36f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.571233 1569 raft_consensus.cc:493] T 7e869c65771f4339a546dfeebb227c62 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:32.571434 1569 raft_consensus.cc:515] T 7e869c65771f4339a546dfeebb227c62 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:32.572295 1569 leader_election.cc:290] T 7e869c65771f4339a546dfeebb227c62 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:32.572005 1170 ts_tablet_manager.cc:1916] T 90764275825c4ffb8712cb883d37a551 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:32.573195 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e869c65771f4339a546dfeebb227c62" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:32.578627 914 raft_consensus.cc:2468] T 7e869c65771f4339a546dfeebb227c62 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:32.579281 983 leader_election.cc:304] T 7e869c65771f4339a546dfeebb227c62 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:32.579818 1569 raft_consensus.cc:2804] T 7e869c65771f4339a546dfeebb227c62 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:32.579928 1569 raft_consensus.cc:493] T 7e869c65771f4339a546dfeebb227c62 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:32.580030 1569 raft_consensus.cc:3060] T 7e869c65771f4339a546dfeebb227c62 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:32.581275 1166 tablet_bootstrap.cc:492] T a13bf4d7eb1440f7a28b85cb1bf99667 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:32.581488 1166 ts_tablet_manager.cc:1403] T a13bf4d7eb1440f7a28b85cb1bf99667 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.522s user 0.004s sys 0.002s
I20251025 07:52:32.582432 1166 raft_consensus.cc:359] T a13bf4d7eb1440f7a28b85cb1bf99667 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:32.582688 1166 raft_consensus.cc:385] T a13bf4d7eb1440f7a28b85cb1bf99667 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:32.583563 1166 raft_consensus.cc:740] T a13bf4d7eb1440f7a28b85cb1bf99667 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:32.587190 1569 raft_consensus.cc:515] T 7e869c65771f4339a546dfeebb227c62 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:32.588451 1569 leader_election.cc:290] T 7e869c65771f4339a546dfeebb227c62 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:32.590641 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e869c65771f4339a546dfeebb227c62" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:32.591810 914 raft_consensus.cc:3060] T 7e869c65771f4339a546dfeebb227c62 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:32.595646 914 raft_consensus.cc:2468] T 7e869c65771f4339a546dfeebb227c62 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:32.596236 983 leader_election.cc:304] T 7e869c65771f4339a546dfeebb227c62 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:32.605849 1013 tablet_service.cc:1505] Processing CreateTablet for tablet 60c60a9ee37f43978910bd067a098bb7 (DEFAULT_TABLE table=default.Table_173708dfd0274b34a7db6fa2b239a8b0 [id=6bc5ca21f8c54249a5150f0fea2686c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.606317 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60c60a9ee37f43978910bd067a098bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.611236 1029 tablet_service.cc:1505] Processing CreateTablet for tablet bbe1d40d67594ad389ccc97bb9e2ac3b (DEFAULT_TABLE table=default.Table_3f291c48b0144f0a8ddb1b51ebbe31cd [id=98db26b2e7bf409dbfe8d201ae701df7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.611680 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbe1d40d67594ad389ccc97bb9e2ac3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.614473 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_48712d8a17e04509888897a9cf2b4e30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:32.615092 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_48712d8a17e04509888897a9cf2b4e30 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:32.615965 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_f6781cf6e9104f51a278cab3d1781a1a" } modify_external_catalogs: true
I20251025 07:52:32.616643 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_50fe33e9d42343ff9194780a28b1b1b7" } modify_external_catalogs: true
I20251025 07:52:32.617429 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_262a9d20d20840aa85bb3dc06aa5dde6" } new_table_name: "default.Table_ab6798879a554075b3deaa059ad077c7" modify_external_catalogs: true
I20251025 07:52:32.617800 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_3bd421885a1b4e06b11d12780c34cad4" } new_table_name: "default.Table_1d0f9cc8a4004024af7c174b2d0a72aa" modify_external_catalogs: true
I20251025 07:52:32.618201 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_d9880621c8ca4025b89b6c1ec47f5cb4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:32.618959 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9880621c8ca4025b89b6c1ec47f5cb4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:32.619540 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_ddc8580fd1674f9ca61d528f22c51c94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:32.620009 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ddc8580fd1674f9ca61d528f22c51c94 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:32.621285 1166 consensus_queue.cc:260] T a13bf4d7eb1440f7a28b85cb1bf99667 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:32.622962 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_50fe33e9d42343ff9194780a28b1b1b7" } modify_external_catalogs: true
I20251025 07:52:32.624219 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_f6781cf6e9104f51a278cab3d1781a1a" } modify_external_catalogs: true
I20251025 07:52:32.626852 1166 ts_tablet_manager.cc:1434] T a13bf4d7eb1440f7a28b85cb1bf99667 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.045s user 0.002s sys 0.000s
W20251025 07:52:32.632622 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 48b73e0c0dce4be686d05f991a21bd19 consensus state has no leader
I20251025 07:52:32.636461 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet cc407c7eb12e4eadb41fb21385611c94
W20251025 07:52:32.641876 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5dab439bc50d4e869548154e1d222ad6 consensus state has no leader
I20251025 07:52:32.642620 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 1b5ac7e845a24b2eb5b2429d47361232
I20251025 07:52:32.643457 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 1b5ac7e845a24b2eb5b2429d47361232 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:32.643736 1569 raft_consensus.cc:2804] T 7e869c65771f4339a546dfeebb227c62 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:32.648217 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 5cba9c05dd424cc788febc614a34819a with delete_type TABLET_DATA_DELETED (replaced by tablet 4b28215f61574cf48c447dd01955ac77) from {username='slave'} at 127.0.0.1:47408
W20251025 07:52:32.649421 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4b28215f61574cf48c447dd01955ac77 consensus state has no leader
I20251025 07:52:32.650102 1166 tablet_bootstrap.cc:492] T fab67b56eaa44b54886c17181d8c52b2 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:32.658335 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet cc407c7eb12e4eadb41fb21385611c94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:32.665519 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 19f1b56cdbd24ebe8688cc05fb795d07 (DEFAULT_TABLE table=default.Table_7955f4594e0e4f7c803f728a43c229ca [id=95040fbb66f94779b7721e2209d82a65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.667320 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19f1b56cdbd24ebe8688cc05fb795d07. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.672835 1171 log.cc:1199] T 85e27fc52e9a4d62a8622645594f59c6 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/85e27fc52e9a4d62a8622645594f59c6
I20251025 07:52:32.673080 751 tablet_service.cc:1505] Processing CreateTablet for tablet 3e79d69d5d004cc395b8e35569e25440 (DEFAULT_TABLE table=default.Table_23d4d4a3c90c47ea8230c39a51138418 [id=a7070a0269fc4fc99447ab671eb226be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.676188 743 tablet_service.cc:1505] Processing CreateTablet for tablet e429b94326cc4827a03968fd92de38e0 (DEFAULT_TABLE table=default.Table_0b0bd6b71b92454fb8cdf967ffd8349e [id=e9b21659de7d4276aa1f9effb7076eb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.677421 1167 tablet_bootstrap.cc:492] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:32.677616 1167 ts_tablet_manager.cc:1403] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.912s user 0.006s sys 0.018s
I20251025 07:52:32.677699 1604 raft_consensus.cc:493] T 1aeaad5c5d01478bb997254ac372d9a6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:32.678546 1167 raft_consensus.cc:359] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:32.678718 1167 raft_consensus.cc:385] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:32.678814 1167 raft_consensus.cc:740] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:32.679032 1167 consensus_queue.cc:260] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:32.679520 1167 ts_tablet_manager.cc:1434] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:32.679811 1167 tablet_bootstrap.cc:492] T cd23b8094d7e472eba6bd170248858a9 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:32.680256 753 tablet_service.cc:1505] Processing CreateTablet for tablet 2e098c9c4613459295b6043048cba587 (DEFAULT_TABLE table=default.Table_7b8e9726fe804f38a47ee1f428f4b666 [id=55440328803042a28df1b6869d279f59]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.681180 738 tablet_service.cc:1505] Processing CreateTablet for tablet af743ee344e543a0aa6da3cd7a30e467 (DEFAULT_TABLE table=default.Table_843881ca867542619b4fff840d2879ca [id=d324f310a01a4ae5abbff6ddafffca7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.683704 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0240d20f9904b4bbc8bf7f0f05e8850. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.689956 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af743ee344e543a0aa6da3cd7a30e467. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.693864 1569 raft_consensus.cc:697] T 7e869c65771f4339a546dfeebb227c62 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:32.694315 1569 consensus_queue.cc:237] T 7e869c65771f4339a546dfeebb227c62 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:32.677878 1604 raft_consensus.cc:515] T 1aeaad5c5d01478bb997254ac372d9a6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:32.697427 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e429b94326cc4827a03968fd92de38e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.698832 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e098c9c4613459295b6043048cba587. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:32.698858 712 leader_election.cc:343] T 1aeaad5c5d01478bb997254ac372d9a6 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 1aeaad5c5d01478bb997254ac372d9a6
I20251025 07:52:32.704913 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_3bbc24e4eb334b6487c9da0142b1fe03" } modify_external_catalogs: true
I20251025 07:52:32.705140 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_3bbc24e4eb334b6487c9da0142b1fe03" } modify_external_catalogs: true
I20251025 07:52:32.705183 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_9b7fc5a1f4944e749c90c1119d0b0054" } modify_external_catalogs: true
I20251025 07:52:32.705539 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_9b7fc5a1f4944e749c90c1119d0b0054" } modify_external_catalogs: true
I20251025 07:52:32.706346 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_843881ca867542619b4fff840d2879ca" } modify_external_catalogs: true
I20251025 07:52:32.706578 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_843881ca867542619b4fff840d2879ca" } modify_external_catalogs: true
I20251025 07:52:32.709259 1021 tablet_service.cc:1505] Processing CreateTablet for tablet 698e9f8de7db4516a811c9eb9143a268 (DEFAULT_TABLE table=default.Table_48712d8a17e04509888897a9cf2b4e30 [id=488b550bbdef476c95e8bad42bd4b4dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.709816 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 698e9f8de7db4516a811c9eb9143a268. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.711483 1170 ts_tablet_manager.cc:1929] T 90764275825c4ffb8712cb883d37a551 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:32.711614 1170 log.cc:1199] T 90764275825c4ffb8712cb883d37a551 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/90764275825c4ffb8712cb883d37a551
I20251025 07:52:32.712545 1016 tablet_service.cc:1505] Processing CreateTablet for tablet 6fb1a1d38fb94f07b79ac4f8334fa719 (DEFAULT_TABLE table=default.Table_d49fa3011bf146b5a0dcb1c2d4557138 [id=fc927d99526b494db7c23ccf3e94fa02]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:32.720402 847 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.624s user 0.004s sys 0.012s
W20251025 07:52:32.720638 847 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.627s user 0.004s sys 0.012s
I20251025 07:52:32.726408 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 050a5d3b5f934dd19bf2e221f2fe0780
I20251025 07:52:32.727061 847 leader_election.cc:304] T 742b3ec3f6604647b9e397532f1673dd P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:32.727559 847 leader_election.cc:343] T 8e6d76b1a97945cfb6c3055e1b622f8a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:32.729971 847 leader_election.cc:304] T 8e6d76b1a97945cfb6c3055e1b622f8a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:32.730391 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 6dcab53a020847a1aa27930265ba0205
I20251025 07:52:32.731137 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet af743ee344e543a0aa6da3cd7a30e467
I20251025 07:52:32.732319 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet af743ee344e543a0aa6da3cd7a30e467 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:32.733137 1597 raft_consensus.cc:493] T 2784bbfb97d5466f9ec856add7d0b29a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:32.733291 1597 raft_consensus.cc:515] T 2784bbfb97d5466f9ec856add7d0b29a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:32.735124 1029 tablet_service.cc:1505] Processing CreateTablet for tablet ffe7c8e1d6c44872b7b65e512764a56a (DEFAULT_TABLE table=default.Table_d9880621c8ca4025b89b6c1ec47f5cb4 [id=be817d9282bd4fbf84adf11e25f06b5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.735872 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet 6dcab53a020847a1aa27930265ba0205 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:32.736605 1010 tablet_service.cc:1505] Processing CreateTablet for tablet 2c4789b9d89f42cb844e6cb595e687db (DEFAULT_TABLE table=default.Table_ddc8580fd1674f9ca61d528f22c51c94 [id=e33f00d4d5844ebba7ea3e200e154eeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.738672 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e869c65771f4339a546dfeebb227c62" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:32.739182 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e869c65771f4339a546dfeebb227c62" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:32.739728 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b278ce7768349e49dcd1131e404ab34. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.739817 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c4789b9d89f42cb844e6cb595e687db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.739566 1604 leader_election.cc:290] T 1aeaad5c5d01478bb997254ac372d9a6 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:32.740926 1599 raft_consensus.cc:493] T 627a10a0714f4be8bdbced503fd6388d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:32.751631 1172 ts_tablet_manager.cc:1929] T 47284f96928749e6a5201103a8ee5f9f P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:32.751842 1172 log.cc:1199] T 47284f96928749e6a5201103a8ee5f9f P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/47284f96928749e6a5201103a8ee5f9f
I20251025 07:52:32.751415 1597 leader_election.cc:290] T 2784bbfb97d5466f9ec856add7d0b29a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:32.753618 1597 raft_consensus.cc:2749] T 8e6d76b1a97945cfb6c3055e1b622f8a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:32.751547 1171 ts_tablet_manager.cc:1950] T 85e27fc52e9a4d62a8622645594f59c6 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:32.758172 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 400) took 1187 ms. Trace:
I20251025 07:52:32.758244 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_1431c97720de4d5a9a92bd2d574eb904"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:32.758567 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_25f26dbeafcf4afdaaa118dc47494351" } new_table_name: "default.Table_25a54dc35a8a4e15aff640922ebf13cf" modify_external_catalogs: true
W20251025 07:52:32.758934 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1431c97720de4d5a9a92bd2d574eb904 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:32.758016 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_ab4b1a5a7abb46fd978c91bb6d74840e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:32.759986 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_29aeaf089ca24270b3f21c2e08d61634"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:32.760147 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab4b1a5a7abb46fd978c91bb6d74840e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:32.760430 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_29aeaf089ca24270b3f21c2e08d61634 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:32.761152 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_b8551dc7ed8c48228c5d1fec6a1e8b0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:32.761600 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8551dc7ed8c48228c5d1fec6a1e8b0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:32.763692 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_2e41b219b43c4746bd24f0cb1e41547f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:32.764145 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e41b219b43c4746bd24f0cb1e41547f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:32.765234 1172 ts_tablet_manager.cc:1950] T 47284f96928749e6a5201103a8ee5f9f P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:32.766216 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 402) took 1024 ms. Trace:
I20251025 07:52:32.766571 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_932ccd57bc1840e7b882d16177ceef9a" } new_table_name: "default.Table_7eb8424d5192485da1a2d67bfdb2bbbf" modify_external_catalogs: true
W20251025 07:52:32.769236 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 141e413d2faa4c67b4c822192e631417 consensus state has no leader
I20251025 07:52:32.761112 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e79d69d5d004cc395b8e35569e25440. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:32.761188 1441 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.007s sys 0.006s
I20251025 07:52:32.778848 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fb1a1d38fb94f07b79ac4f8334fa719. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.780777 525 catalog_manager.cc:5649] T 7e869c65771f4339a546dfeebb227c62 P db42be7ad3c741a0aa427f27e245993b reported cstate change: term changed from 0 to 1, leader changed from <none> to db42be7ad3c741a0aa427f27e245993b (127.0.104.3). New cstate: current_term: 1 leader_uuid: "db42be7ad3c741a0aa427f27e245993b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: UNKNOWN } } }
W20251025 07:52:32.786698 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ab42909afc524f2fa3c1041495d13a59 consensus state has no leader
W20251025 07:52:32.787179 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:46231, user_credentials={real_user=slave}} blocked reactor thread for 59642.3us
I20251025 07:52:32.789644 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 050a5d3b5f934dd19bf2e221f2fe0780 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:32.790669 1569 raft_consensus.cc:493] T 79bc2678df8042e3994b0471afa2c304 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:32.790619 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2784bbfb97d5466f9ec856add7d0b29a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:32.790838 1569 raft_consensus.cc:515] T 79bc2678df8042e3994b0471afa2c304 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:32.806761 1569 leader_election.cc:290] T 79bc2678df8042e3994b0471afa2c304 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:32.807456 1170 ts_tablet_manager.cc:1950] T 90764275825c4ffb8712cb883d37a551 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:32.807307 1017 tablet_service.cc:1505] Processing CreateTablet for tablet 0540d6a10cf04fa89756623e9aaaeb68 (DEFAULT_TABLE table=default.Table_f16b7a3b7d91423f8502d73e4234ab4a [id=cc7db0a7eaea4caa8c17281a8079c372]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:32.810165 752 consensus_meta.cc:287] T 243b8828fc344bacab46d7bab6bd2dec P 23eeffd3de254ce18db5febfde971c6b: Time spent flushing consensus metadata: real 0.502s user 0.003s sys 0.001s
W20251025 07:52:32.812044 983 leader_election.cc:343] T 79bc2678df8042e3994b0471afa2c304 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 79bc2678df8042e3994b0471afa2c304
W20251025 07:52:32.812143 749 consensus_meta.cc:287] T 792f32cbd66449ca98813c669b2a261d P 23eeffd3de254ce18db5febfde971c6b: Time spent flushing consensus metadata: real 0.603s user 0.000s sys 0.003s
I20251025 07:52:32.813513 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 90764275825c4ffb8712cb883d37a551 (table default.Table_072e0092adb64ef0ac6834dc18688ccf [id=1d4cbe4a4f38445297ceeebb8ffdaa8e]) successfully deleted
W20251025 07:52:32.813742 850 leader_election.cc:343] T 2784bbfb97d5466f9ec856add7d0b29a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:32.813793 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 3b940fa5ae26427dafbcfd26372754b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b940fa5ae26427dafbcfd26372754b3 already in progress: creating tablet
W20251025 07:52:32.814798 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 26c849d376d143c6bef0c8e0e1d7d03a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 26c849d376d143c6bef0c8e0e1d7d03a already in progress: creating tablet
I20251025 07:52:32.815560 1166 tablet_bootstrap.cc:654] T fab67b56eaa44b54886c17181d8c52b2 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:32.815941 1170 tablet_replica.cc:333] T 79bc2678df8042e3994b0471afa2c304 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
W20251025 07:52:32.816870 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet f4578ce69a0d4322944e5ce7c8a59831 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4578ce69a0d4322944e5ce7c8a59831 already in progress: creating tablet
I20251025 07:52:32.817119 1170 raft_consensus.cc:2243] T 79bc2678df8042e3994b0471afa2c304 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:32.817309 1170 raft_consensus.cc:2272] T 79bc2678df8042e3994b0471afa2c304 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:32.817809 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet ce1ebb1ec14b4801930a72765bca1a7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce1ebb1ec14b4801930a72765bca1a7d already in progress: creating tablet
I20251025 07:52:32.817838 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffe7c8e1d6c44872b7b65e512764a56a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:32.818025 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 4f7c15529be54a11961f385a953b0b03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f7c15529be54a11961f385a953b0b03 already in progress: creating tablet
W20251025 07:52:32.818269 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet e4da89f863cc43f3bf6df908bb9dc6bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4da89f863cc43f3bf6df908bb9dc6bf already in progress: creating tablet
W20251025 07:52:32.818467 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet d77704b4fe6c4456a2ba40ffec0c60fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d77704b4fe6c4456a2ba40ffec0c60fc already in progress: creating tablet
I20251025 07:52:32.741146 1599 raft_consensus.cc:515] T 627a10a0714f4be8bdbced503fd6388d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:32.823485 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627a10a0714f4be8bdbced503fd6388d" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:32.823766 1046 raft_consensus.cc:2393] T 627a10a0714f4be8bdbced503fd6388d P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 23eeffd3de254ce18db5febfde971c6b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:32.823695 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_07b5304bacea445c92e5dde5059008aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:32.824239 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07b5304bacea445c92e5dde5059008aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:32.830183 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0540d6a10cf04fa89756623e9aaaeb68. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.830885 1170 ts_tablet_manager.cc:1916] T 79bc2678df8042e3994b0471afa2c304 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:32.832594 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_17e5895bc86c441ba014d094c1bf3893"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:32.833164 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_17e5895bc86c441ba014d094c1bf3893 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:32.834750 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_d0a06b2f0df7446ead73934e6f603423" } new_table_name: "default.Table_a3f50508f4b047b6837c67248417b21c" modify_external_catalogs: true
I20251025 07:52:32.836273 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 90764275825c4ffb8712cb883d37a551 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:32.841033 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_5a8143ecfd74418f928ceb9ae9cd50ba" } modify_external_catalogs: true
I20251025 07:52:32.841266 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_5a8143ecfd74418f928ceb9ae9cd50ba" } modify_external_catalogs: true
I20251025 07:52:32.841379 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_d55201c0dabc4134b687cfbd805119e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:32.840989 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_68399b04a61f453da82d016afd85a32d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:32.841948 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d55201c0dabc4134b687cfbd805119e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:32.842068 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68399b04a61f453da82d016afd85a32d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:32.843353 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_cf3a2c0772674c2a8ff5fee551a93a0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:32.843428 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_dbe39538a228480a9ff9cd45159e39f7" } new_table_name: "default.Table_8b4581839f31463bb6b7b91a6cf918e8" modify_external_catalogs: true
W20251025 07:52:32.843858 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf3a2c0772674c2a8ff5fee551a93a0f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:32.844975 1170 ts_tablet_manager.cc:1929] T 79bc2678df8042e3994b0471afa2c304 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:32.845142 1170 log.cc:1199] T 79bc2678df8042e3994b0471afa2c304 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/79bc2678df8042e3994b0471afa2c304
I20251025 07:52:32.845180 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_ce27d8ed3ede46579f2ed3ae81536572" } modify_external_catalogs: true
I20251025 07:52:32.845331 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_ce27d8ed3ede46579f2ed3ae81536572" } modify_external_catalogs: true
I20251025 07:52:32.845675 1170 ts_tablet_manager.cc:1950] T 79bc2678df8042e3994b0471afa2c304 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:32.846992 1169 tablet_bootstrap.cc:492] T 8e6d76b1a97945cfb6c3055e1b622f8a P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:32.847173 1169 ts_tablet_manager.cc:1403] T 8e6d76b1a97945cfb6c3055e1b622f8a P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.288s user 0.033s sys 0.050s
I20251025 07:52:32.847958 1169 raft_consensus.cc:359] T 8e6d76b1a97945cfb6c3055e1b622f8a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:32.848162 1169 raft_consensus.cc:385] T 8e6d76b1a97945cfb6c3055e1b622f8a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:32.848251 1169 raft_consensus.cc:740] T 8e6d76b1a97945cfb6c3055e1b622f8a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:32.848522 1169 consensus_queue.cc:260] T 8e6d76b1a97945cfb6c3055e1b622f8a P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:32.849023 1169 ts_tablet_manager.cc:1434] T 8e6d76b1a97945cfb6c3055e1b622f8a P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:32.849339 1169 tablet_bootstrap.cc:492] T 08da9d2687704725aacd11cd92c6a9f9 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
W20251025 07:52:32.867053 748 consensus_meta.cc:287] T e4da89f863cc43f3bf6df908bb9dc6bf P 23eeffd3de254ce18db5febfde971c6b: Time spent flushing consensus metadata: real 0.524s user 0.005s sys 0.000s
I20251025 07:52:32.867020 1172 rpcz_store.cc:276] 1025 07:52:31.742028 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:31.751897 (+ 9869us) service_pool.cc:225] Handling call
1025 07:52:32.052597 (+300700us) ts_tablet_manager.cc:1151] Deleting tablet 47284f96928749e6a5201103a8ee5f9f
1025 07:52:32.052605 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:32.716360 (+663755us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:32.720182 (+ 3822us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:32.766166 (+ 45984us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":300580}
I20251025 07:52:32.867462 1172 tablet_replica.cc:333] T 85fba8b37f2c4540aa81701052bf222a P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:32.868268 1172 raft_consensus.cc:2243] T 85fba8b37f2c4540aa81701052bf222a P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:32.868558 1172 pending_rounds.cc:70] T 85fba8b37f2c4540aa81701052bf222a P 3aea2869ff814cc7b142012faa274382: Trying to abort 1 pending ops.
I20251025 07:52:32.868625 1561 raft_consensus.cc:2804] T 742b3ec3f6604647b9e397532f1673dd P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:32.869014 1172 pending_rounds.cc:77] T 85fba8b37f2c4540aa81701052bf222a P 3aea2869ff814cc7b142012faa274382: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214607365470502912 op_type: NO_OP noop_request { }
I20251025 07:52:32.869146 1172 raft_consensus.cc:2889] T 85fba8b37f2c4540aa81701052bf222a P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 07:52:32.869556 1172 raft_consensus.cc:2272] T 85fba8b37f2c4540aa81701052bf222a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:32.869923 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 79bc2678df8042e3994b0471afa2c304 (table default.Table_ca46a7a783e547998d1be8476801a9e0 [id=525e8a793cf14cd492ca2ee986231311]) successfully deleted
I20251025 07:52:32.870445 1172 ts_tablet_manager.cc:1916] T 85fba8b37f2c4540aa81701052bf222a P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:32.870818 1561 raft_consensus.cc:697] T 742b3ec3f6604647b9e397532f1673dd P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
W20251025 07:52:32.875300 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 716fb61312e94a4d9eb8ab8336ca140c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 716fb61312e94a4d9eb8ab8336ca140c already in progress: creating tablet
W20251025 07:52:32.876075 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 88eb51447e0f4080baa49dd902deb24f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88eb51447e0f4080baa49dd902deb24f already in progress: creating tablet
I20251025 07:52:32.876502 1169 tablet_bootstrap.cc:654] T 08da9d2687704725aacd11cd92c6a9f9 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:32.879148 1171 rpcz_store.cc:276] 1025 07:52:31.571121 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:31.571163 (+ 42us) service_pool.cc:225] Handling call
1025 07:52:31.695234 (+124071us) ts_tablet_manager.cc:1151] Deleting tablet 85e27fc52e9a4d62a8622645594f59c6
1025 07:52:31.695248 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:32.060880 (+365632us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:32.229713 (+168833us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:32.758115 (+528402us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":118696}
I20251025 07:52:32.879787 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 401) took 1190 ms. Trace:
W20251025 07:52:32.881379 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5361f08f50b745eb98bf02e979d7c71e consensus state has no leader
I20251025 07:52:32.881511 1171 rpcz_store.cc:276] 1025 07:52:31.689033 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:31.689101 (+ 68us) service_pool.cc:225] Handling call
1025 07:52:32.879637 (+1190536us) ts_tablet_manager.cc:1151] Deleting tablet f0bdaa8cfd764f1b83f47881353d26b6
1025 07:52:32.879647 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:32.879750 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1189644}
I20251025 07:52:32.881901 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 402) took 1183 ms. Trace:
I20251025 07:52:32.883479 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_4f9a077b76bf42979239aeb6710131ec" } new_table_name: "default.Table_76158f8cf82e4cdb9a90580e46aaea48" modify_external_catalogs: true
W20251025 07:52:32.883723 711 service_pool.cc:137] DeleteTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:48830 dropped due to backpressure: service queue is full with 50 items
I20251025 07:52:32.887677 746 tablet_service.cc:1552] Processing DeleteTablet for tablet d77704b4fe6c4456a2ba40ffec0c60fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:32.888376 1561 consensus_queue.cc:237] T 742b3ec3f6604647b9e397532f1673dd P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:32.894820 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 2b594d4718344aafbdaec34040695589 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b594d4718344aafbdaec34040695589 already in progress: creating tablet
W20251025 07:52:32.895223 847 leader_election.cc:343] T 08da9d2687704725aacd11cd92c6a9f9 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:32.901953 847 leader_election.cc:304] T 08da9d2687704725aacd11cd92c6a9f9 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:32.902660 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:46231, user_credentials={real_user=slave}} blocked reactor thread for 113894us
W20251025 07:52:32.902913 847 leader_election.cc:343] T fd9b4be139d347f4a6a60767029c342a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:32.903546 847 leader_election.cc:304] T fd9b4be139d347f4a6a60767029c342a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:32.905002 1167 tablet_bootstrap.cc:654] T cd23b8094d7e472eba6bd170248858a9 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:32.906419 1597 raft_consensus.cc:2749] T fd9b4be139d347f4a6a60767029c342a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:32.906773 1614 consensus_queue.cc:1048] T 8c6ff4c73e624af3bf2de36c4493b462 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20251025 07:52:32.907578 740 tablet_service.cc:1552] Processing DeleteTablet for tablet e4da89f863cc43f3bf6df908bb9dc6bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:48830
W20251025 07:52:32.907967 847 leader_election.cc:343] T f3c87a83082b421cb4be546ac4ce7ad8 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:32.908149 847 leader_election.cc:304] T f3c87a83082b421cb4be546ac4ce7ad8 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:32.908056 1171 rpcz_store.cc:276] 1025 07:52:31.698797 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:31.698874 (+ 77us) service_pool.cc:225] Handling call
1025 07:52:32.881764 (+1182890us) ts_tablet_manager.cc:1151] Deleting tablet f43ef51ee45d47a99ec732b49d50d3b4
1025 07:52:32.881773 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:32.881866 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1179782}
W20251025 07:52:32.908972 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 72618756f756484e895e885ef12545c8 consensus state has no leader
W20251025 07:52:32.909468 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 70fe9aa2f58d4abea3b3b71066a29653 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70fe9aa2f58d4abea3b3b71066a29653 already in progress: creating tablet
I20251025 07:52:32.910243 1614 raft_consensus.cc:2749] T f3c87a83082b421cb4be546ac4ce7ad8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:32.911150 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 404) took 1193 ms. Trace:
I20251025 07:52:32.911142 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627a10a0714f4be8bdbced503fd6388d" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:32.911353 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet fd265cdbf6a8440393a2b5d8240d1872 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd265cdbf6a8440393a2b5d8240d1872 already in progress: creating tablet
I20251025 07:52:32.912894 755 tablet_service.cc:1505] Processing CreateTablet for tablet 72dc084f8efc45478771c247e07a16f4 (DEFAULT_TABLE table=default.Table_5a8143ecfd74418f928ceb9ae9cd50ba [id=23d0aa66d6f545d4b579cfd63b7e7143]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.913499 1171 rpcz_store.cc:276] 1025 07:52:31.715654 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:31.715947 (+ 293us) service_pool.cc:225] Handling call
1025 07:52:32.908332 (+1192385us) ts_tablet_manager.cc:1151] Deleting tablet 0f5f6ca6c3cd4d369690e425387a4344
1025 07:52:32.908343 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:32.909038 (+ 695us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1190577}
I20251025 07:52:32.919888 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72dc084f8efc45478771c247e07a16f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.921988 1599 leader_election.cc:290] T 627a10a0714f4be8bdbced503fd6388d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:32.922178 749 tablet_service.cc:1552] Processing DeleteTablet for tablet f4578ce69a0d4322944e5ce7c8a59831 with delete_type TABLET_DATA_DELETED (replaced by tablet 48b73e0c0dce4be686d05f991a21bd19) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:32.922418 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 405) took 1206 ms. Trace:
I20251025 07:52:32.922513 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 567a1f5c216a4a519249da66a2a03ea4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:32.922539 1171 rpcz_store.cc:276] 1025 07:52:31.715731 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:31.716131 (+ 400us) service_pool.cc:225] Handling call
1025 07:52:32.922264 (+1206133us) ts_tablet_manager.cc:1151] Deleting tablet 34315f64b89148efbf1f20a43d5c5efb
1025 07:52:32.922273 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:32.922382 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1158056}
I20251025 07:52:32.922744 747 tablet_service.cc:1505] Processing CreateTablet for tablet 48b73e0c0dce4be686d05f991a21bd19 (DEFAULT_TABLE table=default.Table_3bd421885a1b4e06b11d12780c34cad4 [id=4d9e99d6474b4d00bc66e26a33694e69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.922894 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 403) took 1207 ms. Trace:
I20251025 07:52:32.922999 1171 rpcz_store.cc:276] 1025 07:52:31.715523 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:31.717060 (+ 1537us) service_pool.cc:225] Handling call
1025 07:52:32.922778 (+1205718us) ts_tablet_manager.cc:1151] Deleting tablet 1912e59548874e0e8e989050ec0064c5
1025 07:52:32.922785 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:32.922860 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1154482}
I20251025 07:52:32.923115 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48b73e0c0dce4be686d05f991a21bd19. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.923681 1171 tablet_replica.cc:333] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:32.926209 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 1aeaad5c5d01478bb997254ac372d9a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:32.926626 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 742b3ec3f6604647b9e397532f1673dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:32.926869 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 26c849d376d143c6bef0c8e0e1d7d03a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:32.927804 745 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 410) took 1026 ms. Trace:
I20251025 07:52:32.927930 745 rpcz_store.cc:276] 1025 07:52:31.836593 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:31.837439 (+ 846us) service_pool.cc:225] Handling call
1025 07:52:32.031896 (+194457us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 07:52:32.031918 (+ 22us) ts_tablet_manager.cc:682] Creating new metadata...
1025 07:52:32.787128 (+755210us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:32.863300 (+ 76172us) tablet_replica.cc:206] Creating consensus instance
1025 07:52:32.863476 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 07:52:32.928397 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f7c15529be54a11961f385a953b0b03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:48830
W20251025 07:52:32.928532 1441 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.604s user 0.007s sys 0.006s
I20251025 07:52:32.928608 1441 raft_consensus.cc:697] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
W20251025 07:52:32.928706 744 consensus_meta.cc:287] T 4f7c15529be54a11961f385a953b0b03 P 23eeffd3de254ce18db5febfde971c6b: Time spent flushing consensus metadata: real 0.655s user 0.004s sys 0.000s
I20251025 07:52:32.928932 1441 consensus_queue.cc:237] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:32.928429 737 tablet_service.cc:1552] Processing DeleteTablet for tablet ce1ebb1ec14b4801930a72765bca1a7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:32.932610 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_e8d761a95cdf469b9cba89359c58a5a4" } modify_external_catalogs: true
I20251025 07:52:32.932821 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_e8d761a95cdf469b9cba89359c58a5a4" } modify_external_catalogs: true
I20251025 07:52:32.933610 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_9f52d94037424befbdea5c711068396a" } modify_external_catalogs: true
I20251025 07:52:32.933774 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_9f52d94037424befbdea5c711068396a" } modify_external_catalogs: true
I20251025 07:52:32.936123 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_d9880621c8ca4025b89b6c1ec47f5cb4" } modify_external_catalogs: true
I20251025 07:52:32.936290 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_d9880621c8ca4025b89b6c1ec47f5cb4" } modify_external_catalogs: true
I20251025 07:52:32.944295 1021 tablet_service.cc:1505] Processing CreateTablet for tablet a6587f4be6af44dd944e6b6f9599dfda (DEFAULT_TABLE table=default.Table_1431c97720de4d5a9a92bd2d574eb904 [id=484c28bccff0448bbca48cb7eaaa55df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.944532 1016 tablet_service.cc:1505] Processing CreateTablet for tablet 830fdbc6812e416690d0db42a73af503 (DEFAULT_TABLE table=default.Table_ab4b1a5a7abb46fd978c91bb6d74840e [id=fb21a45b86ab446e8c3dae4512a9598f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.944792 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6587f4be6af44dd944e6b6f9599dfda. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.945025 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 830fdbc6812e416690d0db42a73af503. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.947577 1218 raft_consensus.cc:2749] T 08da9d2687704725aacd11cd92c6a9f9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:32.948055 525 master_service.cc:954] ReplaceTablet: received request to replace tablet 9d6a9937d10a4cc2813671dd1bfa1664 from {username='slave'} at 127.0.0.1:45978
W20251025 07:52:32.949303 710 leader_election.cc:343] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 90764275825c4ffb8712cb883d37a551
I20251025 07:52:32.949488 511 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 47284f96928749e6a5201103a8ee5f9f (table default.Table_373f40b4ac044088a071a6d26d7a6da0 [id=05672e3da0704022b93cc96d61278433]) successfully deleted
W20251025 07:52:32.949839 710 leader_election.cc:343] T 567a1f5c216a4a519249da66a2a03ea4 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 567a1f5c216a4a519249da66a2a03ea4
I20251025 07:52:32.950002 710 leader_election.cc:304] T 567a1f5c216a4a519249da66a2a03ea4 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:32.951045 1599 raft_consensus.cc:2749] T 567a1f5c216a4a519249da66a2a03ea4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:32.952188 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 36262ab8c6964dd684c01b4db6219a4f consensus state has no leader
I20251025 07:52:32.952140 1614 raft_consensus.cc:493] T 5cba9c05dd424cc788febc614a34819a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:32.952347 1614 raft_consensus.cc:515] T 5cba9c05dd424cc788febc614a34819a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:32.953151 1614 leader_election.cc:290] T 5cba9c05dd424cc788febc614a34819a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:32.953641 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5cba9c05dd424cc788febc614a34819a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:32.953876 1048 raft_consensus.cc:2468] T 5cba9c05dd424cc788febc614a34819a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
W20251025 07:52:32.954121 848 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251025 07:52:32.954610 850 leader_election.cc:304] T 5cba9c05dd424cc788febc614a34819a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:32.955935 1614 raft_consensus.cc:2804] T 5cba9c05dd424cc788febc614a34819a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:32.956053 1614 raft_consensus.cc:493] T 5cba9c05dd424cc788febc614a34819a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:32.956137 1614 raft_consensus.cc:3060] T 5cba9c05dd424cc788febc614a34819a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:32.957062 1218 consensus_queue.cc:1048] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:32.957643 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 72dc084f8efc45478771c247e07a16f4
I20251025 07:52:32.959583 1614 raft_consensus.cc:515] T 5cba9c05dd424cc788febc614a34819a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:32.960225 1614 leader_election.cc:290] T 5cba9c05dd424cc788febc614a34819a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:32.960635 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5cba9c05dd424cc788febc614a34819a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:32.960834 1047 raft_consensus.cc:3060] T 5cba9c05dd424cc788febc614a34819a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:32.962836 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 2e098c9c4613459295b6043048cba587
W20251025 07:52:32.965199 513 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 1b5ac7e845a24b2eb5b2429d47361232 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b5ac7e845a24b2eb5b2429d47361232 already in progress: creating tablet
I20251025 07:52:32.965986 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 26c849d376d143c6bef0c8e0e1d7d03a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:32.966301 882 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f7c15529be54a11961f385a953b0b03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:32.966419 890 tablet_service.cc:1552] Processing DeleteTablet for tablet ce1ebb1ec14b4801930a72765bca1a7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:32.966547 893 tablet_service.cc:1552] Processing DeleteTablet for tablet e4da89f863cc43f3bf6df908bb9dc6bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:32.966624 890 tablet_service.cc:1552] Processing DeleteTablet for tablet d77704b4fe6c4456a2ba40ffec0c60fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:32.967128 890 tablet_service.cc:1505] Processing CreateTablet for tablet 5fc092c2bb2d41ac8c5b4fc07460c8ec (DEFAULT_TABLE table=default.Table_974f1ee455d7488593e5c496cc012099 [id=630c274339664b9db18ab4ef7c29c16f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.967404 885 tablet_service.cc:1505] Processing CreateTablet for tablet 22f7db660616457b83b24e947a0b0bae (DEFAULT_TABLE table=default.Table_3d163f45f94f450ebf0447897bdf41a6 [id=bb316a7e000748ca902a9bfb494802d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.967718 878 tablet_service.cc:1505] Processing CreateTablet for tablet aea8fcf0154a4bd4a05b19e27ab78b79 (DEFAULT_TABLE table=default.Table_c26454d3c928434ab0dae167abcc2ff2 [id=4cd0c9a010af47939a90da1132b9e6be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.967943 887 tablet_service.cc:1505] Processing CreateTablet for tablet ab68e5a89b234064b27aa7cae9467e14 (DEFAULT_TABLE table=default.Table_4b1446ea61a9442fb3e397dfe2c5fb03 [id=e9652929eaab46cab42bf8d6896577ed]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:32.968220 710 leader_election.cc:343] T 1aeaad5c5d01478bb997254ac372d9a6 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 1aeaad5c5d01478bb997254ac372d9a6
I20251025 07:52:32.967128 893 tablet_service.cc:1505] Processing CreateTablet for tablet 32a143a715cb485cb4501f4e28c68252 (DEFAULT_TABLE table=default.Table_e386bc02f89041418ff3ddf8ac224eea [id=e800c96844e843b7a6fd3145907e5503]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.969009 891 tablet_service.cc:1505] Processing CreateTablet for tablet 0582ee42ab564c07ae3facfc5935617c (DEFAULT_TABLE table=default.Table_6aaf0d6ed7554bd4afa602cdca717ce2 [id=f32a2592b45047bda447d79e36427e31]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.969673 892 tablet_service.cc:1505] Processing CreateTablet for tablet d40519ed9ee746348b17be2786d60e95 (DEFAULT_TABLE table=default.Table_7d2010ece331457fa18a0f13ac5cbdfa [id=c0e60203ceb3445ca63dc5f8f81c37b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.969863 882 tablet_service.cc:1505] Processing CreateTablet for tablet 9903e80a26fd469a8744dc855534ba75 (DEFAULT_TABLE table=default.Table_3d4495f2005e498a928c68b14b8017a3 [id=f6e8d725be3e47d3a3f682510f90b236]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.904811 711 service_pool.cc:138] DeleteTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:48830 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 455)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 456)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 457)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 458)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 459)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 460)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 461)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 462)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 463)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 464)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 465)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 466)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 467)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 468)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 469)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 470)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 471)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 472)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 473)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 474)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 475)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 476)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 477)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 478)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 479)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 480)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 481)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 482)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 483)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 484)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 485)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 486)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 487)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 488)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 489)
I20251025 07:52:32.972023 738 tablet_service.cc:1505] Processing CreateTablet for tablet 66134661be834f7b811d7927ab629812 (DEFAULT_TABLE table=default.Table_590ce35533c7442390ce8991bcfa80fb [id=7fd21f23b9ca483b8044392285400801]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.972316 742 tablet_service.cc:1505] Processing CreateTablet for tablet 0582ee42ab564c07ae3facfc5935617c (DEFAULT_TABLE table=default.Table_6aaf0d6ed7554bd4afa602cdca717ce2 [id=f32a2592b45047bda447d79e36427e31]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.972023 737 tablet_service.cc:1505] Processing CreateTablet for tablet 9903e80a26fd469a8744dc855534ba75 (DEFAULT_TABLE table=default.Table_3d4495f2005e498a928c68b14b8017a3 [id=f6e8d725be3e47d3a3f682510f90b236]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.974387 748 tablet_service.cc:1505] Processing CreateTablet for tablet ee1c8ea694284d80a30f9c05be4cbf49 (DEFAULT_TABLE table=default.Table_e8d761a95cdf469b9cba89359c58a5a4 [id=b369bec99e4d48c1847a2a327b1d6ac8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.974568 750 tablet_service.cc:1505] Processing CreateTablet for tablet dbb254115a574bf4976aff0ea56f3bd9 (DEFAULT_TABLE table=default.Table_a55d3aa69b804240b042e7b94ca493d5 [id=83841f9dd7b34f27a36830af24022ab9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.974387 749 tablet_service.cc:1505] Processing CreateTablet for tablet d40519ed9ee746348b17be2786d60e95 (DEFAULT_TABLE table=default.Table_7d2010ece331457fa18a0f13ac5cbdfa [id=c0e60203ceb3445ca63dc5f8f81c37b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.974846 743 tablet_service.cc:1505] Processing CreateTablet for tablet aea8fcf0154a4bd4a05b19e27ab78b79 (DEFAULT_TABLE table=default.Table_c26454d3c928434ab0dae167abcc2ff2 [id=4cd0c9a010af47939a90da1132b9e6be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.975160 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66134661be834f7b811d7927ab629812. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.975823 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9903e80a26fd469a8744dc855534ba75. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.976153 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbb254115a574bf4976aff0ea56f3bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.976444 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_2d41b478c59b488bb7aadc3ce393c24c" } new_table_name: "default.Table_2b9b772d4c234ac286bef3658482e490" modify_external_catalogs: true
I20251025 07:52:32.976433 741 tablet_service.cc:1505] Processing CreateTablet for tablet 5fc092c2bb2d41ac8c5b4fc07460c8ec (DEFAULT_TABLE table=default.Table_974f1ee455d7488593e5c496cc012099 [id=630c274339664b9db18ab4ef7c29c16f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.976922 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0582ee42ab564c07ae3facfc5935617c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:32.977236 745 tablet_service.cc:1505] Processing CreateTablet for tablet ab68e5a89b234064b27aa7cae9467e14 (DEFAULT_TABLE table=default.Table_4b1446ea61a9442fb3e397dfe2c5fb03 [id=e9652929eaab46cab42bf8d6896577ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.978927 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_ab87db32c7024d709eee5476d53f17ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:32.979519 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab87db32c7024d709eee5476d53f17ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:32.971719 752 tablet_service.cc:1505] Processing CreateTablet for tablet 3aeaa0e9dc594e21a9ddd923e314017d (DEFAULT_TABLE table=default.Table_51f2b43545db41029568a0321a37e139 [id=5a499fdacb8047dc9ce9bed92d7dd233]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.979804 744 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 408) took 1144 ms. Trace:
I20251025 07:52:32.942428 746 tablet_service.cc:1505] Processing CreateTablet for tablet 32a143a715cb485cb4501f4e28c68252 (DEFAULT_TABLE table=default.Table_e386bc02f89041418ff3ddf8ac224eea [id=e800c96844e843b7a6fd3145907e5503]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.981411 710 leader_election.cc:304] T 1aeaad5c5d01478bb997254ac372d9a6 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:32.981779 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_04bf0d2b2a1d4492b2ec0bc217cd2edf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:32.981963 710 leader_election.cc:304] T 627a10a0714f4be8bdbced503fd6388d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:32.982259 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_04bf0d2b2a1d4492b2ec0bc217cd2edf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:32.981117 740 tablet_service.cc:1505] Processing CreateTablet for tablet 22f7db660616457b83b24e947a0b0bae (DEFAULT_TABLE table=default.Table_3d163f45f94f450ebf0447897bdf41a6 [id=bb316a7e000748ca902a9bfb494802d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.982854 744 rpcz_store.cc:276] 1025 07:52:31.835375 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:31.835584 (+ 209us) service_pool.cc:225] Handling call
1025 07:52:31.970655 (+135071us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 07:52:31.970674 (+ 19us) ts_tablet_manager.cc:682] Creating new metadata...
1025 07:52:32.127394 (+156720us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:32.928834 (+801440us) tablet_replica.cc:206] Creating consensus instance
1025 07:52:32.979763 (+ 50929us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 07:52:32.983410 1599 raft_consensus.cc:2749] T 1aeaad5c5d01478bb997254ac372d9a6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:32.983616 1604 raft_consensus.cc:3060] T 627a10a0714f4be8bdbced503fd6388d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:32.984385 510 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: DeleteTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:48830 dropped due to backpressure: service queue is full with 50 items [suppressed 6 similar messages]
I20251025 07:52:32.990278 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_3d4495f2005e498a928c68b14b8017a3" } new_table_name: "default.Table_3297a14e9a534d31bbe2239df6f6ee37" modify_external_catalogs: true
I20251025 07:52:32.990360 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_56a34b2e5e744f94a5a955e4a796498a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:32.990613 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_74277c3abac14953b2eb73c850e3e727"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:32.990924 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56a34b2e5e744f94a5a955e4a796498a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:32.991047 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74277c3abac14953b2eb73c850e3e727 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:32.997646 744 tablet_service.cc:1505] Processing CreateTablet for tablet f7011f1dcad74fe4ba0356c618890e06 (DEFAULT_TABLE table=default.Table_6f5fbeb82b024bfe95511811dcc51406 [id=9c2312d7095f407d9d30f224cdce66c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:32.964964 1010 tablet_service.cc:1552] Processing DeleteTablet for tablet 72dc084f8efc45478771c247e07a16f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:32.965040 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e098c9c4613459295b6043048cba587 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:47408
W20251025 07:52:32.983887 982 leader_election.cc:343] T 7e869c65771f4339a546dfeebb227c62 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:32.964413 1170 tablet_replica.cc:333] T 5cba9c05dd424cc788febc614a34819a P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:32.998850 1569 raft_consensus.cc:493] T 8e6d76b1a97945cfb6c3055e1b622f8a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 07:52:32.998896 982 leader_election.cc:343] T 7e869c65771f4339a546dfeebb227c62 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:32.999382 1569 raft_consensus.cc:515] T 8e6d76b1a97945cfb6c3055e1b622f8a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.000077 1569 leader_election.cc:290] T 8e6d76b1a97945cfb6c3055e1b622f8a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:33.000737 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e6d76b1a97945cfb6c3055e1b622f8a" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:33.000986 913 raft_consensus.cc:2468] T 8e6d76b1a97945cfb6c3055e1b622f8a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:33.001407 983 leader_election.cc:304] T 8e6d76b1a97945cfb6c3055e1b622f8a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:33.001627 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee1c8ea694284d80a30f9c05be4cbf49. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.001770 1569 raft_consensus.cc:2804] T 8e6d76b1a97945cfb6c3055e1b622f8a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:33.001890 1569 raft_consensus.cc:493] T 8e6d76b1a97945cfb6c3055e1b622f8a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:33.001982 1569 raft_consensus.cc:3060] T 8e6d76b1a97945cfb6c3055e1b622f8a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:33.003644 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7011f1dcad74fe4ba0356c618890e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.004009 756 tablet_service.cc:1505] Processing CreateTablet for tablet 8707b93d491e4e4386d148087a2f0870 (DEFAULT_TABLE table=default.Table_fd5720d69b084df1b36485b7890ac6c2 [id=59840a567e5244e8a9024cbad42471e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.004226 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fc092c2bb2d41ac8c5b4fc07460c8ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.004568 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d40519ed9ee746348b17be2786d60e95. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.004822 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aea8fcf0154a4bd4a05b19e27ab78b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.005084 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac0ea298bc114c56a8093e12def21ce1" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:33.005417 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e52928580c4762b0ae7152490ef955" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:33.006500 753 tablet_service.cc:1505] Processing CreateTablet for tablet 1b22bd172e0e4f86896e9ecdef666b42 (DEFAULT_TABLE table=default.Table_9bde97409538452a99917ba1e4ed04b3 [id=eb909ab8da1d45e28008f447066bea71]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.007716 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0582ee42ab564c07ae3facfc5935617c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.008294 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3aeaa0e9dc594e21a9ddd923e314017d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.008987 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aea8fcf0154a4bd4a05b19e27ab78b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.010654 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab68e5a89b234064b27aa7cae9467e14. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.010748 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_58bd2fb570eb4cfebf3ce98477e23525"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:33.011350 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58bd2fb570eb4cfebf3ce98477e23525 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:33.014453 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32a143a715cb485cb4501f4e28c68252. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.014822 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22f7db660616457b83b24e947a0b0bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.016790 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab68e5a89b234064b27aa7cae9467e14. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.018257 888 tablet_service.cc:1505] Processing CreateTablet for tablet 8707b93d491e4e4386d148087a2f0870 (DEFAULT_TABLE table=default.Table_fd5720d69b084df1b36485b7890ac6c2 [id=59840a567e5244e8a9024cbad42471e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.018770 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8707b93d491e4e4386d148087a2f0870. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.022316 739 tablet_service.cc:1505] Processing CreateTablet for tablet cc407c7eb12e4eadb41fb21385611c94 (DEFAULT_TABLE table=default.Table_50fe33e9d42343ff9194780a28b1b1b7 [id=01b6f41ccb23455b87ed38ed96184aa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.022356 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d40519ed9ee746348b17be2786d60e95. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.023432 1604 raft_consensus.cc:2749] T 627a10a0714f4be8bdbced503fd6388d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:33.023837 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32a143a715cb485cb4501f4e28c68252. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.027887 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9903e80a26fd469a8744dc855534ba75. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.033258 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet ee1c8ea694284d80a30f9c05be4cbf49
I20251025 07:52:33.034996 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet ee1c8ea694284d80a30f9c05be4cbf49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:33.035866 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet c11f674385184ae6b235f1db8bf1fb91
W20251025 07:52:33.036923 510 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:46231, user_credentials={real_user=slave}} blocked reactor thread for 51899.8us
I20251025 07:52:33.037230 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet c11f674385184ae6b235f1db8bf1fb91 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:47408
I20251025 07:52:33.038792 510 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 85e27fc52e9a4d62a8622645594f59c6 (table default.Table_4bafafcc9e8641b687d438725186f639 [id=73f69c9574ee4019ada5af76bd902107]) successfully deleted
W20251025 07:52:33.039073 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet f0bdaa8cfd764f1b83f47881353d26b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0bdaa8cfd764f1b83f47881353d26b6 already in progress: creating tablet
W20251025 07:52:33.039647 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet f43ef51ee45d47a99ec732b49d50d3b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f43ef51ee45d47a99ec732b49d50d3b4 already in progress: creating tablet
W20251025 07:52:33.040100 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 0f5f6ca6c3cd4d369690e425387a4344 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f5f6ca6c3cd4d369690e425387a4344 already in progress: creating tablet
W20251025 07:52:33.040324 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 34315f64b89148efbf1f20a43d5c5efb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34315f64b89148efbf1f20a43d5c5efb already in progress: creating tablet
W20251025 07:52:33.043300 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ef9a2c9f2ccf432ab835396020872208 consensus state has no leader
I20251025 07:52:33.043192 886 tablet_service.cc:1505] Processing CreateTablet for tablet 1b22bd172e0e4f86896e9ecdef666b42 (DEFAULT_TABLE table=default.Table_9bde97409538452a99917ba1e4ed04b3 [id=eb909ab8da1d45e28008f447066bea71]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.020196 1172 ts_tablet_manager.cc:1929] T 85fba8b37f2c4540aa81701052bf222a P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:33.044214 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet ffe7c8e1d6c44872b7b65e512764a56a
I20251025 07:52:33.044934 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet ffe7c8e1d6c44872b7b65e512764a56a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:47408
W20251025 07:52:33.040560 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 1912e59548874e0e8e989050ec0064c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1912e59548874e0e8e989050ec0064c5 already in progress: creating tablet
I20251025 07:52:33.054627 1047 raft_consensus.cc:2468] T 5cba9c05dd424cc788febc614a34819a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
W20251025 07:52:33.054075 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet dda0b26e4f1a44928bc4b84f333901c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dda0b26e4f1a44928bc4b84f333901c0 already in progress: creating tablet
I20251025 07:52:33.057253 1569 raft_consensus.cc:515] T 8e6d76b1a97945cfb6c3055e1b622f8a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:33.056867 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 95efd553493e4966bc48a01cac6e1e10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95efd553493e4966bc48a01cac6e1e10 already in progress: creating tablet
W20251025 07:52:33.059582 510 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 3b940fa5ae26427dafbcfd26372754b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b940fa5ae26427dafbcfd26372754b3 already in progress: creating tablet
I20251025 07:52:33.066020 878 tablet_service.cc:1505] Processing CreateTablet for tablet 66134661be834f7b811d7927ab629812 (DEFAULT_TABLE table=default.Table_590ce35533c7442390ce8991bcfa80fb [id=7fd21f23b9ca483b8044392285400801]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.067587 1172 log.cc:1199] T 85fba8b37f2c4540aa81701052bf222a P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/85fba8b37f2c4540aa81701052bf222a
W20251025 07:52:33.072671 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9903e80a26fd469a8744dc855534ba75 consensus state has no leader
I20251025 07:52:33.078179 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e6d76b1a97945cfb6c3055e1b622f8a" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:33.078401 913 raft_consensus.cc:3060] T 8e6d76b1a97945cfb6c3055e1b622f8a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:33.068117 850 leader_election.cc:304] T 5cba9c05dd424cc788febc614a34819a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:33.080945 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_8d0e78b0bb8d433f9ed8e49d24d7e95d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:33.081193 1561 raft_consensus.cc:2804] T 5cba9c05dd424cc788febc614a34819a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:33.081472 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b22bd172e0e4f86896e9ecdef666b42. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:33.081564 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d0e78b0bb8d433f9ed8e49d24d7e95d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:33.082255 1561 raft_consensus.cc:697] T 5cba9c05dd424cc788febc614a34819a P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:33.082554 1561 consensus_queue.cc:237] T 5cba9c05dd424cc788febc614a34819a P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.087699 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_6eb139482bc940989a606a0d8745d54f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:33.088311 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6eb139482bc940989a606a0d8745d54f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:33.089958 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_04bf0d2b2a1d4492b2ec0bc217cd2edf" } new_table_name: "default.Table_d5c65dce8b1b4214ba519d964012bfc0" modify_external_catalogs: true
I20251025 07:52:33.093277 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66134661be834f7b811d7927ab629812. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.094836 894 tablet_service.cc:1505] Processing CreateTablet for tablet ee1c8ea694284d80a30f9c05be4cbf49 (DEFAULT_TABLE table=default.Table_e8d761a95cdf469b9cba89359c58a5a4 [id=b369bec99e4d48c1847a2a327b1d6ac8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.096783 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_d5ae5bc2cb804015ba50415cd3f62847"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:33.097348 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5ae5bc2cb804015ba50415cd3f62847 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:33.100615 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_a0665a4cb38b40109df3991cb2f71ed6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:33.101058 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2784bbfb97d5466f9ec856add7d0b29a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
W20251025 07:52:33.101166 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0665a4cb38b40109df3991cb2f71ed6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:33.101190 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8707b93d491e4e4386d148087a2f0870. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.102216 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 88eb51447e0f4080baa49dd902deb24f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:33.102461 751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 437) took 1007 ms. Trace:
I20251025 07:52:33.102469 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b22bd172e0e4f86896e9ecdef666b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.102577 751 rpcz_store.cc:276] 1025 07:52:32.020768 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:32.301563 (+280795us) service_pool.cc:225] Handling call
1025 07:52:32.673665 (+372102us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 07:52:32.673742 (+ 77us) ts_tablet_manager.cc:682] Creating new metadata...
1025 07:52:32.882540 (+208798us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:33.027191 (+144651us) tablet_replica.cc:206] Creating consensus instance
1025 07:52:33.028043 (+ 852us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 07:52:33.102751 875 tablet_service.cc:1505] Processing CreateTablet for tablet f7011f1dcad74fe4ba0356c618890e06 (DEFAULT_TABLE table=default.Table_6f5fbeb82b024bfe95511811dcc51406 [id=9c2312d7095f407d9d30f224cdce66c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.103157 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee1c8ea694284d80a30f9c05be4cbf49. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.102535 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 716fb61312e94a4d9eb8ab8336ca140c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:33.103839 751 tablet_service.cc:1505] Processing CreateTablet for tablet b71bdfd09a27449d98509bb220680147 (DEFAULT_TABLE table=default.Table_f24cd210ae3342e68c69aa6670e3e4b0 [id=22cab1e94aa544fa828350f9a70f8555]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.105039 747 tablet_service.cc:1505] Processing CreateTablet for tablet 3c4970c8a491460e8b4b1c29fde83e4c (DEFAULT_TABLE table=default.Table_915572e49894471d8933816077123d6c [id=e372ca4384644779a6b37f389f549dec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.109691 754 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 429) took 1089 ms. Trace:
I20251025 07:52:33.109820 754 rpcz_store.cc:276] 1025 07:52:32.020200 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:32.023836 (+ 3636us) service_pool.cc:225] Handling call
1025 07:52:32.072674 (+ 48838us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 07:52:32.072706 (+ 32us) ts_tablet_manager.cc:682] Creating new metadata...
1025 07:52:32.776391 (+703685us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:33.109465 (+333074us) tablet_replica.cc:206] Creating consensus instance
1025 07:52:33.109643 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 07:52:33.110142 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b594d4718344aafbdaec34040695589 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:33.110353 754 tablet_service.cc:1552] Processing DeleteTablet for tablet fd265cdbf6a8440393a2b5d8240d1872 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:33.110543 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 70fe9aa2f58d4abea3b3b71066a29653 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:33.110688 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_3822b091ab6541efb91252e18b0a2a8b" } new_table_name: "default.Table_a6c869949b744bc98efb735b0e73e7aa" modify_external_catalogs: true
I20251025 07:52:33.110935 754 tablet_service.cc:1505] Processing CreateTablet for tablet 0063a0776b2247afaa8be82f6eff26f1 (DEFAULT_TABLE table=default.Table_2c8bad53187a49858df0c2bde3485e61 [id=1d22c44df6d24fed9c7009b7c7bd5a70]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.114212 877 tablet_service.cc:1552] Processing DeleteTablet for tablet 88eb51447e0f4080baa49dd902deb24f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:33.114635 884 tablet_service.cc:1505] Processing CreateTablet for tablet cc407c7eb12e4eadb41fb21385611c94 (DEFAULT_TABLE table=default.Table_50fe33e9d42343ff9194780a28b1b1b7 [id=01b6f41ccb23455b87ed38ed96184aa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.117274 1172 ts_tablet_manager.cc:1950] T 85fba8b37f2c4540aa81701052bf222a P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:33.126996 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "default.Table_68c19f91d7094d5ab0ab4f77f74a937e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:33.127743 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68c19f91d7094d5ab0ab4f77f74a937e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:33.128696 748 tablet_service.cc:1505] Processing CreateTablet for tablet 421db850a64048218eecb9de65e2bdbf (DEFAULT_TABLE table=default.Table_157e52b1e07f4356afc68bb3ec5e4eb0 [id=a0facec9e7314c59b2175f0196869d73]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.134241 1166 tablet_bootstrap.cc:492] T fab67b56eaa44b54886c17181d8c52b2 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:33.134351 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7011f1dcad74fe4ba0356c618890e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.134523 887 tablet_service.cc:1505] Processing CreateTablet for tablet dbb254115a574bf4976aff0ea56f3bd9 (DEFAULT_TABLE table=default.Table_a55d3aa69b804240b042e7b94ca493d5 [id=83841f9dd7b34f27a36830af24022ab9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.135385 1569 leader_election.cc:290] T 8e6d76b1a97945cfb6c3055e1b622f8a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:33.139528 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22f7db660616457b83b24e947a0b0bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.146639 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e6d76b1a97945cfb6c3055e1b622f8a" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:33.150178 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b71bdfd09a27449d98509bb220680147. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.150921 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e6d76b1a97945cfb6c3055e1b622f8a" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:33.153759 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_05d7a0bdab8b438591303fb459a61a68" } new_table_name: "default.Table_350383f0e98947eba77186e74ad88781" modify_external_catalogs: true
W20251025 07:52:33.159221 982 leader_election.cc:343] T 79bc2678df8042e3994b0471afa2c304 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: 79bc2678df8042e3994b0471afa2c304
I20251025 07:52:33.159392 982 leader_election.cc:304] T 79bc2678df8042e3994b0471afa2c304 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
W20251025 07:52:33.159616 982 raft_consensus.cc:2686] T 79bc2678df8042e3994b0471afa2c304 P db42be7ad3c741a0aa427f27e245993b: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251025 07:52:33.160318 982 leader_election.cc:343] T 8e6d76b1a97945cfb6c3055e1b622f8a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:33.160521 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_192549a3e67840f3aefc31029cf2ff52" } modify_external_catalogs: true
W20251025 07:52:33.160631 982 leader_election.cc:343] T 8e6d76b1a97945cfb6c3055e1b622f8a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:33.160708 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_192549a3e67840f3aefc31029cf2ff52" } modify_external_catalogs: true
I20251025 07:52:33.160759 1170 raft_consensus.cc:2243] T 5cba9c05dd424cc788febc614a34819a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:33.161087 1170 raft_consensus.cc:2272] T 5cba9c05dd424cc788febc614a34819a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:33.161612 1170 ts_tablet_manager.cc:1916] T 5cba9c05dd424cc788febc614a34819a P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:33.162782 1166 ts_tablet_manager.cc:1403] T fab67b56eaa44b54886c17181d8c52b2 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.484s user 0.007s sys 0.005s
I20251025 07:52:33.163277 755 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 440) took 1139 ms. Trace:
I20251025 07:52:33.163416 755 rpcz_store.cc:276] 1025 07:52:32.026342 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:32.803879 (+777537us) service_pool.cc:225] Handling call
1025 07:52:32.919670 (+115791us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 07:52:32.919706 (+ 36us) ts_tablet_manager.cc:682] Creating new metadata...
1025 07:52:32.977610 (+ 57904us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:33.162096 (+184486us) tablet_replica.cc:206] Creating consensus instance
1025 07:52:33.162267 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 07:52:33.164899 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_8773c967a71644baba838946c154aee2" } modify_external_catalogs: true
I20251025 07:52:33.165115 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45978:
table { table_name: "default.Table_8773c967a71644baba838946c154aee2" } modify_external_catalogs: true
I20251025 07:52:33.168442 737 tablet_service.cc:1505] Processing CreateTablet for tablet 294d7d20c2f545b8813e632aa058a60c (DEFAULT_TABLE table=default.Table_4eaf06c0720744ff8b2fc8866cf05e2b [id=2b66e5f736de408bbd711b88361e1c7b]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:33.181645 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 872cb7512d104827b1f1f186d33a4a44 has no consensus state
I20251025 07:52:33.181509 755 tablet_service.cc:1505] Processing CreateTablet for tablet 7e18611cbc1348eca927162fec70e36f (DEFAULT_TABLE table=default.Table_d67fb372b12e40d082d119e53b9ebc3e [id=a4f956a8e567440cbb5ad5214a0aa10a]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:33.185878 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d46673a283fe42b78ddeb99fbd2678eb consensus state has no leader
I20251025 07:52:33.189271 521 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 8e6d76b1a97945cfb6c3055e1b622f8a (table default.Table_10e57b42f97544208675708b1e028846 [id=f5c1be203e2b4831a025871cf8a2ef17]). Expected version 2 got 0
I20251025 07:52:33.196610 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc407c7eb12e4eadb41fb21385611c94. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.208326 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c4970c8a491460e8b4b1c29fde83e4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.213244 743 tablet_service.cc:1505] Processing CreateTablet for tablet 19f1b56cdbd24ebe8688cc05fb795d07 (DEFAULT_TABLE table=default.Table_7955f4594e0e4f7c803f728a43c229ca [id=95040fbb66f94779b7721e2209d82a65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.218355 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskRBHlYt/build/asan/bin/kudu pid 492
I20251025 07:52:33.219985 742 tablet_service.cc:1505] Processing CreateTablet for tablet 0b47d1b0e6a342888135d7e4e5fa1f43 (DEFAULT_TABLE table=default.Table_c78d2d5f287d45ab9318f5c8a4c1e7ae [id=4d099e6134f548b49dec1897f6660560]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.220655 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fc092c2bb2d41ac8c5b4fc07460c8ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.220743 556 generic_service.cc:196] Checking for leaks (request via RPC)
I20251025 07:52:33.171854 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 716fb61312e94a4d9eb8ab8336ca140c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:33.172096 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc407c7eb12e4eadb41fb21385611c94. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.173970 913 raft_consensus.cc:2468] T 8e6d76b1a97945cfb6c3055e1b622f8a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:33.174314 890 tablet_service.cc:1505] Processing CreateTablet for tablet b71bdfd09a27449d98509bb220680147 (DEFAULT_TABLE table=default.Table_f24cd210ae3342e68c69aa6670e3e4b0 [id=22cab1e94aa544fa828350f9a70f8555]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.173766 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 403) took 1384 ms. Trace:
I20251025 07:52:33.187019 877 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b594d4718344aafbdaec34040695589 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:33.188894 881 tablet_service.cc:1552] Processing DeleteTablet for tablet fd265cdbf6a8440393a2b5d8240d1872 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:33.191200 894 tablet_service.cc:1505] Processing CreateTablet for tablet 3c4970c8a491460e8b4b1c29fde83e4c (DEFAULT_TABLE table=default.Table_915572e49894471d8933816077123d6c [id=e372ca4384644779a6b37f389f549dec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.190057 1629 tablet.cc:1722] T 3d5e0dd14e474fe09993a03851f8b1f0 P 3aea2869ff814cc7b142012faa274382: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251025 07:52:33.202024 880 tablet_service.cc:1552] Processing DeleteTablet for tablet 70fe9aa2f58d4abea3b3b71066a29653 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:33.223500 1167 tablet_bootstrap.cc:492] T cd23b8094d7e472eba6bd170248858a9 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:33.223565 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 294d7d20c2f545b8813e632aa058a60c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.224499 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b71bdfd09a27449d98509bb220680147. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.225374 750 tablet_service.cc:1505] Processing CreateTablet for tablet 60c60a9ee37f43978910bd067a098bb7 (DEFAULT_TABLE table=default.Table_173708dfd0274b34a7db6fa2b239a8b0 [id=6bc5ca21f8c54249a5150f0fea2686c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.228560 1172 rpcz_store.cc:276] 1025 07:52:31.742150 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:31.752039 (+ 9889us) service_pool.cc:225] Handling call
1025 07:52:32.867389 (+1115350us) ts_tablet_manager.cc:1151] Deleting tablet 85fba8b37f2c4540aa81701052bf222a
1025 07:52:32.867400 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:32.942110 (+ 74710us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:33.018416 (+ 76306us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:33.126903 (+108487us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1115217}
I20251025 07:52:33.229182 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 405) took 1477 ms. Trace:
I20251025 07:52:33.229336 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19f1b56cdbd24ebe8688cc05fb795d07. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.229362 1171 raft_consensus.cc:2243] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:33.229832 1171 pending_rounds.cc:70] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b: Trying to abort 1 pending ops.
I20251025 07:52:33.229918 1171 pending_rounds.cc:77] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214607371999559680 op_type: NO_OP noop_request { }
I20251025 07:52:33.230032 1171 raft_consensus.cc:2889] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 07:52:33.230170 1171 raft_consensus.cc:2272] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:33.227738 888 tablet_service.cc:1505] Processing CreateTablet for tablet 0063a0776b2247afaa8be82f6eff26f1 (DEFAULT_TABLE table=default.Table_2c8bad53187a49858df0c2bde3485e61 [id=1d22c44df6d24fed9c7009b7c7bd5a70]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.232973 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbb254115a574bf4976aff0ea56f3bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.237149 737 tablet_service.cc:1552] Processing DeleteTablet for tablet cc407c7eb12e4eadb41fb21385611c94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:33.237186 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Sending DeleteTablet for 3 replicas of tablet 627a10a0714f4be8bdbced503fd6388d
I20251025 07:52:33.241210 1167 ts_tablet_manager.cc:1403] T cd23b8094d7e472eba6bd170248858a9 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.544s user 0.016s sys 0.042s
I20251025 07:52:33.243593 1172 rpcz_store.cc:276] 1025 07:52:31.751296 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:31.761772 (+ 10476us) service_pool.cc:225] Handling call
1025 07:52:33.229043 (+1467271us) ts_tablet_manager.cc:1151] Deleting tablet 95efd553493e4966bc48a01cac6e1e10
1025 07:52:33.229054 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:33.229152 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1467051}
I20251025 07:52:33.243002 983 leader_election.cc:304] T 8e6d76b1a97945cfb6c3055e1b622f8a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b
I20251025 07:52:33.247236 1017 tablet_service.cc:1505] Processing CreateTablet for tablet b02f5c0ab0b24288bb50a3ba0e6bd78a (DEFAULT_TABLE table=default.Table_29aeaf089ca24270b3f21c2e08d61634 [id=3294d42240024d25ab2487e75640ddd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.250247 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 3e1ffd02aa384b73835b4361ae3d5467 (DEFAULT_TABLE table=default.Table_cf3a2c0772674c2a8ff5fee551a93a0f [id=1ffca9f550ce498296d664517e9fba86]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.251170 738 tablet_service.cc:1505] Processing CreateTablet for tablet bbe1d40d67594ad389ccc97bb9e2ac3b (DEFAULT_TABLE table=default.Table_3f291c48b0144f0a8ddb1b51ebbe31cd [id=98db26b2e7bf409dbfe8d201ae701df7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.237233 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 1b5ac7e845a24b2eb5b2429d47361232 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:33.253183 1630 tablet.cc:1722] T 3d5e0dd14e474fe09993a03851f8b1f0 P db42be7ad3c741a0aa427f27e245993b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251025 07:52:33.254010 1612 raft_consensus.cc:2804] T 8e6d76b1a97945cfb6c3055e1b622f8a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:33.254040 1169 tablet_bootstrap.cc:492] T 08da9d2687704725aacd11cd92c6a9f9 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:33.254663 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b02f5c0ab0b24288bb50a3ba0e6bd78a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.237416 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60c60a9ee37f43978910bd067a098bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.255924 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0063a0776b2247afaa8be82f6eff26f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.256877 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e18611cbc1348eca927162fec70e36f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.257273 737 tablet_service.cc:1505] Processing CreateTablet for tablet 0540d6a10cf04fa89756623e9aaaeb68 (DEFAULT_TABLE table=default.Table_f16b7a3b7d91423f8502d73e4234ab4a [id=cc7db0a7eaea4caa8c17281a8079c372]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.257678 891 tablet_service.cc:1505] Processing CreateTablet for tablet 421db850a64048218eecb9de65e2bdbf (DEFAULT_TABLE table=default.Table_157e52b1e07f4356afc68bb3ec5e4eb0 [id=a0facec9e7314c59b2175f0196869d73]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.257983 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e1ffd02aa384b73835b4361ae3d5467. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.256347 1010 tablet_service.cc:1505] Processing CreateTablet for tablet 2e87a81c87ec4353a8ad920307809390 (DEFAULT_TABLE table=default.Table_b8551dc7ed8c48228c5d1fec6a1e8b0c [id=52d442791ab94896b1e377af37ab6154]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.259485 1018 tablet_service.cc:1505] Processing CreateTablet for tablet de8da6abeda74a2a96d94ce5ac8fb7e5 (DEFAULT_TABLE table=default.Table_17e5895bc86c441ba014d094c1bf3893 [id=c8c37f4cecb94242b3cea6e61a2c1cc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.259987 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e87a81c87ec4353a8ad920307809390. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.254616 1612 raft_consensus.cc:697] T 8e6d76b1a97945cfb6c3055e1b622f8a P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:33.263231 1169 ts_tablet_manager.cc:1403] T 08da9d2687704725aacd11cd92c6a9f9 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.406s user 0.000s sys 0.013s
I20251025 07:52:33.263303 1167 raft_consensus.cc:359] T cd23b8094d7e472eba6bd170248858a9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.263617 880 tablet_service.cc:1505] Processing CreateTablet for tablet 0b47d1b0e6a342888135d7e4e5fa1f43 (DEFAULT_TABLE table=default.Table_c78d2d5f287d45ab9318f5c8a4c1e7ae [id=4d099e6134f548b49dec1897f6660560]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.264596 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c4970c8a491460e8b4b1c29fde83e4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.265117 881 tablet_service.cc:1505] Processing CreateTablet for tablet 7e18611cbc1348eca927162fec70e36f (DEFAULT_TABLE table=default.Table_d67fb372b12e40d082d119e53b9ebc3e [id=a4f956a8e567440cbb5ad5214a0aa10a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.266911 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de8da6abeda74a2a96d94ce5ac8fb7e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.269472 1612 consensus_queue.cc:237] T 8e6d76b1a97945cfb6c3055e1b622f8a P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.274647 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 421db850a64048218eecb9de65e2bdbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.274698 1011 tablet_service.cc:1505] Processing CreateTablet for tablet 165575a8b5b54d3c9dbcc1cc93044623 (DEFAULT_TABLE table=default.Table_d55201c0dabc4134b687cfbd805119e3 [id=3f7923f091ac46e0bc20bf7401b5458d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.274963 749 tablet_service.cc:1505] Processing CreateTablet for tablet 698e9f8de7db4516a811c9eb9143a268 (DEFAULT_TABLE table=default.Table_48712d8a17e04509888897a9cf2b4e30 [id=488b550bbdef476c95e8bad42bd4b4dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.279999 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0540d6a10cf04fa89756623e9aaaeb68. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.280515 1167 raft_consensus.cc:385] T cd23b8094d7e472eba6bd170248858a9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:33.280623 1167 raft_consensus.cc:740] T cd23b8094d7e472eba6bd170248858a9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:33.281361 743 tablet_service.cc:1505] Processing CreateTablet for tablet ffe7c8e1d6c44872b7b65e512764a56a (DEFAULT_TABLE table=default.Table_d9880621c8ca4025b89b6c1ec47f5cb4 [id=be817d9282bd4fbf84adf11e25f06b5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.292734 1167 consensus_queue.cc:260] T cd23b8094d7e472eba6bd170248858a9 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.294777 756 tablet_service.cc:1505] Processing CreateTablet for tablet 6fb1a1d38fb94f07b79ac4f8334fa719 (DEFAULT_TABLE table=default.Table_d49fa3011bf146b5a0dcb1c2d4557138 [id=fc927d99526b494db7c23ccf3e94fa02]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.295920 740 tablet_service.cc:1505] Processing CreateTablet for tablet 2c4789b9d89f42cb844e6cb595e687db (DEFAULT_TABLE table=default.Table_ddc8580fd1674f9ca61d528f22c51c94 [id=e33f00d4d5844ebba7ea3e200e154eeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.297348 1167 ts_tablet_manager.cc:1434] T cd23b8094d7e472eba6bd170248858a9 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.052s user 0.002s sys 0.000s
I20251025 07:52:33.297729 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 698e9f8de7db4516a811c9eb9143a268. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.299104 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fb1a1d38fb94f07b79ac4f8334fa719. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.299078 752 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 441) took 1270 ms. Trace:
I20251025 07:52:33.299197 752 rpcz_store.cc:276] 1025 07:52:32.026458 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:32.810557 (+784099us) service_pool.cc:225] Handling call
1025 07:52:32.979920 (+169363us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 07:52:32.979944 (+ 24us) ts_tablet_manager.cc:682] Creating new metadata...
1025 07:52:33.207819 (+227875us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:33.296812 (+ 88993us) tablet_replica.cc:206] Creating consensus instance
1025 07:52:33.296991 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2471}
I20251025 07:52:33.300371 752 tablet_service.cc:1552] Processing DeleteTablet for tablet af743ee344e543a0aa6da3cd7a30e467 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:33.302670 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c4789b9d89f42cb844e6cb595e687db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.306209 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 165575a8b5b54d3c9dbcc1cc93044623. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.309985 1170 ts_tablet_manager.cc:1929] T 5cba9c05dd424cc788febc614a34819a P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:33.310284 744 tablet_service.cc:1505] Processing CreateTablet for tablet a6587f4be6af44dd944e6b6f9599dfda (DEFAULT_TABLE table=default.Table_1431c97720de4d5a9a92bd2d574eb904 [id=484c28bccff0448bbca48cb7eaaa55df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.314059 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffe7c8e1d6c44872b7b65e512764a56a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.315609 746 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 453) took 1069 ms. Trace:
I20251025 07:52:33.317698 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 050a5d3b5f934dd19bf2e221f2fe0780 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:33.321628 1167 tablet_bootstrap.cc:492] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:33.323120 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6587f4be6af44dd944e6b6f9599dfda. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.321727 746 rpcz_store.cc:276] 1025 07:52:32.229984 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:32.888354 (+658370us) service_pool.cc:225] Handling call
1025 07:52:32.980096 (+ 91742us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 07:52:32.980115 (+ 19us) ts_tablet_manager.cc:682] Creating new metadata...
1025 07:52:33.224812 (+244697us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:33.294487 (+ 69675us) tablet_replica.cc:206] Creating consensus instance
1025 07:52:33.298929 (+ 4442us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4300}
I20251025 07:52:33.325762 892 tablet_service.cc:1505] Processing CreateTablet for tablet 60c60a9ee37f43978910bd067a098bb7 (DEFAULT_TABLE table=default.Table_173708dfd0274b34a7db6fa2b239a8b0 [id=6bc5ca21f8c54249a5150f0fea2686c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.329497 1629 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:56590 (request call id 439) took 1238 ms. Trace:
I20251025 07:52:33.329636 1629 rpcz_store.cc:276] 1025 07:52:32.090720 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:32.456149 (+365429us) service_pool.cc:225] Handling call
1025 07:52:33.329461 (+873312us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:32.753105 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
1025 07:52:32.753264 (+ 159us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
1025 07:52:32.753275 (+ 11us) alter_schema_op.cc:66] Acquired schema lock
1025 07:52:32.753285 (+ 10us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
1025 07:52:32.753320 (+ 35us) alter_schema_op.cc:128] START. Timestamp: 1761378752753295
1025 07:52:32.753325 (+ 5us) op_driver.cc:348] REPLICATION: starting
1025 07:52:32.753987 (+ 662us) log.cc:844] Serialized 173 byte log entry
1025 07:52:33.121323 (+367336us) op_driver.cc:464] REPLICATION: finished
1025 07:52:33.187203 (+ 65880us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
1025 07:52:33.231384 (+ 44181us) log.cc:844] Serialized 5 byte log entry
1025 07:52:33.329112 (+ 97728us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:33.329380 (+ 268us) log.cc:844] Serialized 14 byte log entry
1025 07:52:33.329414 (+ 34us) alter_schema_op.cc:73] Released schema lock
1025 07:52:33.329419 (+ 5us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":65792,"prepare.queue_time_us":296525,"prepare.run_cpu_time_us":15987,"prepare.run_wall_time_us":101693,"raft.queue_time_us":248,"raft.run_cpu_time_us":432,"raft.run_wall_time_us":432,"replication_time_us":367965,"thread_start_us":513209,"threads_started":3,"wal-append.queue_time_us":121558}]]}
I20251025 07:52:33.330564 1629 tablet.cc:1722] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251025 07:52:33.326109 1623 consensus_queue.cc:1175] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "23eeffd3de254ce18db5febfde971c6b" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20251025 07:52:33.331378 1171 ts_tablet_manager.cc:1916] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:33.336326 889 tablet_service.cc:1505] Processing CreateTablet for tablet 19f1b56cdbd24ebe8688cc05fb795d07 (DEFAULT_TABLE table=default.Table_7955f4594e0e4f7c803f728a43c229ca [id=95040fbb66f94779b7721e2209d82a65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.337603 877 tablet_service.cc:1505] Processing CreateTablet for tablet 294d7d20c2f545b8813e632aa058a60c (DEFAULT_TABLE table=default.Table_4eaf06c0720744ff8b2fc8866cf05e2b [id=2b66e5f736de408bbd711b88361e1c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.339283 879 tablet_service.cc:1552] Processing DeleteTablet for tablet cc407c7eb12e4eadb41fb21385611c94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:33.339931 879 tablet_service.cc:1505] Processing CreateTablet for tablet ffe7c8e1d6c44872b7b65e512764a56a (DEFAULT_TABLE table=default.Table_d9880621c8ca4025b89b6c1ec47f5cb4 [id=be817d9282bd4fbf84adf11e25f06b5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.341711 876 tablet_service.cc:1552] Processing DeleteTablet for tablet 1b5ac7e845a24b2eb5b2429d47361232 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:33.341971 883 tablet_service.cc:1505] Processing CreateTablet for tablet 0540d6a10cf04fa89756623e9aaaeb68 (DEFAULT_TABLE table=default.Table_f16b7a3b7d91423f8502d73e4234ab4a [id=cc7db0a7eaea4caa8c17281a8079c372]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.355088 876 tablet_service.cc:1505] Processing CreateTablet for tablet 698e9f8de7db4516a811c9eb9143a268 (DEFAULT_TABLE table=default.Table_48712d8a17e04509888897a9cf2b4e30 [id=488b550bbdef476c95e8bad42bd4b4dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.334218 1630 tablet.cc:1722] T 627a10a0714f4be8bdbced503fd6388d P db42be7ad3c741a0aa427f27e245993b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251025 07:52:33.362160 748 tablet_service.cc:1505] Processing CreateTablet for tablet 3e1ffd02aa384b73835b4361ae3d5467 (DEFAULT_TABLE table=default.Table_cf3a2c0772674c2a8ff5fee551a93a0f [id=1ffca9f550ce498296d664517e9fba86]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.354341 1169 raft_consensus.cc:359] T 08da9d2687704725aacd11cd92c6a9f9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.363737 1169 raft_consensus.cc:385] T 08da9d2687704725aacd11cd92c6a9f9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:33.363837 1169 raft_consensus.cc:740] T 08da9d2687704725aacd11cd92c6a9f9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:33.364147 1169 consensus_queue.cc:260] T 08da9d2687704725aacd11cd92c6a9f9 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.364838 1169 ts_tablet_manager.cc:1434] T 08da9d2687704725aacd11cd92c6a9f9 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.101s user 0.001s sys 0.001s
I20251025 07:52:33.365199 1169 tablet_bootstrap.cc:492] T 21e52928580c4762b0ae7152490ef955 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:33.310120 1170 log.cc:1199] T 5cba9c05dd424cc788febc614a34819a P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/5cba9c05dd424cc788febc614a34819a
I20251025 07:52:33.366729 1170 ts_tablet_manager.cc:1950] T 5cba9c05dd424cc788febc614a34819a P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:33.367393 1167 tablet_bootstrap.cc:654] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:33.367888 513 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 5cba9c05dd424cc788febc614a34819a (table default.Table_a437a2623d704e3ea3c5b8c65c88b936 [id=51b12903e6024e1a9adc2f9fa7400937]) successfully deleted
I20251025 07:52:33.373711 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5cba9c05dd424cc788febc614a34819a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:33.372426 1166 raft_consensus.cc:359] T fab67b56eaa44b54886c17181d8c52b2 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.376420 744 tablet_service.cc:1505] Processing CreateTablet for tablet de8da6abeda74a2a96d94ce5ac8fb7e5 (DEFAULT_TABLE table=default.Table_17e5895bc86c441ba014d094c1bf3893 [id=c8c37f4cecb94242b3cea6e61a2c1cc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.380968 1172 tablet_replica.cc:333] T 742b3ec3f6604647b9e397532f1673dd P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:33.381129 741 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48830 (request call id 454) took 1017 ms. Trace:
I20251025 07:52:33.381253 741 rpcz_store.cc:276] 1025 07:52:32.299460 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:32.899039 (+599579us) service_pool.cc:225] Handling call
1025 07:52:32.976699 (+ 77660us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1025 07:52:32.976752 (+ 53us) ts_tablet_manager.cc:682] Creating new metadata...
1025 07:52:33.254253 (+277501us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:33.316741 (+ 62488us) tablet_replica.cc:206] Creating consensus instance
1025 07:52:33.316865 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 07:52:33.380226 750 tablet_service.cc:1505] Processing CreateTablet for tablet 165575a8b5b54d3c9dbcc1cc93044623 (DEFAULT_TABLE table=default.Table_d55201c0dabc4134b687cfbd805119e3 [id=3f7923f091ac46e0bc20bf7401b5458d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.382673 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 72dc084f8efc45478771c247e07a16f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:33.382925 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b47d1b0e6a342888135d7e4e5fa1f43. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.383270 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e098c9c4613459295b6043048cba587 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:33.382925 885 tablet_service.cc:1505] Processing CreateTablet for tablet bbe1d40d67594ad389ccc97bb9e2ac3b (DEFAULT_TABLE table=default.Table_3f291c48b0144f0a8ddb1b51ebbe31cd [id=98db26b2e7bf409dbfe8d201ae701df7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.383677 751 tablet_service.cc:1505] Processing CreateTablet for tablet 830fdbc6812e416690d0db42a73af503 (DEFAULT_TABLE table=default.Table_ab4b1a5a7abb46fd978c91bb6d74840e [id=fb21a45b86ab446e8c3dae4512a9598f]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:33.384167 847 leader_election.cc:343] T ac0ea298bc114c56a8093e12def21ce1 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:33.384306 746 tablet_service.cc:1552] Processing DeleteTablet for tablet ee1c8ea694284d80a30f9c05be4cbf49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:33.384985 886 tablet_service.cc:1552] Processing DeleteTablet for tablet 6dcab53a020847a1aa27930265ba0205 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:33.385246 882 tablet_service.cc:1505] Processing CreateTablet for tablet 2c4789b9d89f42cb844e6cb595e687db (DEFAULT_TABLE table=default.Table_ddc8580fd1674f9ca61d528f22c51c94 [id=e33f00d4d5844ebba7ea3e200e154eeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.386072 741 tablet_service.cc:1552] Processing DeleteTablet for tablet 627a10a0714f4be8bdbced503fd6388d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:33 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:33.386297 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 830fdbc6812e416690d0db42a73af503. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.388962 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 6dcab53a020847a1aa27930265ba0205 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:33.389966 756 tablet_service.cc:1552] Processing DeleteTablet for tablet ffe7c8e1d6c44872b7b65e512764a56a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:33.391950 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5cba9c05dd424cc788febc614a34819a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:33.393397 740 tablet_service.cc:1505] Processing CreateTablet for tablet 2e87a81c87ec4353a8ad920307809390 (DEFAULT_TABLE table=default.Table_b8551dc7ed8c48228c5d1fec6a1e8b0c [id=52d442791ab94896b1e377af37ab6154]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.393029 747 tablet_service.cc:1505] Processing CreateTablet for tablet b02f5c0ab0b24288bb50a3ba0e6bd78a (DEFAULT_TABLE table=default.Table_29aeaf089ca24270b3f21c2e08d61634 [id=3294d42240024d25ab2487e75640ddd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.400318 1169 tablet_bootstrap.cc:654] T 21e52928580c4762b0ae7152490ef955 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:33.400826 753 tablet_service.cc:1552] Processing DeleteTablet for tablet c11f674385184ae6b235f1db8bf1fb91 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:48830
I20251025 07:52:33.401969 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbe1d40d67594ad389ccc97bb9e2ac3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.405042 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 050a5d3b5f934dd19bf2e221f2fe0780 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:33.412922 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b02f5c0ab0b24288bb50a3ba0e6bd78a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.415475 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e1ffd02aa384b73835b4361ae3d5467. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.426292 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 72dc084f8efc45478771c247e07a16f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:33.426826 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e098c9c4613459295b6043048cba587 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:33.428053 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0063a0776b2247afaa8be82f6eff26f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.435389 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de8da6abeda74a2a96d94ce5ac8fb7e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.439653 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e87a81c87ec4353a8ad920307809390. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.442929 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 165575a8b5b54d3c9dbcc1cc93044623. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.428941 878 tablet_service.cc:1505] Processing CreateTablet for tablet 6fb1a1d38fb94f07b79ac4f8334fa719 (DEFAULT_TABLE table=default.Table_d49fa3011bf146b5a0dcb1c2d4557138 [id=fc927d99526b494db7c23ccf3e94fa02]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.434473 847 leader_election.cc:304] T ac0ea298bc114c56a8093e12def21ce1 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:33.445791 1614 raft_consensus.cc:2749] T ac0ea298bc114c56a8093e12def21ce1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:33.434517 886 tablet_service.cc:1552] Processing DeleteTablet for tablet af743ee344e543a0aa6da3cd7a30e467 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:33.446424 886 tablet_service.cc:1552] Processing DeleteTablet for tablet c11f674385184ae6b235f1db8bf1fb91 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:33.446646 886 tablet_service.cc:1552] Processing DeleteTablet for tablet ffe7c8e1d6c44872b7b65e512764a56a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:33.447026 886 tablet_service.cc:1505] Processing CreateTablet for tablet de8da6abeda74a2a96d94ce5ac8fb7e5 (DEFAULT_TABLE table=default.Table_17e5895bc86c441ba014d094c1bf3893 [id=c8c37f4cecb94242b3cea6e61a2c1cc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.431223 1166 raft_consensus.cc:385] T fab67b56eaa44b54886c17181d8c52b2 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:33.447391 1166 raft_consensus.cc:740] T fab67b56eaa44b54886c17181d8c52b2 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:33.447763 1166 consensus_queue.cc:260] T fab67b56eaa44b54886c17181d8c52b2 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.448594 1166 ts_tablet_manager.cc:1434] T fab67b56eaa44b54886c17181d8c52b2 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.225s user 0.002s sys 0.000s
I20251025 07:52:33.434643 894 tablet_service.cc:1552] Processing DeleteTablet for tablet ee1c8ea694284d80a30f9c05be4cbf49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:56590
I20251025 07:52:33.449280 894 tablet_service.cc:1505] Processing CreateTablet for tablet 3e1ffd02aa384b73835b4361ae3d5467 (DEFAULT_TABLE table=default.Table_cf3a2c0772674c2a8ff5fee551a93a0f [id=1ffca9f550ce498296d664517e9fba86]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.434731 890 tablet_service.cc:1505] Processing CreateTablet for tablet a6587f4be6af44dd944e6b6f9599dfda (DEFAULT_TABLE table=default.Table_1431c97720de4d5a9a92bd2d574eb904 [id=484c28bccff0448bbca48cb7eaaa55df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.434944 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffe7c8e1d6c44872b7b65e512764a56a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.452257 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19f1b56cdbd24ebe8688cc05fb795d07. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.453110 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 421db850a64048218eecb9de65e2bdbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.453374 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0540d6a10cf04fa89756623e9aaaeb68. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.434841 1172 raft_consensus.cc:2243] T 742b3ec3f6604647b9e397532f1673dd P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:33.453680 1172 pending_rounds.cc:70] T 742b3ec3f6604647b9e397532f1673dd P 3aea2869ff814cc7b142012faa274382: Trying to abort 1 pending ops.
I20251025 07:52:33.453748 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 698e9f8de7db4516a811c9eb9143a268. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.453784 1172 pending_rounds.cc:77] T 742b3ec3f6604647b9e397532f1673dd P 3aea2869ff814cc7b142012faa274382: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214607371833827328 op_type: NO_OP noop_request { }
I20251025 07:52:33.453899 1172 raft_consensus.cc:2889] T 742b3ec3f6604647b9e397532f1673dd P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 07:52:33.454027 1172 raft_consensus.cc:2272] T 742b3ec3f6604647b9e397532f1673dd P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:33.454790 1172 ts_tablet_manager.cc:1916] T 742b3ec3f6604647b9e397532f1673dd P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:33.456377 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c4789b9d89f42cb844e6cb595e687db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.458382 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e18611cbc1348eca927162fec70e36f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.462289 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 294d7d20c2f545b8813e632aa058a60c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.463328 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b47d1b0e6a342888135d7e4e5fa1f43. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.464249 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60c60a9ee37f43978910bd067a098bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.469751 1171 ts_tablet_manager.cc:1929] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:33.470501 1171 log.cc:1199] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/90764275825c4ffb8712cb883d37a551
I20251025 07:52:33.472182 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbe1d40d67594ad389ccc97bb9e2ac3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.480432 1171 ts_tablet_manager.cc:1950] T 90764275825c4ffb8712cb883d37a551 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:33.493677 893 tablet_service.cc:1505] Processing CreateTablet for tablet 830fdbc6812e416690d0db42a73af503 (DEFAULT_TABLE table=default.Table_ab4b1a5a7abb46fd978c91bb6d74840e [id=fb21a45b86ab446e8c3dae4512a9598f]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:33.494040 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:46231, user_credentials={real_user=slave}} blocked reactor thread for 74941.4us
W20251025 07:52:33.495306 847 leader_election.cc:343] T 21e52928580c4762b0ae7152490ef955 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:33.495481 847 leader_election.cc:304] T 21e52928580c4762b0ae7152490ef955 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:33.496142 1614 raft_consensus.cc:2749] T 21e52928580c4762b0ae7152490ef955 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:33.498597 884 tablet_service.cc:1505] Processing CreateTablet for tablet b02f5c0ab0b24288bb50a3ba0e6bd78a (DEFAULT_TABLE table=default.Table_29aeaf089ca24270b3f21c2e08d61634 [id=3294d42240024d25ab2487e75640ddd0]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:33.500890 847 leader_election.cc:343] T 2784bbfb97d5466f9ec856add7d0b29a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:33.501037 847 leader_election.cc:304] T 2784bbfb97d5466f9ec856add7d0b29a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:33.501804 847 leader_election.cc:343] T 5cba9c05dd424cc788febc614a34819a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:33.502166 847 leader_election.cc:343] T 5cba9c05dd424cc788febc614a34819a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:33.503723 1218 consensus_queue.cc:1048] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20251025 07:52:33.503928 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 445) took 1425 ms. Trace:
I20251025 07:52:33.505800 1614 raft_consensus.cc:2749] T 2784bbfb97d5466f9ec856add7d0b29a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:33.508769 1171 rpcz_store.cc:276] 1025 07:52:32.077358 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:32.836201 (+758843us) service_pool.cc:225] Handling call
1025 07:52:32.923622 (+ 87421us) ts_tablet_manager.cc:1151] Deleting tablet 90764275825c4ffb8712cb883d37a551
1025 07:52:32.923628 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:33.229632 (+306004us) spinlock_profiling.cc:243] Waited 305 ms on lock 0x6160000c1654. stack: 00007f6155e019f6 00007f6155e017b0 00007f6154ddd3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f6161cc7100 00007f6163014ae3 00007f6165bd1287 00007f6165c042d4 00007f6165be4d70 00007f6165be4afd 0000000000a8867c 00007f6155e47d7f 00007f6155e4b04b 00007f6155e4aec8
1025 07:52:33.405348 (+175716us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:33.469060 (+ 63712us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:33.499800 (+ 30740us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":701003520,"tablet-delete.queue_time_us":86683}
I20251025 07:52:33.514796 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de8da6abeda74a2a96d94ce5ac8fb7e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.515502 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 452) took 1292 ms. Trace:
I20251025 07:52:33.515636 1171 rpcz_store.cc:276] 1025 07:52:32.222951 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:32.887595 (+664644us) service_pool.cc:225] Handling call
1025 07:52:33.509503 (+621908us) ts_tablet_manager.cc:1151] Deleting tablet d77704b4fe6c4456a2ba40ffec0c60fc
1025 07:52:33.515315 (+ 5812us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:33.515444 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":621146}
I20251025 07:52:33.516014 1167 tablet_bootstrap.cc:492] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:33.516162 1167 ts_tablet_manager.cc:1403] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.219s user 0.013s sys 0.012s
I20251025 07:52:33.516014 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 451) took 1308 ms. Trace:
I20251025 07:52:33.516384 1171 rpcz_store.cc:276] 1025 07:52:32.209510 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:32.883639 (+674129us) service_pool.cc:225] Handling call
1025 07:52:33.515881 (+632242us) ts_tablet_manager.cc:1151] Deleting tablet e4da89f863cc43f3bf6df908bb9dc6bf
1025 07:52:33.515888 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:33.515968 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":608135}
I20251025 07:52:33.516502 1166 tablet_bootstrap.cc:492] T 467324e2432f4caca148951329caa292 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:33.516642 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6587f4be6af44dd944e6b6f9599dfda. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.516774 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 442) took 1490 ms. Trace:
I20251025 07:52:33.516907 1171 rpcz_store.cc:276] 1025 07:52:32.026544 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:32.813572 (+787028us) service_pool.cc:225] Handling call
1025 07:52:33.516588 (+703016us) ts_tablet_manager.cc:1151] Deleting tablet f4578ce69a0d4322944e5ce7c8a59831
1025 07:52:33.516595 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:33.516739 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":594297}
I20251025 07:52:33.517161 1171 tablet_replica.cc:333] T 567a1f5c216a4a519249da66a2a03ea4 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:33.517071 1167 raft_consensus.cc:359] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.517248 1167 raft_consensus.cc:385] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:33.517328 1167 raft_consensus.cc:740] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:33.517370 1171 raft_consensus.cc:2243] T 567a1f5c216a4a519249da66a2a03ea4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:33.517567 1167 consensus_queue.cc:260] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.518193 1171 raft_consensus.cc:2272] T 567a1f5c216a4a519249da66a2a03ea4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:33.526193 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e1ffd02aa384b73835b4361ae3d5467. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.530774 1171 ts_tablet_manager.cc:1916] T 567a1f5c216a4a519249da66a2a03ea4 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:33.531143 1167 ts_tablet_manager.cc:1434] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20251025 07:52:33.531502 1167 tablet_bootstrap.cc:492] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:33.533273 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b02f5c0ab0b24288bb50a3ba0e6bd78a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.533712 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fb1a1d38fb94f07b79ac4f8334fa719. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.533931 1167 tablet_bootstrap.cc:654] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:33.536863 1612 consensus_queue.cc:1048] T 7e869c65771f4339a546dfeebb227c62 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:33.539646 1171 ts_tablet_manager.cc:1929] T 567a1f5c216a4a519249da66a2a03ea4 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:33.541180 875 tablet_service.cc:1505] Processing CreateTablet for tablet 165575a8b5b54d3c9dbcc1cc93044623 (DEFAULT_TABLE table=default.Table_d55201c0dabc4134b687cfbd805119e3 [id=3f7923f091ac46e0bc20bf7401b5458d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:33.541486 1171 log.cc:1199] T 567a1f5c216a4a519249da66a2a03ea4 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/567a1f5c216a4a519249da66a2a03ea4
I20251025 07:52:33.541623 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 165575a8b5b54d3c9dbcc1cc93044623. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.541994 1171 ts_tablet_manager.cc:1950] T 567a1f5c216a4a519249da66a2a03ea4 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:33.543044 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 444) took 1474 ms. Trace:
I20251025 07:52:33.546211 1171 rpcz_store.cc:276] 1025 07:52:32.068969 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:32.832251 (+763282us) service_pool.cc:225] Handling call
1025 07:52:33.517104 (+684853us) ts_tablet_manager.cc:1151] Deleting tablet 567a1f5c216a4a519249da66a2a03ea4
1025 07:52:33.517112 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:33.535307 (+ 18195us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:33.539503 (+ 4196us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:33.542980 (+ 3477us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11881,"tablet-delete.queue_time_us":594487}
I20251025 07:52:33.547014 1171 tablet_replica.cc:333] T 1aeaad5c5d01478bb997254ac372d9a6 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:33.547382 1171 raft_consensus.cc:2243] T 1aeaad5c5d01478bb997254ac372d9a6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:33.547621 1171 raft_consensus.cc:2272] T 1aeaad5c5d01478bb997254ac372d9a6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:33.548475 1171 ts_tablet_manager.cc:1916] T 1aeaad5c5d01478bb997254ac372d9a6 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:33.554322 1166 tablet_bootstrap.cc:654] T 467324e2432f4caca148951329caa292 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:33.557806 1167 tablet_bootstrap.cc:492] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:33.557991 1167 ts_tablet_manager.cc:1403] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.012s
I20251025 07:52:33.558820 1167 raft_consensus.cc:359] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.559031 1167 raft_consensus.cc:385] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:33.559118 1167 raft_consensus.cc:740] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:33.559278 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 830fdbc6812e416690d0db42a73af503. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:33.559393 1167 consensus_queue.cc:260] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.560006 1167 ts_tablet_manager.cc:1434] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251025 07:52:33.560304 1167 tablet_bootstrap.cc:492] T 08da9d2687704725aacd11cd92c6a9f9 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:33.561959 1167 tablet_bootstrap.cc:654] T 08da9d2687704725aacd11cd92c6a9f9 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:33.565814 1167 tablet_bootstrap.cc:492] T 08da9d2687704725aacd11cd92c6a9f9 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:33.565973 1167 ts_tablet_manager.cc:1403] T 08da9d2687704725aacd11cd92c6a9f9 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20251025 07:52:33.566830 1441 raft_consensus.cc:493] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:33.566985 1441 raft_consensus.cc:515] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.567675 1441 leader_election.cc:290] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:33.569025 1167 raft_consensus.cc:359] T 08da9d2687704725aacd11cd92c6a9f9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.569293 1167 raft_consensus.cc:385] T 08da9d2687704725aacd11cd92c6a9f9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:33.569432 1167 raft_consensus.cc:740] T 08da9d2687704725aacd11cd92c6a9f9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:33.569764 1167 consensus_queue.cc:260] T 08da9d2687704725aacd11cd92c6a9f9 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.570549 1167 ts_tablet_manager.cc:1434] T 08da9d2687704725aacd11cd92c6a9f9 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20251025 07:52:33.570864 1614 raft_consensus.cc:493] T fab67b56eaa44b54886c17181d8c52b2 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 07:52:33.570858 712 leader_election.cc:343] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 2aa522477e934c89834026dc480fe7c0
I20251025 07:52:33.571328 1167 tablet_bootstrap.cc:492] T 8e6d76b1a97945cfb6c3055e1b622f8a P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:33.572280 1171 ts_tablet_manager.cc:1929] T 1aeaad5c5d01478bb997254ac372d9a6 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:33.572549 1171 log.cc:1199] T 1aeaad5c5d01478bb997254ac372d9a6 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/1aeaad5c5d01478bb997254ac372d9a6
I20251025 07:52:33.573005 1171 ts_tablet_manager.cc:1950] T 1aeaad5c5d01478bb997254ac372d9a6 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:33.573419 1167 tablet_bootstrap.cc:654] T 8e6d76b1a97945cfb6c3055e1b622f8a P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:33.573948 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 447) took 1410 ms. Trace:
I20251025 07:52:33.574365 1171 rpcz_store.cc:276] 1025 07:52:32.163693 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:32.837643 (+673950us) service_pool.cc:225] Handling call
1025 07:52:33.546910 (+709267us) ts_tablet_manager.cc:1151] Deleting tablet 1aeaad5c5d01478bb997254ac372d9a6
1025 07:52:33.546923 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:33.553941 (+ 7018us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:33.572120 (+ 18179us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:33.573898 (+ 1778us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":620544}
I20251025 07:52:33.574472 1614 raft_consensus.cc:515] T fab67b56eaa44b54886c17181d8c52b2 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.575134 1614 leader_election.cc:290] T fab67b56eaa44b54886c17181d8c52b2 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:33.578087 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fab67b56eaa44b54886c17181d8c52b2" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:33.578817 850 leader_election.cc:343] T fab67b56eaa44b54886c17181d8c52b2 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:33.582280 1171 tablet_replica.cc:333] T 742b3ec3f6604647b9e397532f1673dd P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:33.582482 1167 tablet_bootstrap.cc:492] T 8e6d76b1a97945cfb6c3055e1b622f8a P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:33.582505 1171 raft_consensus.cc:2243] T 742b3ec3f6604647b9e397532f1673dd P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:33.582660 1167 ts_tablet_manager.cc:1403] T 8e6d76b1a97945cfb6c3055e1b622f8a P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20251025 07:52:33.582705 1171 raft_consensus.cc:2272] T 742b3ec3f6604647b9e397532f1673dd P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:33.583245 1171 ts_tablet_manager.cc:1916] T 742b3ec3f6604647b9e397532f1673dd P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:33.583452 1167 raft_consensus.cc:359] T 8e6d76b1a97945cfb6c3055e1b622f8a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.583688 1167 raft_consensus.cc:385] T 8e6d76b1a97945cfb6c3055e1b622f8a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:33.583781 1167 raft_consensus.cc:740] T 8e6d76b1a97945cfb6c3055e1b622f8a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:33.584102 1167 consensus_queue.cc:260] T 8e6d76b1a97945cfb6c3055e1b622f8a P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.584762 1167 ts_tablet_manager.cc:1434] T 8e6d76b1a97945cfb6c3055e1b622f8a P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:33.591254 1172 ts_tablet_manager.cc:1929] T 742b3ec3f6604647b9e397532f1673dd P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:33.592665 1172 log.cc:1199] T 742b3ec3f6604647b9e397532f1673dd P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/742b3ec3f6604647b9e397532f1673dd
I20251025 07:52:33.593713 1172 ts_tablet_manager.cc:1950] T 742b3ec3f6604647b9e397532f1673dd P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:33.594902 1167 tablet_bootstrap.cc:492] T 7e869c65771f4339a546dfeebb227c62 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:33.604198 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fab67b56eaa44b54886c17181d8c52b2" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
W20251025 07:52:33.604583 710 leader_election.cc:343] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 2aa522477e934c89834026dc480fe7c0
I20251025 07:52:33.604981 710 leader_election.cc:304] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:33.606581 1167 tablet_bootstrap.cc:654] T 7e869c65771f4339a546dfeebb227c62 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:33.607256 1441 raft_consensus.cc:2749] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:33.608063 982 consensus_peers.cc:597] T 7e869c65771f4339a546dfeebb227c62 P db42be7ad3c741a0aa427f27e245993b -> Peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Couldn't send request to peer 23eeffd3de254ce18db5febfde971c6b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:33.611730 1169 tablet_bootstrap.cc:492] T 21e52928580c4762b0ae7152490ef955 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:33.611991 1169 ts_tablet_manager.cc:1403] T 21e52928580c4762b0ae7152490ef955 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.247s user 0.005s sys 0.013s
I20251025 07:52:33.612739 1169 raft_consensus.cc:359] T 21e52928580c4762b0ae7152490ef955 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.612960 1169 raft_consensus.cc:385] T 21e52928580c4762b0ae7152490ef955 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:33.613023 1169 raft_consensus.cc:740] T 21e52928580c4762b0ae7152490ef955 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:33.613242 1169 consensus_queue.cc:260] T 21e52928580c4762b0ae7152490ef955 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.613718 1169 ts_tablet_manager.cc:1434] T 21e52928580c4762b0ae7152490ef955 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 07:52:33.614027 1169 tablet_bootstrap.cc:492] T fd9b4be139d347f4a6a60767029c342a P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:33.616179 775 raft_consensus.cc:3060] T cd23b8094d7e472eba6bd170248858a9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:33.597805 1166 tablet_bootstrap.cc:492] T 467324e2432f4caca148951329caa292 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:33.618326 1166 ts_tablet_manager.cc:1403] T 467324e2432f4caca148951329caa292 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.105s user 0.007s sys 0.004s
I20251025 07:52:33.619732 1166 raft_consensus.cc:359] T 467324e2432f4caca148951329caa292 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.620033 1166 raft_consensus.cc:385] T 467324e2432f4caca148951329caa292 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:33.620158 1166 raft_consensus.cc:740] T 467324e2432f4caca148951329caa292 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:33.620424 1166 consensus_queue.cc:260] T 467324e2432f4caca148951329caa292 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.622843 1166 ts_tablet_manager.cc:1434] T 467324e2432f4caca148951329caa292 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251025 07:52:33.623195 1166 tablet_bootstrap.cc:492] T 4037dee385d04e85bb78c8e77cb00379 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:33.625250 1166 tablet_bootstrap.cc:654] T 4037dee385d04e85bb78c8e77cb00379 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:33.607659 847 leader_election.cc:343] T fab67b56eaa44b54886c17181d8c52b2 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:33.626405 847 leader_election.cc:304] T fab67b56eaa44b54886c17181d8c52b2 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:33.626749 1614 raft_consensus.cc:2749] T fab67b56eaa44b54886c17181d8c52b2 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:33.627938 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 401) took 1886 ms. Trace:
I20251025 07:52:33.628476 1172 rpcz_store.cc:276] 1025 07:52:31.741902 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:31.764532 (+ 22630us) service_pool.cc:225] Handling call
1025 07:52:33.243900 (+1479368us) ts_tablet_manager.cc:1151] Deleting tablet 742b3ec3f6604647b9e397532f1673dd
1025 07:52:33.370839 (+126939us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:33.484655 (+113816us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:33.591098 (+106443us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:33.627896 (+ 36798us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1479125}
I20251025 07:52:33.635419 1172 tablet_replica.cc:333] T ac0ea298bc114c56a8093e12def21ce1 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:33.635744 1172 raft_consensus.cc:2243] T ac0ea298bc114c56a8093e12def21ce1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:33.636381 1172 raft_consensus.cc:2272] T ac0ea298bc114c56a8093e12def21ce1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:33.638327 1169 tablet_bootstrap.cc:654] T fd9b4be139d347f4a6a60767029c342a P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:33.641402 1172 ts_tablet_manager.cc:1916] T ac0ea298bc114c56a8093e12def21ce1 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:33.642446 1645 tablet.cc:1722] T 3d5e0dd14e474fe09993a03851f8b1f0 P 23eeffd3de254ce18db5febfde971c6b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251025 07:52:33.649055 1167 tablet_bootstrap.cc:492] T 7e869c65771f4339a546dfeebb227c62 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:33.649084 1166 tablet_bootstrap.cc:492] T 4037dee385d04e85bb78c8e77cb00379 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:33.649214 1167 ts_tablet_manager.cc:1403] T 7e869c65771f4339a546dfeebb227c62 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.010s
I20251025 07:52:33.649238 1166 ts_tablet_manager.cc:1403] T 4037dee385d04e85bb78c8e77cb00379 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.003s
I20251025 07:52:33.648938 1650 tablet.cc:1722] T 627a10a0714f4be8bdbced503fd6388d P 23eeffd3de254ce18db5febfde971c6b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251025 07:52:33.650017 1167 raft_consensus.cc:359] T 7e869c65771f4339a546dfeebb227c62 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.650266 1167 raft_consensus.cc:385] T 7e869c65771f4339a546dfeebb227c62 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:33.650382 1167 raft_consensus.cc:740] T 7e869c65771f4339a546dfeebb227c62 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:33.650065 1166 raft_consensus.cc:359] T 4037dee385d04e85bb78c8e77cb00379 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.650723 1166 raft_consensus.cc:385] T 4037dee385d04e85bb78c8e77cb00379 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:33.650717 1167 consensus_queue.cc:260] T 7e869c65771f4339a546dfeebb227c62 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.650842 1166 raft_consensus.cc:740] T 4037dee385d04e85bb78c8e77cb00379 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:33.651131 1166 consensus_queue.cc:260] T 4037dee385d04e85bb78c8e77cb00379 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.651731 1166 ts_tablet_manager.cc:1434] T 4037dee385d04e85bb78c8e77cb00379 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:33.653249 1166 tablet_bootstrap.cc:492] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:33.655190 1166 tablet_bootstrap.cc:654] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:33.658797 1171 ts_tablet_manager.cc:1929] T 742b3ec3f6604647b9e397532f1673dd P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:33.662282 1171 log.cc:1199] T 742b3ec3f6604647b9e397532f1673dd P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/742b3ec3f6604647b9e397532f1673dd
I20251025 07:52:33.662762 1171 ts_tablet_manager.cc:1950] T 742b3ec3f6604647b9e397532f1673dd P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:33.667965 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 446) took 1589 ms. Trace:
I20251025 07:52:33.670199 1171 rpcz_store.cc:276] 1025 07:52:32.078896 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:32.836970 (+758074us) service_pool.cc:225] Handling call
1025 07:52:33.582201 (+745231us) ts_tablet_manager.cc:1151] Deleting tablet 742b3ec3f6604647b9e397532f1673dd
1025 07:52:33.582210 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:33.633845 (+ 51635us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:33.658664 (+ 24819us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:33.667908 (+ 9244us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":655473}
I20251025 07:52:33.670948 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 448) took 1494 ms. Trace:
I20251025 07:52:33.671118 1171 rpcz_store.cc:276] 1025 07:52:32.176952 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:32.844370 (+667418us) service_pool.cc:225] Handling call
1025 07:52:33.670761 (+826391us) ts_tablet_manager.cc:1151] Deleting tablet 26c849d376d143c6bef0c8e0e1d7d03a
1025 07:52:33.670771 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:33.670897 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":743804}
I20251025 07:52:33.671685 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 449) took 1492 ms. Trace:
I20251025 07:52:33.671838 1171 rpcz_store.cc:276] 1025 07:52:32.179374 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:32.867564 (+688190us) service_pool.cc:225] Handling call
1025 07:52:33.671541 (+803977us) ts_tablet_manager.cc:1151] Deleting tablet 4f7c15529be54a11961f385a953b0b03
1025 07:52:33.671550 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:33.671638 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":743015}
I20251025 07:52:33.672374 1171 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48830 (request call id 450) took 1491 ms. Trace:
I20251025 07:52:33.672684 1171 rpcz_store.cc:276] 1025 07:52:32.180711 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:32.869532 (+688821us) service_pool.cc:225] Handling call
1025 07:52:33.672232 (+802700us) ts_tablet_manager.cc:1151] Deleting tablet ce1ebb1ec14b4801930a72765bca1a7d
1025 07:52:33.672242 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:33.672323 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":742289}
I20251025 07:52:33.676510 1169 tablet_bootstrap.cc:492] T fd9b4be139d347f4a6a60767029c342a P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:33.676730 1169 ts_tablet_manager.cc:1403] T fd9b4be139d347f4a6a60767029c342a P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.063s user 0.001s sys 0.002s
I20251025 07:52:33.665197 1645 tablet.cc:1722] T cd23b8094d7e472eba6bd170248858a9 P 23eeffd3de254ce18db5febfde971c6b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251025 07:52:33.662837 1167 ts_tablet_manager.cc:1434] T 7e869c65771f4339a546dfeebb227c62 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:33.677909 1167 tablet_bootstrap.cc:492] T fd9b4be139d347f4a6a60767029c342a P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:33.678753 1169 raft_consensus.cc:359] T fd9b4be139d347f4a6a60767029c342a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.679011 1169 raft_consensus.cc:385] T fd9b4be139d347f4a6a60767029c342a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:33.679126 1169 raft_consensus.cc:740] T fd9b4be139d347f4a6a60767029c342a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:33.680821 1169 consensus_queue.cc:260] T fd9b4be139d347f4a6a60767029c342a P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.681496 1169 ts_tablet_manager.cc:1434] T fd9b4be139d347f4a6a60767029c342a P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20251025 07:52:33.694614 1169 tablet_bootstrap.cc:492] T ac0ea298bc114c56a8093e12def21ce1 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:33.696372 1171 tablet_replica.cc:333] T 627a10a0714f4be8bdbced503fd6388d P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:33.696671 1171 raft_consensus.cc:2243] T 627a10a0714f4be8bdbced503fd6388d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:33.696970 1171 raft_consensus.cc:2272] T 627a10a0714f4be8bdbced503fd6388d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:33.699272 1167 tablet_bootstrap.cc:654] T fd9b4be139d347f4a6a60767029c342a P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:33.702520 1169 tablet_bootstrap.cc:654] T ac0ea298bc114c56a8093e12def21ce1 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:33.703967 1172 ts_tablet_manager.cc:1929] T ac0ea298bc114c56a8093e12def21ce1 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:33.704207 1172 log.cc:1199] T ac0ea298bc114c56a8093e12def21ce1 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/ac0ea298bc114c56a8093e12def21ce1
I20251025 07:52:33.704731 1172 ts_tablet_manager.cc:1950] T ac0ea298bc114c56a8093e12def21ce1 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:33.706388 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 409) took 1771 ms. Trace:
I20251025 07:52:33.707767 1166 tablet_bootstrap.cc:492] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:33.710204 1172 rpcz_store.cc:276] 1025 07:52:31.934569 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:31.934683 (+ 114us) service_pool.cc:225] Handling call
1025 07:52:33.635338 (+1700655us) ts_tablet_manager.cc:1151] Deleting tablet ac0ea298bc114c56a8093e12def21ce1
1025 07:52:33.635351 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:33.671580 (+ 36229us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:33.700469 (+ 28889us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:33.706317 (+ 5848us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1693030}
I20251025 07:52:33.710376 1166 ts_tablet_manager.cc:1403] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.057s user 0.010s sys 0.030s
I20251025 07:52:33.711089 1172 tablet_replica.cc:333] T 5cba9c05dd424cc788febc614a34819a P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:33.711563 1171 ts_tablet_manager.cc:1916] T 627a10a0714f4be8bdbced503fd6388d P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:33.711493 1166 raft_consensus.cc:359] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.712133 1166 raft_consensus.cc:385] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:33.712157 1172 raft_consensus.cc:2243] T 5cba9c05dd424cc788febc614a34819a P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:33.712307 1166 raft_consensus.cc:740] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:33.712539 1172 pending_rounds.cc:70] T 5cba9c05dd424cc788febc614a34819a P 3aea2869ff814cc7b142012faa274382: Trying to abort 1 pending ops.
I20251025 07:52:33.712648 1172 pending_rounds.cc:77] T 5cba9c05dd424cc788febc614a34819a P 3aea2869ff814cc7b142012faa274382: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214607372628738048 op_type: NO_OP noop_request { }
I20251025 07:52:33.714082 1653 raft_consensus.cc:493] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8603bddd8a084ff7a73a25641768dfa9)
I20251025 07:52:33.714566 1653 raft_consensus.cc:515] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } }
I20251025 07:52:33.712685 1166 consensus_queue.cc:260] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.716341 1172 raft_consensus.cc:2889] T 5cba9c05dd424cc788febc614a34819a P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 07:52:33.716470 1218 raft_consensus.cc:493] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:33.716687 1172 raft_consensus.cc:2272] T 5cba9c05dd424cc788febc614a34819a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:33.716735 1218 raft_consensus.cc:515] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.717564 1166 ts_tablet_manager.cc:1434] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251025 07:52:33.717648 1218 leader_election.cc:290] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:33.717823 1172 ts_tablet_manager.cc:1916] T 5cba9c05dd424cc788febc614a34819a P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:33.718720 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7fbae8be73bf4e18b5a5c4ec4cdebc12" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:33.718943 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7fbae8be73bf4e18b5a5c4ec4cdebc12" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:33.719973 1653 leader_election.cc:290] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b2dfbd0b44a441c9be9254f660974944 (127.0.104.62:37421), 8603bddd8a084ff7a73a25641768dfa9 (127.0.104.61:42203)
I20251025 07:52:33.720261 463 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b4a999d754c941ec850c143e4e24c407" candidate_term: 2 candidate_status { last_received { term: 1 index: 667 } } ignore_live_leader: false dest_uuid: "b2dfbd0b44a441c9be9254f660974944" is_pre_election: true
I20251025 07:52:33.721097 1169 tablet_bootstrap.cc:492] T ac0ea298bc114c56a8093e12def21ce1 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:33.721269 1169 ts_tablet_manager.cc:1403] T ac0ea298bc114c56a8093e12def21ce1 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.002s
I20251025 07:52:33.721911 1169 raft_consensus.cc:359] T ac0ea298bc114c56a8093e12def21ce1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.722082 1169 raft_consensus.cc:385] T ac0ea298bc114c56a8093e12def21ce1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:33.722186 1169 raft_consensus.cc:740] T ac0ea298bc114c56a8093e12def21ce1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:33.722424 1169 consensus_queue.cc:260] T ac0ea298bc114c56a8093e12def21ce1 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.722882 1169 ts_tablet_manager.cc:1434] T ac0ea298bc114c56a8093e12def21ce1 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251025 07:52:33.717880 1166 tablet_bootstrap.cc:492] T 8c76d1b509f04ab1a10b83cff675c11e P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
W20251025 07:52:33.720546 847 leader_election.cc:343] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:33.723450 850 leader_election.cc:343] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:33.724515 1169 tablet_bootstrap.cc:492] T 2784bbfb97d5466f9ec856add7d0b29a P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:33.725082 850 leader_election.cc:304] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:33.726326 1218 raft_consensus.cc:2749] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:33.727480 1169 tablet_bootstrap.cc:654] T 2784bbfb97d5466f9ec856add7d0b29a P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:33.729398 1166 tablet_bootstrap.cc:654] T 8c76d1b509f04ab1a10b83cff675c11e P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:33.737469 1441 raft_consensus.cc:493] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:33.737721 1441 raft_consensus.cc:515] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:33.739523 712 leader_election.cc:343] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 87a93778620f4c84b3b3f7a3d7d7e87f
W20251025 07:52:33.740618 710 leader_election.cc:343] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 87a93778620f4c84b3b3f7a3d7d7e87f
I20251025 07:52:33.740748 710 leader_election.cc:304] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:33.738601 1441 leader_election.cc:290] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:33.741787 1441 raft_consensus.cc:2749] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:33.752979 1171 ts_tablet_manager.cc:1929] T 627a10a0714f4be8bdbced503fd6388d P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 07:52:33.766549 1171 log.cc:1199] T 627a10a0714f4be8bdbced503fd6388d P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/627a10a0714f4be8bdbced503fd6388d
I20251025 07:52:33.767023 1171 ts_tablet_manager.cc:1950] T 627a10a0714f4be8bdbced503fd6388d P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:33.768765 1166 tablet_bootstrap.cc:492] T 8c76d1b509f04ab1a10b83cff675c11e P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:33.757907 1167 tablet_bootstrap.cc:492] T fd9b4be139d347f4a6a60767029c342a P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:33.768998 1166 ts_tablet_manager.cc:1403] T 8c76d1b509f04ab1a10b83cff675c11e P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.012s
I20251025 07:52:33.769176 1167 ts_tablet_manager.cc:1403] T fd9b4be139d347f4a6a60767029c342a P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.092s user 0.011s sys 0.010s
I20251025 07:52:33.769958 1166 raft_consensus.cc:359] T 8c76d1b509f04ab1a10b83cff675c11e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.770351 1166 raft_consensus.cc:385] T 8c76d1b509f04ab1a10b83cff675c11e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:33.770299 1167 raft_consensus.cc:359] T fd9b4be139d347f4a6a60767029c342a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.770473 1167 raft_consensus.cc:385] T fd9b4be139d347f4a6a60767029c342a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:33.770539 1167 raft_consensus.cc:740] T fd9b4be139d347f4a6a60767029c342a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:33.770557 1166 raft_consensus.cc:740] T 8c76d1b509f04ab1a10b83cff675c11e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:33.770802 1167 consensus_queue.cc:260] T fd9b4be139d347f4a6a60767029c342a P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.770975 1166 consensus_queue.cc:260] T 8c76d1b509f04ab1a10b83cff675c11e P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.771399 1167 ts_tablet_manager.cc:1434] T fd9b4be139d347f4a6a60767029c342a P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 07:52:33.771885 1166 ts_tablet_manager.cc:1434] T 8c76d1b509f04ab1a10b83cff675c11e P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251025 07:52:33.772387 1166 tablet_bootstrap.cc:492] T b772025852794a49b072f37cccbd3291 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:33.772444 1658 raft_consensus.cc:493] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8603bddd8a084ff7a73a25641768dfa9)
I20251025 07:52:33.772630 1658 raft_consensus.cc:515] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } }
I20251025 07:52:33.774420 1166 tablet_bootstrap.cc:654] T b772025852794a49b072f37cccbd3291 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:33.777191 1658 leader_election.cc:290] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8603bddd8a084ff7a73a25641768dfa9 (127.0.104.61:42203), b4a999d754c941ec850c143e4e24c407 (127.0.104.60:37611)
I20251025 07:52:33.778992 1422 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2dfbd0b44a441c9be9254f660974944" candidate_term: 2 candidate_status { last_received { term: 1 index: 667 } } ignore_live_leader: false dest_uuid: "b4a999d754c941ec850c143e4e24c407" is_pre_election: true
I20251025 07:52:33.779495 1422 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2dfbd0b44a441c9be9254f660974944 in term 1.
I20251025 07:52:33.780242 440 leader_election.cc:304] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2dfbd0b44a441c9be9254f660974944, b4a999d754c941ec850c143e4e24c407; no voters:
I20251025 07:52:33.780709 1658 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 07:52:33.780900 1658 raft_consensus.cc:493] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8603bddd8a084ff7a73a25641768dfa9)
I20251025 07:52:33.781064 1658 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [term 1 FOLLOWER]: Advancing to term 2
I20251025 07:52:33.782491 1167 tablet_bootstrap.cc:492] T 21e52928580c4762b0ae7152490ef955 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:33.784544 1167 tablet_bootstrap.cc:654] T 21e52928580c4762b0ae7152490ef955 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:33.788034 1172 ts_tablet_manager.cc:1929] T 5cba9c05dd424cc788febc614a34819a P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:33.791131 1658 raft_consensus.cc:515] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } }
I20251025 07:52:33.792448 1167 tablet_bootstrap.cc:492] T 21e52928580c4762b0ae7152490ef955 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:33.792536 1169 tablet_bootstrap.cc:492] T 2784bbfb97d5466f9ec856add7d0b29a P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:33.792743 1167 ts_tablet_manager.cc:1403] T 21e52928580c4762b0ae7152490ef955 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20251025 07:52:33.792747 1169 ts_tablet_manager.cc:1403] T 2784bbfb97d5466f9ec856add7d0b29a P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.068s user 0.002s sys 0.004s
I20251025 07:52:33.793751 1167 raft_consensus.cc:359] T 21e52928580c4762b0ae7152490ef955 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.793846 1169 raft_consensus.cc:359] T 2784bbfb97d5466f9ec856add7d0b29a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.794086 1169 raft_consensus.cc:385] T 2784bbfb97d5466f9ec856add7d0b29a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:33.794106 1167 raft_consensus.cc:385] T 21e52928580c4762b0ae7152490ef955 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:33.794219 1169 raft_consensus.cc:740] T 2784bbfb97d5466f9ec856add7d0b29a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:33.794375 1167 raft_consensus.cc:740] T 21e52928580c4762b0ae7152490ef955 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:33.794494 1169 consensus_queue.cc:260] T 2784bbfb97d5466f9ec856add7d0b29a P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.794747 1167 consensus_queue.cc:260] T 21e52928580c4762b0ae7152490ef955 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.795065 1169 ts_tablet_manager.cc:1434] T 2784bbfb97d5466f9ec856add7d0b29a P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:33.795346 1167 ts_tablet_manager.cc:1434] T 21e52928580c4762b0ae7152490ef955 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:33.798434 1169 tablet_bootstrap.cc:492] T f3c87a83082b421cb4be546ac4ce7ad8 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:33.798820 1167 tablet_bootstrap.cc:492] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:33.800657 1167 tablet_bootstrap.cc:654] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:33.790119 1218 raft_consensus.cc:493] T a13bf4d7eb1440f7a28b85cb1bf99667 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:33.801470 1172 log.cc:1199] T 5cba9c05dd424cc788febc614a34819a P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/5cba9c05dd424cc788febc614a34819a
I20251025 07:52:33.801517 1218 raft_consensus.cc:515] T a13bf4d7eb1440f7a28b85cb1bf99667 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.801925 1172 ts_tablet_manager.cc:1950] T 5cba9c05dd424cc788febc614a34819a P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:33.802697 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a13bf4d7eb1440f7a28b85cb1bf99667" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:33.802820 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 407) took 1973 ms. Trace:
I20251025 07:52:33.802857 1218 leader_election.cc:290] T a13bf4d7eb1440f7a28b85cb1bf99667 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:33.802928 1172 rpcz_store.cc:276] 1025 07:52:31.829153 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:31.830234 (+ 1081us) service_pool.cc:225] Handling call
1025 07:52:33.710993 (+1880759us) ts_tablet_manager.cc:1151] Deleting tablet 5cba9c05dd424cc788febc614a34819a
1025 07:52:33.711003 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:33.759515 (+ 48512us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:33.787858 (+ 28343us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:33.802758 (+ 14900us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1768624}
I20251025 07:52:33.803716 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a13bf4d7eb1440f7a28b85cb1bf99667" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:33.810288 1167 tablet_bootstrap.cc:492] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:33.810607 1167 ts_tablet_manager.cc:1403] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.005s
I20251025 07:52:33.811580 1167 raft_consensus.cc:359] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.811861 1167 raft_consensus.cc:385] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:33.811966 1167 raft_consensus.cc:740] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:33.812284 1167 consensus_queue.cc:260] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.812852 1167 ts_tablet_manager.cc:1434] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251025 07:52:33.813241 1167 tablet_bootstrap.cc:492] T 2784bbfb97d5466f9ec856add7d0b29a P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:33.815014 1167 tablet_bootstrap.cc:654] T 2784bbfb97d5466f9ec856add7d0b29a P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:33.821837 1169 tablet_bootstrap.cc:654] T f3c87a83082b421cb4be546ac4ce7ad8 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:33.823987 1167 tablet_bootstrap.cc:492] T 2784bbfb97d5466f9ec856add7d0b29a P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:33.824235 1167 ts_tablet_manager.cc:1403] T 2784bbfb97d5466f9ec856add7d0b29a P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20251025 07:52:33.824481 1422 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2dfbd0b44a441c9be9254f660974944" candidate_term: 2 candidate_status { last_received { term: 1 index: 667 } } ignore_live_leader: false dest_uuid: "b4a999d754c941ec850c143e4e24c407"
I20251025 07:52:33.824795 1422 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [term 1 FOLLOWER]: Advancing to term 2
I20251025 07:52:33.825384 1167 raft_consensus.cc:359] T 2784bbfb97d5466f9ec856add7d0b29a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.825604 1167 raft_consensus.cc:385] T 2784bbfb97d5466f9ec856add7d0b29a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:33.825717 1167 raft_consensus.cc:740] T 2784bbfb97d5466f9ec856add7d0b29a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:33.825958 1167 consensus_queue.cc:260] T 2784bbfb97d5466f9ec856add7d0b29a P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.826503 1167 ts_tablet_manager.cc:1434] T 2784bbfb97d5466f9ec856add7d0b29a P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251025 07:52:33.826908 1167 tablet_bootstrap.cc:492] T a13bf4d7eb1440f7a28b85cb1bf99667 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:33.828666 1167 tablet_bootstrap.cc:654] T a13bf4d7eb1440f7a28b85cb1bf99667 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:33.831936 1422 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2dfbd0b44a441c9be9254f660974944 in term 2.
I20251025 07:52:33.832648 440 leader_election.cc:304] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2dfbd0b44a441c9be9254f660974944, b4a999d754c941ec850c143e4e24c407; no voters:
I20251025 07:52:33.842453 1658 leader_election.cc:290] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [CANDIDATE]: Term 2 election: Requested vote from peers 8603bddd8a084ff7a73a25641768dfa9 (127.0.104.61:42203), b4a999d754c941ec850c143e4e24c407 (127.0.104.60:37611)
I20251025 07:52:33.842693 1658 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [term 2 FOLLOWER]: Leader election won for term 2
I20251025 07:52:33.842826 1658 raft_consensus.cc:697] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [term 2 LEADER]: Becoming Leader. State: Replica: b2dfbd0b44a441c9be9254f660974944, State: Running, Role: LEADER
I20251025 07:52:33.843241 1658 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 666, Committed index: 666, Last appended: 1.667, Last appended by leader: 667, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } }
I20251025 07:52:33.844540 1658 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2dfbd0b44a441c9be9254f660974944. Latest consensus state: current_term: 2 leader_uuid: "b2dfbd0b44a441c9be9254f660974944" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } } }
I20251025 07:52:33.844775 1658 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [sys.catalog]: This master's current role is: LEADER
W20251025 07:52:33.803244 847 leader_election.cc:343] T a13bf4d7eb1440f7a28b85cb1bf99667 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:33.845268 1172 tablet_replica.cc:333] T 08da9d2687704725aacd11cd92c6a9f9 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
W20251025 07:52:33.845333 850 leader_election.cc:343] T a13bf4d7eb1440f7a28b85cb1bf99667 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:33.845495 1172 raft_consensus.cc:2243] T 08da9d2687704725aacd11cd92c6a9f9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:33.845567 850 leader_election.cc:304] T a13bf4d7eb1440f7a28b85cb1bf99667 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:33.845703 1172 raft_consensus.cc:2272] T 08da9d2687704725aacd11cd92c6a9f9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:33.846010 1218 raft_consensus.cc:2749] T a13bf4d7eb1440f7a28b85cb1bf99667 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:33.846287 1172 ts_tablet_manager.cc:1916] T 08da9d2687704725aacd11cd92c6a9f9 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:33.839214 1166 tablet_bootstrap.cc:492] T b772025852794a49b072f37cccbd3291 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:33.848651 1166 ts_tablet_manager.cc:1403] T b772025852794a49b072f37cccbd3291 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.076s user 0.027s sys 0.032s
I20251025 07:52:33.849573 1166 raft_consensus.cc:359] T b772025852794a49b072f37cccbd3291 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:33.849854 1166 raft_consensus.cc:385] T b772025852794a49b072f37cccbd3291 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:33.850021 1166 raft_consensus.cc:740] T b772025852794a49b072f37cccbd3291 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:33.850353 1166 consensus_queue.cc:260] T b772025852794a49b072f37cccbd3291 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:33.855613 1167 tablet_bootstrap.cc:492] T a13bf4d7eb1440f7a28b85cb1bf99667 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:33.855790 1167 ts_tablet_manager.cc:1403] T a13bf4d7eb1440f7a28b85cb1bf99667 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.005s
I20251025 07:52:33.856542 1167 raft_consensus.cc:359] T a13bf4d7eb1440f7a28b85cb1bf99667 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.856734 1167 raft_consensus.cc:385] T a13bf4d7eb1440f7a28b85cb1bf99667 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:33.856803 1167 raft_consensus.cc:740] T a13bf4d7eb1440f7a28b85cb1bf99667 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:33.857005 1166 ts_tablet_manager.cc:1434] T b772025852794a49b072f37cccbd3291 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20251025 07:52:33.857021 1167 consensus_queue.cc:260] T a13bf4d7eb1440f7a28b85cb1bf99667 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.861145 1167 ts_tablet_manager.cc:1434] T a13bf4d7eb1440f7a28b85cb1bf99667 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251025 07:52:33.862895 1166 tablet_bootstrap.cc:492] T d37758f606344127b3db74c87babe69c P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:33.863873 1169 tablet_bootstrap.cc:492] T f3c87a83082b421cb4be546ac4ce7ad8 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:33.864068 1169 ts_tablet_manager.cc:1403] T f3c87a83082b421cb4be546ac4ce7ad8 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.066s user 0.005s sys 0.001s
I20251025 07:52:33.864882 1169 raft_consensus.cc:359] T f3c87a83082b421cb4be546ac4ce7ad8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.865103 1169 raft_consensus.cc:385] T f3c87a83082b421cb4be546ac4ce7ad8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:33.865111 1659 raft_consensus.cc:493] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:33.865190 1169 raft_consensus.cc:740] T f3c87a83082b421cb4be546ac4ce7ad8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:33.865360 1166 tablet_bootstrap.cc:654] T d37758f606344127b3db74c87babe69c P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:33.865242 1659 raft_consensus.cc:515] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.865525 1169 consensus_queue.cc:260] T f3c87a83082b421cb4be546ac4ce7ad8 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.865955 1659 leader_election.cc:290] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:33.866070 1169 ts_tablet_manager.cc:1434] T f3c87a83082b421cb4be546ac4ce7ad8 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
W20251025 07:52:33.868347 710 leader_election.cc:343] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: ac0ea298bc114c56a8093e12def21ce1
I20251025 07:52:33.868685 1167 tablet_bootstrap.cc:492] T f3c87a83082b421cb4be546ac4ce7ad8 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:33.869758 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac0ea298bc114c56a8093e12def21ce1" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:33.869961 1048 raft_consensus.cc:2468] T ac0ea298bc114c56a8093e12def21ce1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:33.870519 712 leader_election.cc:304] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters: 3aea2869ff814cc7b142012faa274382
I20251025 07:52:33.870798 1659 raft_consensus.cc:2804] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:33.870893 1659 raft_consensus.cc:493] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:33.870968 1659 raft_consensus.cc:3060] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:33.871230 1169 tablet_bootstrap.cc:492] T a13bf4d7eb1440f7a28b85cb1bf99667 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:33.872977 1169 tablet_bootstrap.cc:654] T a13bf4d7eb1440f7a28b85cb1bf99667 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:33.873884 1659 raft_consensus.cc:515] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.874918 1172 ts_tablet_manager.cc:1929] T 08da9d2687704725aacd11cd92c6a9f9 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:33.874931 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac0ea298bc114c56a8093e12def21ce1" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:33.875092 1048 raft_consensus.cc:3060] T ac0ea298bc114c56a8093e12def21ce1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:33.875599 1569 raft_consensus.cc:493] T f3c87a83082b421cb4be546ac4ce7ad8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:33.875794 1569 raft_consensus.cc:515] T f3c87a83082b421cb4be546ac4ce7ad8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.876415 1569 leader_election.cc:290] T f3c87a83082b421cb4be546ac4ce7ad8 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:33.877043 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3c87a83082b421cb4be546ac4ce7ad8" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:33.877236 1172 log.cc:1199] T 08da9d2687704725aacd11cd92c6a9f9 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/08da9d2687704725aacd11cd92c6a9f9
I20251025 07:52:33.877456 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3c87a83082b421cb4be546ac4ce7ad8" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:33.877681 913 raft_consensus.cc:2468] T f3c87a83082b421cb4be546ac4ce7ad8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:33.877954 1172 ts_tablet_manager.cc:1950] T 08da9d2687704725aacd11cd92c6a9f9 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:33.878063 1048 raft_consensus.cc:2468] T ac0ea298bc114c56a8093e12def21ce1 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
W20251025 07:52:33.878230 982 leader_election.cc:343] T f3c87a83082b421cb4be546ac4ce7ad8 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:33.878544 712 leader_election.cc:304] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:33.878804 1441 raft_consensus.cc:2804] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:33.878896 983 leader_election.cc:304] T f3c87a83082b421cb4be546ac4ce7ad8 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b
I20251025 07:52:33.879463 1569 raft_consensus.cc:2804] T f3c87a83082b421cb4be546ac4ce7ad8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:33.879578 1569 raft_consensus.cc:493] T f3c87a83082b421cb4be546ac4ce7ad8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:33.879691 1569 raft_consensus.cc:3060] T f3c87a83082b421cb4be546ac4ce7ad8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:33.883014 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 412) took 1942 ms. Trace:
I20251025 07:52:33.883165 1172 rpcz_store.cc:276] 1025 07:52:31.940921 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:31.941012 (+ 91us) service_pool.cc:225] Handling call
1025 07:52:33.845155 (+1904143us) ts_tablet_manager.cc:1151] Deleting tablet 08da9d2687704725aacd11cd92c6a9f9
1025 07:52:33.845170 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:33.852018 (+ 6848us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:33.874763 (+ 22745us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:33.882959 (+ 8196us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1899501}
I20251025 07:52:33.883436 1569 raft_consensus.cc:515] T f3c87a83082b421cb4be546ac4ce7ad8 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.884790 1569 leader_election.cc:290] T f3c87a83082b421cb4be546ac4ce7ad8 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:33.885536 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3c87a83082b421cb4be546ac4ce7ad8" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:33.885516 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3c87a83082b421cb4be546ac4ce7ad8" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:33.885751 913 raft_consensus.cc:3060] T f3c87a83082b421cb4be546ac4ce7ad8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:33.886171 710 leader_election.cc:343] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: ac0ea298bc114c56a8093e12def21ce1
I20251025 07:52:33.885512 1659 leader_election.cc:290] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:33.886348 1172 tablet_replica.cc:333] T 21e52928580c4762b0ae7152490ef955 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:33.886570 1172 raft_consensus.cc:2243] T 21e52928580c4762b0ae7152490ef955 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:33.887241 1172 raft_consensus.cc:2272] T 21e52928580c4762b0ae7152490ef955 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:33.888237 1172 ts_tablet_manager.cc:1916] T 21e52928580c4762b0ae7152490ef955 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:33.890892 913 raft_consensus.cc:2468] T f3c87a83082b421cb4be546ac4ce7ad8 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:33.892935 1166 tablet_bootstrap.cc:492] T d37758f606344127b3db74c87babe69c P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:33.893134 1166 ts_tablet_manager.cc:1403] T d37758f606344127b3db74c87babe69c P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.007s
I20251025 07:52:33.893931 1166 raft_consensus.cc:359] T d37758f606344127b3db74c87babe69c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.894174 1166 raft_consensus.cc:385] T d37758f606344127b3db74c87babe69c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:33.894304 1166 raft_consensus.cc:740] T d37758f606344127b3db74c87babe69c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:33.894583 1166 consensus_queue.cc:260] T d37758f606344127b3db74c87babe69c P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.889114 1169 tablet_bootstrap.cc:492] T a13bf4d7eb1440f7a28b85cb1bf99667 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:33.889382 1612 raft_consensus.cc:493] T 08da9d2687704725aacd11cd92c6a9f9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:33.899165 1612 raft_consensus.cc:515] T 08da9d2687704725aacd11cd92c6a9f9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.899546 1167 tablet_bootstrap.cc:654] T f3c87a83082b421cb4be546ac4ce7ad8 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:33.899788 1612 leader_election.cc:290] T 08da9d2687704725aacd11cd92c6a9f9 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:33.889703 982 leader_election.cc:343] T f3c87a83082b421cb4be546ac4ce7ad8 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:33.900157 983 leader_election.cc:304] T f3c87a83082b421cb4be546ac4ce7ad8 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b
I20251025 07:52:33.900723 1169 ts_tablet_manager.cc:1403] T a13bf4d7eb1440f7a28b85cb1bf99667 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20251025 07:52:33.900662 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "08da9d2687704725aacd11cd92c6a9f9" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:33.900929 775 raft_consensus.cc:2468] T 08da9d2687704725aacd11cd92c6a9f9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:33.901397 982 leader_election.cc:304] T 08da9d2687704725aacd11cd92c6a9f9 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:33.901860 1612 raft_consensus.cc:2804] T f3c87a83082b421cb4be546ac4ce7ad8 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
W20251025 07:52:33.902397 983 leader_election.cc:343] T 08da9d2687704725aacd11cd92c6a9f9 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 08da9d2687704725aacd11cd92c6a9f9
I20251025 07:52:33.902616 1169 raft_consensus.cc:359] T a13bf4d7eb1440f7a28b85cb1bf99667 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.902788 1169 raft_consensus.cc:385] T a13bf4d7eb1440f7a28b85cb1bf99667 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:33.902889 1169 raft_consensus.cc:740] T a13bf4d7eb1440f7a28b85cb1bf99667 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:33.903137 1169 consensus_queue.cc:260] T a13bf4d7eb1440f7a28b85cb1bf99667 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.903457 1569 raft_consensus.cc:2804] T 08da9d2687704725aacd11cd92c6a9f9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:33.903715 1166 ts_tablet_manager.cc:1434] T d37758f606344127b3db74c87babe69c P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20251025 07:52:33.903712 1569 raft_consensus.cc:493] T 08da9d2687704725aacd11cd92c6a9f9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:33.903983 1569 raft_consensus.cc:3060] T 08da9d2687704725aacd11cd92c6a9f9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:33.904142 1166 tablet_bootstrap.cc:492] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:33.909070 1569 raft_consensus.cc:515] T 08da9d2687704725aacd11cd92c6a9f9 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.911250 1166 tablet_bootstrap.cc:654] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:33.911748 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "08da9d2687704725aacd11cd92c6a9f9" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:33.912019 775 raft_consensus.cc:3060] T 08da9d2687704725aacd11cd92c6a9f9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:33.914597 983 leader_election.cc:343] T 08da9d2687704725aacd11cd92c6a9f9 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 08da9d2687704725aacd11cd92c6a9f9
I20251025 07:52:33.916322 775 raft_consensus.cc:2468] T 08da9d2687704725aacd11cd92c6a9f9 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:33.916908 1569 leader_election.cc:290] T 08da9d2687704725aacd11cd92c6a9f9 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:33.916949 982 leader_election.cc:304] T 08da9d2687704725aacd11cd92c6a9f9 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters: 3aea2869ff814cc7b142012faa274382
I20251025 07:52:33.918674 1172 ts_tablet_manager.cc:1929] T 21e52928580c4762b0ae7152490ef955 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:33.918915 1172 log.cc:1199] T 21e52928580c4762b0ae7152490ef955 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/21e52928580c4762b0ae7152490ef955
I20251025 07:52:33.920009 1659 raft_consensus.cc:493] T 8e6d76b1a97945cfb6c3055e1b622f8a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:33.920245 1659 raft_consensus.cc:515] T 8e6d76b1a97945cfb6c3055e1b622f8a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.921787 1659 leader_election.cc:290] T 8e6d76b1a97945cfb6c3055e1b622f8a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:33.922456 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e6d76b1a97945cfb6c3055e1b622f8a" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:33.921908 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e6d76b1a97945cfb6c3055e1b622f8a" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:33.925472 913 raft_consensus.cc:2393] T 8e6d76b1a97945cfb6c3055e1b622f8a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 23eeffd3de254ce18db5febfde971c6b in current term 1: Already voted for candidate db42be7ad3c741a0aa427f27e245993b in this term.
I20251025 07:52:33.926074 710 leader_election.cc:304] T 8e6d76b1a97945cfb6c3055e1b622f8a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:33.926442 1659 raft_consensus.cc:3060] T 8e6d76b1a97945cfb6c3055e1b622f8a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:33.927064 1172 ts_tablet_manager.cc:1950] T 21e52928580c4762b0ae7152490ef955 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:33.928051 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 410) took 1992 ms. Trace:
I20251025 07:52:33.928520 1673 raft_consensus.cc:493] T a13bf4d7eb1440f7a28b85cb1bf99667 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:33.928673 1172 rpcz_store.cc:276] 1025 07:52:31.935282 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:31.935364 (+ 82us) service_pool.cc:225] Handling call
1025 07:52:33.886278 (+1950914us) ts_tablet_manager.cc:1151] Deleting tablet 21e52928580c4762b0ae7152490ef955
1025 07:52:33.886288 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:33.901939 (+ 15651us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:33.918482 (+ 16543us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:33.927974 (+ 9492us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1940585}
I20251025 07:52:33.928570 1675 raft_consensus.cc:2804] T 08da9d2687704725aacd11cd92c6a9f9 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:33.929091 1675 raft_consensus.cc:697] T 08da9d2687704725aacd11cd92c6a9f9 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:33.929221 1172 tablet_replica.cc:333] T f3c87a83082b421cb4be546ac4ce7ad8 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:33.929412 1172 raft_consensus.cc:2243] T f3c87a83082b421cb4be546ac4ce7ad8 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:33.929531 1659 raft_consensus.cc:2749] T 8e6d76b1a97945cfb6c3055e1b622f8a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:33.928736 1673 raft_consensus.cc:515] T a13bf4d7eb1440f7a28b85cb1bf99667 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.929602 1172 raft_consensus.cc:2272] T f3c87a83082b421cb4be546ac4ce7ad8 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:33.931151 1675 consensus_queue.cc:237] T 08da9d2687704725aacd11cd92c6a9f9 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.932009 1172 ts_tablet_manager.cc:1916] T f3c87a83082b421cb4be546ac4ce7ad8 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:33.932986 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a13bf4d7eb1440f7a28b85cb1bf99667" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:33.933125 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a13bf4d7eb1440f7a28b85cb1bf99667" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:33.933315 913 raft_consensus.cc:2468] T a13bf4d7eb1440f7a28b85cb1bf99667 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:33.933537 775 raft_consensus.cc:2468] T a13bf4d7eb1440f7a28b85cb1bf99667 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:33.933957 1673 leader_election.cc:290] T a13bf4d7eb1440f7a28b85cb1bf99667 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:33.934028 983 leader_election.cc:304] T a13bf4d7eb1440f7a28b85cb1bf99667 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:33.937569 1166 tablet_bootstrap.cc:492] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:33.938671 1166 ts_tablet_manager.cc:1403] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.006s
I20251025 07:52:33.939483 1672 raft_consensus.cc:2804] T a13bf4d7eb1440f7a28b85cb1bf99667 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:33.939600 1672 raft_consensus.cc:493] T a13bf4d7eb1440f7a28b85cb1bf99667 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:33.939747 1672 raft_consensus.cc:3060] T a13bf4d7eb1440f7a28b85cb1bf99667 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:33.940172 1169 ts_tablet_manager.cc:1434] T a13bf4d7eb1440f7a28b85cb1bf99667 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.039s user 0.001s sys 0.009s
I20251025 07:52:33.940606 1169 tablet_bootstrap.cc:492] T fab67b56eaa44b54886c17181d8c52b2 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:33.942977 1672 raft_consensus.cc:515] T a13bf4d7eb1440f7a28b85cb1bf99667 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.943531 1167 tablet_bootstrap.cc:492] T f3c87a83082b421cb4be546ac4ce7ad8 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:33.943713 1672 leader_election.cc:290] T a13bf4d7eb1440f7a28b85cb1bf99667 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:33.943764 1167 ts_tablet_manager.cc:1403] T f3c87a83082b421cb4be546ac4ce7ad8 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.075s user 0.004s sys 0.000s
I20251025 07:52:33.944159 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a13bf4d7eb1440f7a28b85cb1bf99667" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:33.944327 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a13bf4d7eb1440f7a28b85cb1bf99667" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:33.944442 775 raft_consensus.cc:3060] T a13bf4d7eb1440f7a28b85cb1bf99667 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:33.944548 913 raft_consensus.cc:3060] T a13bf4d7eb1440f7a28b85cb1bf99667 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:33.944780 1167 raft_consensus.cc:359] T f3c87a83082b421cb4be546ac4ce7ad8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.945415 1167 raft_consensus.cc:385] T f3c87a83082b421cb4be546ac4ce7ad8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:33.945683 1167 raft_consensus.cc:740] T f3c87a83082b421cb4be546ac4ce7ad8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:33.946359 1167 consensus_queue.cc:260] T f3c87a83082b421cb4be546ac4ce7ad8 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.947717 1167 ts_tablet_manager.cc:1434] T f3c87a83082b421cb4be546ac4ce7ad8 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251025 07:52:33.947726 913 raft_consensus.cc:2468] T a13bf4d7eb1440f7a28b85cb1bf99667 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:33.948237 983 leader_election.cc:304] T a13bf4d7eb1440f7a28b85cb1bf99667 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:33.948448 1167 tablet_bootstrap.cc:492] T fab67b56eaa44b54886c17181d8c52b2 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:33.948439 1166 raft_consensus.cc:359] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.948637 1166 raft_consensus.cc:385] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:33.948727 1166 raft_consensus.cc:740] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:33.948961 1166 consensus_queue.cc:260] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.949484 1166 ts_tablet_manager.cc:1434] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.010s user 0.000s sys 0.000s
I20251025 07:52:33.949764 1169 tablet_bootstrap.cc:654] T fab67b56eaa44b54886c17181d8c52b2 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:33.950184 1167 tablet_bootstrap.cc:654] T fab67b56eaa44b54886c17181d8c52b2 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:33.952896 775 raft_consensus.cc:2468] T a13bf4d7eb1440f7a28b85cb1bf99667 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:33.953843 1166 tablet_bootstrap.cc:492] T 35ba9d23262248cda6fdf03c51be11a4 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:33.954869 1671 raft_consensus.cc:2804] T a13bf4d7eb1440f7a28b85cb1bf99667 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:33.955052 1671 raft_consensus.cc:697] T a13bf4d7eb1440f7a28b85cb1bf99667 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:33.955469 1671 consensus_queue.cc:237] T a13bf4d7eb1440f7a28b85cb1bf99667 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.955942 1612 raft_consensus.cc:697] T f3c87a83082b421cb4be546ac4ce7ad8 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:33.956254 1612 consensus_queue.cc:237] T f3c87a83082b421cb4be546ac4ce7ad8 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.959118 1166 tablet_bootstrap.cc:654] T 35ba9d23262248cda6fdf03c51be11a4 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:33.960027 1441 raft_consensus.cc:697] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:33.960319 1441 consensus_queue.cc:237] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.966300 1167 tablet_bootstrap.cc:492] T fab67b56eaa44b54886c17181d8c52b2 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:33.966482 1167 ts_tablet_manager.cc:1403] T fab67b56eaa44b54886c17181d8c52b2 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.001s
I20251025 07:52:33.967403 1167 raft_consensus.cc:359] T fab67b56eaa44b54886c17181d8c52b2 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.967602 1167 raft_consensus.cc:385] T fab67b56eaa44b54886c17181d8c52b2 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:33.967669 1167 raft_consensus.cc:740] T fab67b56eaa44b54886c17181d8c52b2 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:33.968860 1167 consensus_queue.cc:260] T fab67b56eaa44b54886c17181d8c52b2 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.969362 1167 ts_tablet_manager.cc:1434] T fab67b56eaa44b54886c17181d8c52b2 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251025 07:52:33.969672 1167 tablet_bootstrap.cc:492] T 4037dee385d04e85bb78c8e77cb00379 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:33.977387 1167 tablet_bootstrap.cc:654] T 4037dee385d04e85bb78c8e77cb00379 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:33.979444 1172 ts_tablet_manager.cc:1929] T f3c87a83082b421cb4be546ac4ce7ad8 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:33.979647 1172 log.cc:1199] T f3c87a83082b421cb4be546ac4ce7ad8 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/f3c87a83082b421cb4be546ac4ce7ad8
I20251025 07:52:33.982059 1172 ts_tablet_manager.cc:1950] T f3c87a83082b421cb4be546ac4ce7ad8 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:33.983135 1166 tablet_bootstrap.cc:492] T 35ba9d23262248cda6fdf03c51be11a4 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:33.983348 1166 ts_tablet_manager.cc:1403] T 35ba9d23262248cda6fdf03c51be11a4 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.000s
I20251025 07:52:33.983410 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 413) took 2037 ms. Trace:
I20251025 07:52:33.983588 1172 rpcz_store.cc:276] 1025 07:52:31.946172 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:31.946232 (+ 60us) service_pool.cc:225] Handling call
1025 07:52:33.929117 (+1982885us) ts_tablet_manager.cc:1151] Deleting tablet f3c87a83082b421cb4be546ac4ce7ad8
1025 07:52:33.929127 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:33.936367 (+ 7240us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:33.979288 (+ 42921us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:33.983355 (+ 4067us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1978496}
I20251025 07:52:33.984395 1166 raft_consensus.cc:359] T 35ba9d23262248cda6fdf03c51be11a4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:33.984622 1166 raft_consensus.cc:385] T 35ba9d23262248cda6fdf03c51be11a4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:33.984725 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 421) took 2007 ms. Trace:
I20251025 07:52:33.984725 1166 raft_consensus.cc:740] T 35ba9d23262248cda6fdf03c51be11a4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:33.984995 1169 tablet_bootstrap.cc:492] T fab67b56eaa44b54886c17181d8c52b2 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:33.985258 1169 ts_tablet_manager.cc:1403] T fab67b56eaa44b54886c17181d8c52b2 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.045s user 0.006s sys 0.002s
I20251025 07:52:33.985466 1172 rpcz_store.cc:276] 1025 07:52:31.977212 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:31.977329 (+ 117us) service_pool.cc:225] Handling call
1025 07:52:33.984529 (+2007200us) ts_tablet_manager.cc:1151] Deleting tablet 3b940fa5ae26427dafbcfd26372754b3
1025 07:52:33.984538 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:33.984683 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2005370}
I20251025 07:52:33.985556 1166 consensus_queue.cc:260] T 35ba9d23262248cda6fdf03c51be11a4 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:33.986004 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 436) took 1955 ms. Trace:
I20251025 07:52:33.986063 1169 raft_consensus.cc:359] T fab67b56eaa44b54886c17181d8c52b2 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.986296 1169 raft_consensus.cc:385] T fab67b56eaa44b54886c17181d8c52b2 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:33.986382 1169 raft_consensus.cc:740] T fab67b56eaa44b54886c17181d8c52b2 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:33.986198 1172 rpcz_store.cc:276] 1025 07:52:32.030892 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:32.155247 (+124355us) service_pool.cc:225] Handling call
1025 07:52:33.985869 (+1830622us) ts_tablet_manager.cc:1151] Deleting tablet f4578ce69a0d4322944e5ce7c8a59831
1025 07:52:33.985878 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:33.985967 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1745510}
I20251025 07:52:33.986636 1169 consensus_queue.cc:260] T fab67b56eaa44b54886c17181d8c52b2 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.987142 1169 ts_tablet_manager.cc:1434] T fab67b56eaa44b54886c17181d8c52b2 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:33.987690 1169 tablet_bootstrap.cc:492] T 467324e2432f4caca148951329caa292 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:33.990797 1169 tablet_bootstrap.cc:654] T 467324e2432f4caca148951329caa292 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:33.990975 1166 ts_tablet_manager.cc:1434] T 35ba9d23262248cda6fdf03c51be11a4 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251025 07:52:33.991359 1166 tablet_bootstrap.cc:492] T 27651312cc2e42a8bfcd245f5d301679 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:33.991794 1671 raft_consensus.cc:493] T 21e52928580c4762b0ae7152490ef955 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:33.991950 1671 raft_consensus.cc:515] T 21e52928580c4762b0ae7152490ef955 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.992635 1671 leader_election.cc:290] T 21e52928580c4762b0ae7152490ef955 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:33.994118 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e52928580c4762b0ae7152490ef955" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:33.994372 775 raft_consensus.cc:2468] T 21e52928580c4762b0ae7152490ef955 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
W20251025 07:52:33.995003 983 leader_election.cc:343] T 21e52928580c4762b0ae7152490ef955 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 21e52928580c4762b0ae7152490ef955
I20251025 07:52:33.995275 982 leader_election.cc:304] T 21e52928580c4762b0ae7152490ef955 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters: 3aea2869ff814cc7b142012faa274382
I20251025 07:52:33.995713 1671 raft_consensus.cc:2804] T 21e52928580c4762b0ae7152490ef955 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:33.995816 1671 raft_consensus.cc:493] T 21e52928580c4762b0ae7152490ef955 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:33.995931 1671 raft_consensus.cc:3060] T 21e52928580c4762b0ae7152490ef955 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:33.997615 1166 tablet_bootstrap.cc:654] T 27651312cc2e42a8bfcd245f5d301679 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:33.993239 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 440) took 1027 ms. Trace:
I20251025 07:52:33.999112 1671 raft_consensus.cc:515] T 21e52928580c4762b0ae7152490ef955 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:33.999830 1671 leader_election.cc:290] T 21e52928580c4762b0ae7152490ef955 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:34.000088 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e52928580c4762b0ae7152490ef955" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:34.000301 1167 tablet_bootstrap.cc:492] T 4037dee385d04e85bb78c8e77cb00379 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:34.000349 775 raft_consensus.cc:3060] T 21e52928580c4762b0ae7152490ef955 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.000460 1167 ts_tablet_manager.cc:1403] T 4037dee385d04e85bb78c8e77cb00379 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.001s
I20251025 07:52:34.001179 1167 raft_consensus.cc:359] T 4037dee385d04e85bb78c8e77cb00379 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.001338 1167 raft_consensus.cc:385] T 4037dee385d04e85bb78c8e77cb00379 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.001423 1167 raft_consensus.cc:740] T 4037dee385d04e85bb78c8e77cb00379 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.001659 1167 consensus_queue.cc:260] T 4037dee385d04e85bb78c8e77cb00379 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.002352 1167 ts_tablet_manager.cc:1434] T 4037dee385d04e85bb78c8e77cb00379 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:34.003050 1167 tablet_bootstrap.cc:492] T 467324e2432f4caca148951329caa292 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:34.003352 775 raft_consensus.cc:2468] T 21e52928580c4762b0ae7152490ef955 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:34.003931 982 leader_election.cc:304] T 21e52928580c4762b0ae7152490ef955 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:34.004202 1671 raft_consensus.cc:2804] T 21e52928580c4762b0ae7152490ef955 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:34.004424 1671 raft_consensus.cc:697] T 21e52928580c4762b0ae7152490ef955 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:34.005154 1671 consensus_queue.cc:237] T 21e52928580c4762b0ae7152490ef955 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.006017 1167 tablet_bootstrap.cc:654] T 467324e2432f4caca148951329caa292 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:34.008785 983 leader_election.cc:343] T 21e52928580c4762b0ae7152490ef955 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 21e52928580c4762b0ae7152490ef955
I20251025 07:52:34.012548 1172 rpcz_store.cc:276] 1025 07:52:32.965784 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:32.965847 (+ 63us) service_pool.cc:225] Handling call
1025 07:52:33.986780 (+1020933us) ts_tablet_manager.cc:1151] Deleting tablet 26c849d376d143c6bef0c8e0e1d7d03a
1025 07:52:33.986856 (+ 76us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:33.993193 (+ 6337us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1020655}
I20251025 07:52:34.013756 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 441) took 1047 ms. Trace:
I20251025 07:52:34.014374 1172 rpcz_store.cc:276] 1025 07:52:32.966117 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:32.966202 (+ 85us) service_pool.cc:225] Handling call
1025 07:52:34.013525 (+1047323us) ts_tablet_manager.cc:1151] Deleting tablet 4f7c15529be54a11961f385a953b0b03
1025 07:52:34.013540 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:34.013716 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1047061}
I20251025 07:52:34.015304 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 442) took 1049 ms. Trace:
I20251025 07:52:34.017297 1167 tablet_bootstrap.cc:492] T 467324e2432f4caca148951329caa292 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:34.017462 1167 ts_tablet_manager.cc:1403] T 467324e2432f4caca148951329caa292 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.001s
I20251025 07:52:34.019243 1167 raft_consensus.cc:359] T 467324e2432f4caca148951329caa292 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.019436 1167 raft_consensus.cc:385] T 467324e2432f4caca148951329caa292 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.019497 1167 raft_consensus.cc:740] T 467324e2432f4caca148951329caa292 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.019783 1167 consensus_queue.cc:260] T 467324e2432f4caca148951329caa292 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.024097 1167 ts_tablet_manager.cc:1434] T 467324e2432f4caca148951329caa292 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251025 07:52:34.027396 1166 tablet_bootstrap.cc:492] T 27651312cc2e42a8bfcd245f5d301679 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:34.027629 1166 ts_tablet_manager.cc:1403] T 27651312cc2e42a8bfcd245f5d301679 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.036s user 0.000s sys 0.009s
I20251025 07:52:34.028146 1167 tablet_bootstrap.cc:492] T b772025852794a49b072f37cccbd3291 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:34.028772 1166 raft_consensus.cc:359] T 27651312cc2e42a8bfcd245f5d301679 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.029081 1166 raft_consensus.cc:385] T 27651312cc2e42a8bfcd245f5d301679 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.029220 1166 raft_consensus.cc:740] T 27651312cc2e42a8bfcd245f5d301679 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.029546 1166 consensus_queue.cc:260] T 27651312cc2e42a8bfcd245f5d301679 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.030169 1166 ts_tablet_manager.cc:1434] T 27651312cc2e42a8bfcd245f5d301679 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251025 07:52:34.030532 1166 tablet_bootstrap.cc:492] T 4f25c9caa05b4da4bd56a8a21fde423c P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:34.032829 1166 tablet_bootstrap.cc:654] T 4f25c9caa05b4da4bd56a8a21fde423c P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.030164 1172 rpcz_store.cc:276] 1025 07:52:32.966304 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:32.966349 (+ 45us) service_pool.cc:225] Handling call
1025 07:52:34.015155 (+1048806us) ts_tablet_manager.cc:1151] Deleting tablet ce1ebb1ec14b4801930a72765bca1a7d
1025 07:52:34.015166 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:34.015260 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1048641}
I20251025 07:52:34.035041 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 443) took 1068 ms. Trace:
I20251025 07:52:34.035281 1172 rpcz_store.cc:276] 1025 07:52:32.966427 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:32.966490 (+ 63us) service_pool.cc:225] Handling call
1025 07:52:34.034850 (+1068360us) ts_tablet_manager.cc:1151] Deleting tablet e4da89f863cc43f3bf6df908bb9dc6bf
1025 07:52:34.034864 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:34.035002 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1068204}
I20251025 07:52:34.035988 1172 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56590 (request call id 444) took 1069 ms. Trace:
I20251025 07:52:34.036198 1172 rpcz_store.cc:276] 1025 07:52:32.966519 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:32.966555 (+ 36us) service_pool.cc:225] Handling call
1025 07:52:34.035823 (+1069268us) ts_tablet_manager.cc:1151] Deleting tablet d77704b4fe6c4456a2ba40ffec0c60fc
1025 07:52:34.035836 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:34.035942 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1069104}
I20251025 07:52:34.038568 1167 tablet_bootstrap.cc:654] T b772025852794a49b072f37cccbd3291 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.040844 1169 tablet_bootstrap.cc:492] T 467324e2432f4caca148951329caa292 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:34.041083 1169 ts_tablet_manager.cc:1403] T 467324e2432f4caca148951329caa292 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.053s user 0.014s sys 0.012s
I20251025 07:52:34.042104 1169 raft_consensus.cc:359] T 467324e2432f4caca148951329caa292 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.042423 1169 raft_consensus.cc:385] T 467324e2432f4caca148951329caa292 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.042577 1169 raft_consensus.cc:740] T 467324e2432f4caca148951329caa292 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.042882 1169 consensus_queue.cc:260] T 467324e2432f4caca148951329caa292 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.043535 1169 ts_tablet_manager.cc:1434] T 467324e2432f4caca148951329caa292 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:34.043849 1169 tablet_bootstrap.cc:492] T 4037dee385d04e85bb78c8e77cb00379 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:34.050063 1169 tablet_bootstrap.cc:654] T 4037dee385d04e85bb78c8e77cb00379 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.052002 1167 tablet_bootstrap.cc:492] T b772025852794a49b072f37cccbd3291 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:34.052184 1167 ts_tablet_manager.cc:1403] T b772025852794a49b072f37cccbd3291 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.005s
I20251025 07:52:34.053210 1167 raft_consensus.cc:359] T b772025852794a49b072f37cccbd3291 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:34.053450 1167 raft_consensus.cc:385] T b772025852794a49b072f37cccbd3291 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.053575 1167 raft_consensus.cc:740] T b772025852794a49b072f37cccbd3291 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.053860 1167 consensus_queue.cc:260] T b772025852794a49b072f37cccbd3291 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:34.058686 1167 ts_tablet_manager.cc:1434] T b772025852794a49b072f37cccbd3291 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20251025 07:52:34.058925 1166 tablet_bootstrap.cc:492] T 4f25c9caa05b4da4bd56a8a21fde423c P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:34.059091 1166 ts_tablet_manager.cc:1403] T 4f25c9caa05b4da4bd56a8a21fde423c P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.008s
I20251025 07:52:34.059862 1166 raft_consensus.cc:359] T 4f25c9caa05b4da4bd56a8a21fde423c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:34.060055 1166 raft_consensus.cc:385] T 4f25c9caa05b4da4bd56a8a21fde423c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.060135 1166 raft_consensus.cc:740] T 4f25c9caa05b4da4bd56a8a21fde423c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.060371 1166 consensus_queue.cc:260] T 4f25c9caa05b4da4bd56a8a21fde423c P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:34.060882 1166 ts_tablet_manager.cc:1434] T 4f25c9caa05b4da4bd56a8a21fde423c P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:34.061201 1166 tablet_bootstrap.cc:492] T 6897405bbc1c47a4ab0aa3af80e60e3d P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:34.061496 1169 tablet_bootstrap.cc:492] T 4037dee385d04e85bb78c8e77cb00379 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:34.061656 1169 ts_tablet_manager.cc:1403] T 4037dee385d04e85bb78c8e77cb00379 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.007s
I20251025 07:52:34.062695 1169 raft_consensus.cc:359] T 4037dee385d04e85bb78c8e77cb00379 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.062985 1169 raft_consensus.cc:385] T 4037dee385d04e85bb78c8e77cb00379 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.063090 1169 raft_consensus.cc:740] T 4037dee385d04e85bb78c8e77cb00379 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.064524 1166 tablet_bootstrap.cc:654] T 6897405bbc1c47a4ab0aa3af80e60e3d P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.064893 1169 consensus_queue.cc:260] T 4037dee385d04e85bb78c8e77cb00379 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.068552 1169 ts_tablet_manager.cc:1434] T 4037dee385d04e85bb78c8e77cb00379 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251025 07:52:34.071344 1169 tablet_bootstrap.cc:492] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:34.073179 1169 tablet_bootstrap.cc:654] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.078881 1167 tablet_bootstrap.cc:492] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:34.081782 1167 tablet_bootstrap.cc:654] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.084940 1169 tablet_bootstrap.cc:492] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:34.085053 1166 tablet_bootstrap.cc:492] T 6897405bbc1c47a4ab0aa3af80e60e3d P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:34.085168 1169 ts_tablet_manager.cc:1403] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20251025 07:52:34.085644 1166 ts_tablet_manager.cc:1403] T 6897405bbc1c47a4ab0aa3af80e60e3d P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.010s
I20251025 07:52:34.086004 1169 raft_consensus.cc:359] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.086277 1169 raft_consensus.cc:385] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.086397 1169 raft_consensus.cc:740] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.086714 1169 consensus_queue.cc:260] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.086777 1166 raft_consensus.cc:359] T 6897405bbc1c47a4ab0aa3af80e60e3d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.087119 1166 raft_consensus.cc:385] T 6897405bbc1c47a4ab0aa3af80e60e3d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.087256 1166 raft_consensus.cc:740] T 6897405bbc1c47a4ab0aa3af80e60e3d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.087533 1166 consensus_queue.cc:260] T 6897405bbc1c47a4ab0aa3af80e60e3d P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.090384 1169 ts_tablet_manager.cc:1434] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251025 07:52:34.092804 1166 ts_tablet_manager.cc:1434] T 6897405bbc1c47a4ab0aa3af80e60e3d P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251025 07:52:34.094175 1167 tablet_bootstrap.cc:492] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:34.094256 1169 tablet_bootstrap.cc:492] T d37758f606344127b3db74c87babe69c P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:34.094365 1167 ts_tablet_manager.cc:1403] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.004s
I20251025 07:52:34.094539 1218 raft_consensus.cc:493] T 4f25c9caa05b4da4bd56a8a21fde423c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:34.094689 1218 raft_consensus.cc:515] T 4f25c9caa05b4da4bd56a8a21fde423c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:34.095161 1167 raft_consensus.cc:359] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.095321 1218 leader_election.cc:290] T 4f25c9caa05b4da4bd56a8a21fde423c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:34.095961 1167 raft_consensus.cc:385] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.096071 1167 raft_consensus.cc:740] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.096139 1169 tablet_bootstrap.cc:654] T d37758f606344127b3db74c87babe69c P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.096191 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f25c9caa05b4da4bd56a8a21fde423c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:34.096230 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f25c9caa05b4da4bd56a8a21fde423c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:34.096378 1167 consensus_queue.cc:260] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:34.097060 847 leader_election.cc:343] T 4f25c9caa05b4da4bd56a8a21fde423c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:34.097265 850 leader_election.cc:343] T 4f25c9caa05b4da4bd56a8a21fde423c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:34.097421 850 leader_election.cc:304] T 4f25c9caa05b4da4bd56a8a21fde423c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:34.097849 1218 raft_consensus.cc:2749] T 4f25c9caa05b4da4bd56a8a21fde423c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:34.098686 1166 tablet_bootstrap.cc:492] T 7f29b64704e147aa9b0b5f6cda860f09 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:34.100997 1166 tablet_bootstrap.cc:654] T 7f29b64704e147aa9b0b5f6cda860f09 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.101143 1167 ts_tablet_manager.cc:1434] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20251025 07:52:34.105508 1167 tablet_bootstrap.cc:492] T 8c76d1b509f04ab1a10b83cff675c11e P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:34.107550 1167 tablet_bootstrap.cc:654] T 8c76d1b509f04ab1a10b83cff675c11e P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.113410 1671 consensus_queue.cc:1048] T 8e6d76b1a97945cfb6c3055e1b622f8a P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:34.115428 1166 tablet_bootstrap.cc:492] T 7f29b64704e147aa9b0b5f6cda860f09 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:34.115774 1166 ts_tablet_manager.cc:1403] T 7f29b64704e147aa9b0b5f6cda860f09 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.005s
I20251025 07:52:34.116904 1166 raft_consensus.cc:359] T 7f29b64704e147aa9b0b5f6cda860f09 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.117200 1166 raft_consensus.cc:385] T 7f29b64704e147aa9b0b5f6cda860f09 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.117336 1166 raft_consensus.cc:740] T 7f29b64704e147aa9b0b5f6cda860f09 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.117455 1167 tablet_bootstrap.cc:492] T 8c76d1b509f04ab1a10b83cff675c11e P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:34.117609 1167 ts_tablet_manager.cc:1403] T 8c76d1b509f04ab1a10b83cff675c11e P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.001s
I20251025 07:52:34.117805 1166 consensus_queue.cc:260] T 7f29b64704e147aa9b0b5f6cda860f09 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.117935 1169 tablet_bootstrap.cc:492] T d37758f606344127b3db74c87babe69c P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:34.118525 1166 ts_tablet_manager.cc:1434] T 7f29b64704e147aa9b0b5f6cda860f09 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251025 07:52:34.118448 1167 raft_consensus.cc:359] T 8c76d1b509f04ab1a10b83cff675c11e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.118642 1167 raft_consensus.cc:385] T 8c76d1b509f04ab1a10b83cff675c11e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.118657 1169 ts_tablet_manager.cc:1403] T d37758f606344127b3db74c87babe69c P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.024s user 0.010s sys 0.000s
I20251025 07:52:34.118752 1167 raft_consensus.cc:740] T 8c76d1b509f04ab1a10b83cff675c11e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.119033 1166 tablet_bootstrap.cc:492] T 2f26667f053a4f16b5da330c0b5e40f9 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:34.119004 1167 consensus_queue.cc:260] T 8c76d1b509f04ab1a10b83cff675c11e P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.123379 1169 raft_consensus.cc:359] T d37758f606344127b3db74c87babe69c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.123481 1167 ts_tablet_manager.cc:1434] T 8c76d1b509f04ab1a10b83cff675c11e P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251025 07:52:34.123981 1169 raft_consensus.cc:385] T d37758f606344127b3db74c87babe69c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.124238 1169 raft_consensus.cc:740] T d37758f606344127b3db74c87babe69c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.124843 1169 consensus_queue.cc:260] T d37758f606344127b3db74c87babe69c P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.128911 1169 ts_tablet_manager.cc:1434] T d37758f606344127b3db74c87babe69c P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251025 07:52:34.130427 1166 tablet_bootstrap.cc:654] T 2f26667f053a4f16b5da330c0b5e40f9 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.130888 1441 raft_consensus.cc:493] T 7e869c65771f4339a546dfeebb227c62 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:34.131060 1441 raft_consensus.cc:515] T 7e869c65771f4339a546dfeebb227c62 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.131793 1441 leader_election.cc:290] T 7e869c65771f4339a546dfeebb227c62 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:34.132115 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e869c65771f4339a546dfeebb227c62" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:34.132849 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e869c65771f4339a546dfeebb227c62" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:34.133416 1169 tablet_bootstrap.cc:492] T 02e4c4d6ffb34cecaf17beb5ba197e81 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:34.133571 710 leader_election.cc:304] T 7e869c65771f4339a546dfeebb227c62 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:34.134210 1441 raft_consensus.cc:2749] T 7e869c65771f4339a546dfeebb227c62 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:34.136328 1166 tablet_bootstrap.cc:492] T 2f26667f053a4f16b5da330c0b5e40f9 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:34.136574 1166 ts_tablet_manager.cc:1403] T 2f26667f053a4f16b5da330c0b5e40f9 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.006s
I20251025 07:52:34.137486 1166 raft_consensus.cc:359] T 2f26667f053a4f16b5da330c0b5e40f9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.137739 1166 raft_consensus.cc:385] T 2f26667f053a4f16b5da330c0b5e40f9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.137876 1166 raft_consensus.cc:740] T 2f26667f053a4f16b5da330c0b5e40f9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.138249 1167 tablet_bootstrap.cc:492] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:34.138223 1166 consensus_queue.cc:260] T 2f26667f053a4f16b5da330c0b5e40f9 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.138969 1166 ts_tablet_manager.cc:1434] T 2f26667f053a4f16b5da330c0b5e40f9 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:34.139314 1166 tablet_bootstrap.cc:492] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:34.140009 1167 tablet_bootstrap.cc:654] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.141605 1169 tablet_bootstrap.cc:654] T 02e4c4d6ffb34cecaf17beb5ba197e81 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.142763 1167 tablet_bootstrap.cc:492] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:34.142920 1167 ts_tablet_manager.cc:1403] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20251025 07:52:34.143653 1167 raft_consensus.cc:359] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.143887 1167 raft_consensus.cc:385] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.143967 1167 raft_consensus.cc:740] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.144302 1167 consensus_queue.cc:260] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.145375 1166 tablet_bootstrap.cc:654] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.148860 1167 ts_tablet_manager.cc:1434] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20251025 07:52:34.154667 1167 tablet_bootstrap.cc:492] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:34.156672 1167 tablet_bootstrap.cc:654] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.165305 1218 raft_consensus.cc:493] T 4037dee385d04e85bb78c8e77cb00379 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:34.165565 1218 raft_consensus.cc:515] T 4037dee385d04e85bb78c8e77cb00379 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.166407 1218 leader_election.cc:290] T 4037dee385d04e85bb78c8e77cb00379 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:34.167304 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4037dee385d04e85bb78c8e77cb00379" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:34.167511 1048 raft_consensus.cc:2468] T 4037dee385d04e85bb78c8e77cb00379 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:34.167793 1166 tablet_bootstrap.cc:492] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:34.168015 1166 ts_tablet_manager.cc:1403] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.000s
I20251025 07:52:34.168081 850 leader_election.cc:304] T 4037dee385d04e85bb78c8e77cb00379 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:34.168473 1218 raft_consensus.cc:2804] T 4037dee385d04e85bb78c8e77cb00379 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:34.168557 1218 raft_consensus.cc:493] T 4037dee385d04e85bb78c8e77cb00379 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:34.168648 1218 raft_consensus.cc:3060] T 4037dee385d04e85bb78c8e77cb00379 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.169131 1166 raft_consensus.cc:359] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:34.169420 1166 raft_consensus.cc:385] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.169516 1166 raft_consensus.cc:740] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.169811 1166 consensus_queue.cc:260] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:34.170559 1166 ts_tablet_manager.cc:1434] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251025 07:52:34.171643 1218 raft_consensus.cc:515] T 4037dee385d04e85bb78c8e77cb00379 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.173445 1166 tablet_bootstrap.cc:492] T fa77085e81c7440d9e255c44d1e80c26 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:34.174763 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4037dee385d04e85bb78c8e77cb00379" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:34.174947 1048 raft_consensus.cc:3060] T 4037dee385d04e85bb78c8e77cb00379 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.175109 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4037dee385d04e85bb78c8e77cb00379" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:34.175136 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4037dee385d04e85bb78c8e77cb00379" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:34.175387 775 raft_consensus.cc:2468] T 4037dee385d04e85bb78c8e77cb00379 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:34.177978 1048 raft_consensus.cc:2468] T 4037dee385d04e85bb78c8e77cb00379 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:34.180104 1441 raft_consensus.cc:493] T fd9b4be139d347f4a6a60767029c342a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:34.180258 1441 raft_consensus.cc:515] T fd9b4be139d347f4a6a60767029c342a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.180923 1441 leader_election.cc:290] T fd9b4be139d347f4a6a60767029c342a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:34.181481 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd9b4be139d347f4a6a60767029c342a" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:34.181766 1048 raft_consensus.cc:2468] T fd9b4be139d347f4a6a60767029c342a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:34.182210 712 leader_election.cc:304] T fd9b4be139d347f4a6a60767029c342a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:34.182420 1169 tablet_bootstrap.cc:492] T 02e4c4d6ffb34cecaf17beb5ba197e81 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:34.182560 1169 ts_tablet_manager.cc:1403] T 02e4c4d6ffb34cecaf17beb5ba197e81 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.003s
I20251025 07:52:34.182559 1441 raft_consensus.cc:2804] T fd9b4be139d347f4a6a60767029c342a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:34.182667 1441 raft_consensus.cc:493] T fd9b4be139d347f4a6a60767029c342a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:34.182745 1441 raft_consensus.cc:3060] T fd9b4be139d347f4a6a60767029c342a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.183319 1169 raft_consensus.cc:359] T 02e4c4d6ffb34cecaf17beb5ba197e81 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.183472 1169 raft_consensus.cc:385] T 02e4c4d6ffb34cecaf17beb5ba197e81 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.183550 1169 raft_consensus.cc:740] T 02e4c4d6ffb34cecaf17beb5ba197e81 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.186328 1166 tablet_bootstrap.cc:654] T fa77085e81c7440d9e255c44d1e80c26 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.186272 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd9b4be139d347f4a6a60767029c342a" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:34.186172 1441 raft_consensus.cc:515] T fd9b4be139d347f4a6a60767029c342a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.187557 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd9b4be139d347f4a6a60767029c342a" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:34.187742 1046 raft_consensus.cc:3060] T fd9b4be139d347f4a6a60767029c342a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.188369 850 leader_election.cc:304] T 4037dee385d04e85bb78c8e77cb00379 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b
I20251025 07:52:34.188846 1561 raft_consensus.cc:2804] T 4037dee385d04e85bb78c8e77cb00379 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:34.189144 914 raft_consensus.cc:2468] T fd9b4be139d347f4a6a60767029c342a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:34.189389 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd9b4be139d347f4a6a60767029c342a" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:34.189566 912 raft_consensus.cc:3060] T fd9b4be139d347f4a6a60767029c342a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.190712 1046 raft_consensus.cc:2468] T fd9b4be139d347f4a6a60767029c342a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:34.191485 712 leader_election.cc:304] T fd9b4be139d347f4a6a60767029c342a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:34.191848 1670 raft_consensus.cc:2804] T fd9b4be139d347f4a6a60767029c342a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:34.191998 1670 raft_consensus.cc:697] T fd9b4be139d347f4a6a60767029c342a P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:34.192282 1670 consensus_queue.cc:237] T fd9b4be139d347f4a6a60767029c342a P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.192729 912 raft_consensus.cc:2468] T fd9b4be139d347f4a6a60767029c342a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:34.193148 1441 leader_election.cc:290] T fd9b4be139d347f4a6a60767029c342a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:34.194923 1218 leader_election.cc:290] T 4037dee385d04e85bb78c8e77cb00379 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:34.197263 1561 raft_consensus.cc:697] T 4037dee385d04e85bb78c8e77cb00379 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:34.197626 1561 consensus_queue.cc:237] T 4037dee385d04e85bb78c8e77cb00379 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.198438 1167 tablet_bootstrap.cc:492] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:34.198586 1167 ts_tablet_manager.cc:1403] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.044s user 0.012s sys 0.004s
I20251025 07:52:34.199306 1167 raft_consensus.cc:359] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.199594 1167 raft_consensus.cc:385] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.199707 1167 raft_consensus.cc:740] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.199927 1167 consensus_queue.cc:260] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.200704 1167 ts_tablet_manager.cc:1434] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:34.201074 1167 tablet_bootstrap.cc:492] T d37758f606344127b3db74c87babe69c P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:34.201231 1169 consensus_queue.cc:260] T 02e4c4d6ffb34cecaf17beb5ba197e81 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.202767 1166 tablet_bootstrap.cc:492] T fa77085e81c7440d9e255c44d1e80c26 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:34.202898 1166 ts_tablet_manager.cc:1403] T fa77085e81c7440d9e255c44d1e80c26 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.001s
I20251025 07:52:34.203579 1166 raft_consensus.cc:359] T fa77085e81c7440d9e255c44d1e80c26 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.203752 1166 raft_consensus.cc:385] T fa77085e81c7440d9e255c44d1e80c26 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.203824 1166 raft_consensus.cc:740] T fa77085e81c7440d9e255c44d1e80c26 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.203997 1166 consensus_queue.cc:260] T fa77085e81c7440d9e255c44d1e80c26 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.204444 1166 ts_tablet_manager.cc:1434] T fa77085e81c7440d9e255c44d1e80c26 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251025 07:52:34.204691 1166 tablet_bootstrap.cc:492] T a6ddefa1738f4ff98586900c344359d2 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:34.205217 1218 raft_consensus.cc:493] T 467324e2432f4caca148951329caa292 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:34.205471 1169 ts_tablet_manager.cc:1434] T 02e4c4d6ffb34cecaf17beb5ba197e81 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20251025 07:52:34.205485 1218 raft_consensus.cc:515] T 467324e2432f4caca148951329caa292 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.206861 1166 tablet_bootstrap.cc:654] T a6ddefa1738f4ff98586900c344359d2 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.206820 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "467324e2432f4caca148951329caa292" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:34.207047 775 raft_consensus.cc:2468] T 467324e2432f4caca148951329caa292 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:34.207793 847 leader_election.cc:304] T 467324e2432f4caca148951329caa292 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:34.208875 1169 tablet_bootstrap.cc:492] T b772025852794a49b072f37cccbd3291 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:34.211131 1169 tablet_bootstrap.cc:654] T b772025852794a49b072f37cccbd3291 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.213150 1712 raft_consensus.cc:2804] T 467324e2432f4caca148951329caa292 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:34.213590 1167 tablet_bootstrap.cc:654] T d37758f606344127b3db74c87babe69c P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.216774 1712 raft_consensus.cc:493] T 467324e2432f4caca148951329caa292 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:34.216984 1712 raft_consensus.cc:3060] T 467324e2432f4caca148951329caa292 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.218391 1670 raft_consensus.cc:493] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:34.218617 1670 raft_consensus.cc:515] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.219470 1670 leader_election.cc:290] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:34.220233 712 leader_election.cc:343] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 5cba9c05dd424cc788febc614a34819a
W20251025 07:52:34.221168 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskRBHlYt/build/asan/bin/kudu pid 492: Timed out: CheckLeaks RPC to 127.0.104.61:42203 timed out after 1.000s (SENT)
I20251025 07:52:34.221372 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRBHlYt/build/asan/bin/kudu with pid 492
I20251025 07:52:34.223376 1712 raft_consensus.cc:515] T 467324e2432f4caca148951329caa292 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.227183 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "467324e2432f4caca148951329caa292" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:34.227424 1048 raft_consensus.cc:3060] T 467324e2432f4caca148951329caa292 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.232694 1218 leader_election.cc:290] T 467324e2432f4caca148951329caa292 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:34.233007 1218 raft_consensus.cc:493] T 2f26667f053a4f16b5da330c0b5e40f9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:34.233145 1218 raft_consensus.cc:515] T 2f26667f053a4f16b5da330c0b5e40f9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.234025 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "467324e2432f4caca148951329caa292" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:34.234167 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f26667f053a4f16b5da330c0b5e40f9" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:34.234743 850 leader_election.cc:343] T 2f26667f053a4f16b5da330c0b5e40f9 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:34.236081 1048 raft_consensus.cc:2468] T 467324e2432f4caca148951329caa292 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:34.236693 850 leader_election.cc:304] T 467324e2432f4caca148951329caa292 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:34.237087 1712 leader_election.cc:290] T 467324e2432f4caca148951329caa292 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:34.237742 1712 raft_consensus.cc:2804] T 467324e2432f4caca148951329caa292 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:34.237969 1712 raft_consensus.cc:697] T 467324e2432f4caca148951329caa292 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:34.238317 1712 consensus_queue.cc:237] T 467324e2432f4caca148951329caa292 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.238809 1612 consensus_queue.cc:1048] T 8e6d76b1a97945cfb6c3055e1b622f8a P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:34.238934 1218 leader_election.cc:290] T 2f26667f053a4f16b5da330c0b5e40f9 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:34.240792 1169 tablet_bootstrap.cc:492] T b772025852794a49b072f37cccbd3291 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:34.241005 1169 ts_tablet_manager.cc:1403] T b772025852794a49b072f37cccbd3291 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.004s
I20251025 07:52:34.241868 1169 raft_consensus.cc:359] T b772025852794a49b072f37cccbd3291 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:34.242117 1169 raft_consensus.cc:385] T b772025852794a49b072f37cccbd3291 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.242274 1169 raft_consensus.cc:740] T b772025852794a49b072f37cccbd3291 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.242604 1169 consensus_queue.cc:260] T b772025852794a49b072f37cccbd3291 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:34.244314 1169 ts_tablet_manager.cc:1434] T b772025852794a49b072f37cccbd3291 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251025 07:52:34.244700 1169 tablet_bootstrap.cc:492] T 35ba9d23262248cda6fdf03c51be11a4 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:34.254760 1169 tablet_bootstrap.cc:654] T 35ba9d23262248cda6fdf03c51be11a4 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.261034 1167 tablet_bootstrap.cc:492] T d37758f606344127b3db74c87babe69c P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:34.261300 1167 ts_tablet_manager.cc:1403] T d37758f606344127b3db74c87babe69c P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.060s user 0.023s sys 0.025s
I20251025 07:52:34.262346 1167 raft_consensus.cc:359] T d37758f606344127b3db74c87babe69c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.262576 1167 raft_consensus.cc:385] T d37758f606344127b3db74c87babe69c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.262673 1167 raft_consensus.cc:740] T d37758f606344127b3db74c87babe69c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.262953 1169 tablet_bootstrap.cc:492] T 35ba9d23262248cda6fdf03c51be11a4 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:34.263144 1169 ts_tablet_manager.cc:1403] T 35ba9d23262248cda6fdf03c51be11a4 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.004s
I20251025 07:52:34.262955 1167 consensus_queue.cc:260] T d37758f606344127b3db74c87babe69c P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.263796 1167 ts_tablet_manager.cc:1434] T d37758f606344127b3db74c87babe69c P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:34.264113 1167 tablet_bootstrap.cc:492] T 6897405bbc1c47a4ab0aa3af80e60e3d P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:34.264814 1169 raft_consensus.cc:359] T 35ba9d23262248cda6fdf03c51be11a4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:34.265630 1169 raft_consensus.cc:385] T 35ba9d23262248cda6fdf03c51be11a4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.265872 1169 raft_consensus.cc:740] T 35ba9d23262248cda6fdf03c51be11a4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.266340 1167 tablet_bootstrap.cc:654] T 6897405bbc1c47a4ab0aa3af80e60e3d P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.266551 1169 consensus_queue.cc:260] T 35ba9d23262248cda6fdf03c51be11a4 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:34.267377 1169 ts_tablet_manager.cc:1434] T 35ba9d23262248cda6fdf03c51be11a4 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251025 07:52:34.268193 1169 tablet_bootstrap.cc:492] T 8c76d1b509f04ab1a10b83cff675c11e P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:34.272130 1169 tablet_bootstrap.cc:654] T 8c76d1b509f04ab1a10b83cff675c11e P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.274806 1167 tablet_bootstrap.cc:492] T 6897405bbc1c47a4ab0aa3af80e60e3d P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:34.274955 1167 ts_tablet_manager.cc:1403] T 6897405bbc1c47a4ab0aa3af80e60e3d P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20251025 07:52:34.275892 1167 raft_consensus.cc:359] T 6897405bbc1c47a4ab0aa3af80e60e3d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.276098 1167 raft_consensus.cc:385] T 6897405bbc1c47a4ab0aa3af80e60e3d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.276196 1167 raft_consensus.cc:740] T 6897405bbc1c47a4ab0aa3af80e60e3d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.276427 1167 consensus_queue.cc:260] T 6897405bbc1c47a4ab0aa3af80e60e3d P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.277148 1167 ts_tablet_manager.cc:1434] T 6897405bbc1c47a4ab0aa3af80e60e3d P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20251025 07:52:34.277459 1167 tablet_bootstrap.cc:492] T 35ba9d23262248cda6fdf03c51be11a4 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:34.279564 1167 tablet_bootstrap.cc:654] T 35ba9d23262248cda6fdf03c51be11a4 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.285701 1169 tablet_bootstrap.cc:492] T 8c76d1b509f04ab1a10b83cff675c11e P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:34.286489 1169 ts_tablet_manager.cc:1403] T 8c76d1b509f04ab1a10b83cff675c11e P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.002s
I20251025 07:52:34.287714 1169 raft_consensus.cc:359] T 8c76d1b509f04ab1a10b83cff675c11e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.292124 1169 raft_consensus.cc:385] T 8c76d1b509f04ab1a10b83cff675c11e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.292441 1169 raft_consensus.cc:740] T 8c76d1b509f04ab1a10b83cff675c11e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.293033 1169 consensus_queue.cc:260] T 8c76d1b509f04ab1a10b83cff675c11e P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.293395 1670 raft_consensus.cc:493] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:34.293634 1670 raft_consensus.cc:515] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.294407 1169 ts_tablet_manager.cc:1434] T 8c76d1b509f04ab1a10b83cff675c11e P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251025 07:52:34.294585 1670 leader_election.cc:290] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:34.296128 1167 tablet_bootstrap.cc:492] T 35ba9d23262248cda6fdf03c51be11a4 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:34.296303 1167 ts_tablet_manager.cc:1403] T 35ba9d23262248cda6fdf03c51be11a4 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.002s
I20251025 07:52:34.297037 1167 raft_consensus.cc:359] T 35ba9d23262248cda6fdf03c51be11a4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:34.297243 1167 raft_consensus.cc:385] T 35ba9d23262248cda6fdf03c51be11a4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.297322 1167 raft_consensus.cc:740] T 35ba9d23262248cda6fdf03c51be11a4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.297561 1167 consensus_queue.cc:260] T 35ba9d23262248cda6fdf03c51be11a4 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:34.298075 1167 ts_tablet_manager.cc:1434] T 35ba9d23262248cda6fdf03c51be11a4 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 07:52:34.298629 1167 tablet_bootstrap.cc:492] T 2f26667f053a4f16b5da330c0b5e40f9 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:34.300235 1218 raft_consensus.cc:493] T 6897405bbc1c47a4ab0aa3af80e60e3d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:34.300595 1167 tablet_bootstrap.cc:654] T 2f26667f053a4f16b5da330c0b5e40f9 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.300474 1218 raft_consensus.cc:515] T 6897405bbc1c47a4ab0aa3af80e60e3d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.294865 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27651312cc2e42a8bfcd245f5d301679" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:34.301819 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6897405bbc1c47a4ab0aa3af80e60e3d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:34.302238 1169 tablet_bootstrap.cc:492] T 27651312cc2e42a8bfcd245f5d301679 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
W20251025 07:52:34.302330 712 leader_election.cc:343] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:34.302819 850 leader_election.cc:343] T 6897405bbc1c47a4ab0aa3af80e60e3d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:34.302963 1167 tablet_bootstrap.cc:492] T 2f26667f053a4f16b5da330c0b5e40f9 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:34.303079 1218 leader_election.cc:290] T 6897405bbc1c47a4ab0aa3af80e60e3d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:34.303124 1167 ts_tablet_manager.cc:1403] T 2f26667f053a4f16b5da330c0b5e40f9 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20251025 07:52:34.297770 1675 raft_consensus.cc:493] T fab67b56eaa44b54886c17181d8c52b2 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:34.303879 1169 tablet_bootstrap.cc:654] T 27651312cc2e42a8bfcd245f5d301679 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.303768 1675 raft_consensus.cc:515] T fab67b56eaa44b54886c17181d8c52b2 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.304049 1167 raft_consensus.cc:359] T 2f26667f053a4f16b5da330c0b5e40f9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.304252 1167 raft_consensus.cc:385] T 2f26667f053a4f16b5da330c0b5e40f9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.304397 1167 raft_consensus.cc:740] T 2f26667f053a4f16b5da330c0b5e40f9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.304661 1167 consensus_queue.cc:260] T 2f26667f053a4f16b5da330c0b5e40f9 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.305202 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fab67b56eaa44b54886c17181d8c52b2" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:34.305284 1167 ts_tablet_manager.cc:1434] T 2f26667f053a4f16b5da330c0b5e40f9 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 07:52:34.305455 1675 leader_election.cc:290] T fab67b56eaa44b54886c17181d8c52b2 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:34.305761 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fab67b56eaa44b54886c17181d8c52b2" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:34.306003 913 raft_consensus.cc:2468] T fab67b56eaa44b54886c17181d8c52b2 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:34.306393 983 leader_election.cc:304] T fab67b56eaa44b54886c17181d8c52b2 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:34.306437 1167 tablet_bootstrap.cc:492] T 7f29b64704e147aa9b0b5f6cda860f09 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:34.306439 776 raft_consensus.cc:2468] T fab67b56eaa44b54886c17181d8c52b2 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:34.308516 1167 tablet_bootstrap.cc:654] T 7f29b64704e147aa9b0b5f6cda860f09 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.309530 1422 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [term 2 FOLLOWER]: Refusing update from remote peer b2dfbd0b44a441c9be9254f660974944: Log matching property violated. Preceding OpId in replica: term: 1 index: 667. Preceding OpId from leader: term: 2 index: 668. (index mismatch)
I20251025 07:52:34.310348 1658 consensus_queue.cc:1048] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 668, Last known committed idx: 664, Time since last communication: 0.000s
W20251025 07:52:34.311095 1341 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.104.61:42203 timed out after 0.590s (SENT)
I20251025 07:52:34.325981 1670 raft_consensus.cc:493] T 8c76d1b509f04ab1a10b83cff675c11e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:34.326463 1670 raft_consensus.cc:515] T 8c76d1b509f04ab1a10b83cff675c11e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.327348 1670 leader_election.cc:290] T 8c76d1b509f04ab1a10b83cff675c11e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:34.327802 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c76d1b509f04ab1a10b83cff675c11e" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:34.328011 1047 raft_consensus.cc:2468] T 8c76d1b509f04ab1a10b83cff675c11e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:34.328454 712 leader_election.cc:304] T 8c76d1b509f04ab1a10b83cff675c11e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:34.328768 1675 raft_consensus.cc:2804] T fab67b56eaa44b54886c17181d8c52b2 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:34.328804 1670 raft_consensus.cc:2804] T 8c76d1b509f04ab1a10b83cff675c11e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:34.328873 1675 raft_consensus.cc:493] T fab67b56eaa44b54886c17181d8c52b2 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:34.328918 1670 raft_consensus.cc:493] T 8c76d1b509f04ab1a10b83cff675c11e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:34.328959 1675 raft_consensus.cc:3060] T fab67b56eaa44b54886c17181d8c52b2 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.329031 1670 raft_consensus.cc:3060] T 8c76d1b509f04ab1a10b83cff675c11e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.332268 1675 raft_consensus.cc:515] T fab67b56eaa44b54886c17181d8c52b2 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.332788 1670 raft_consensus.cc:515] T 8c76d1b509f04ab1a10b83cff675c11e P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.333822 1670 leader_election.cc:290] T 8c76d1b509f04ab1a10b83cff675c11e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:34.341248 1707 raft_consensus.cc:493] T 8c76d1b509f04ab1a10b83cff675c11e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:34.341523 1707 raft_consensus.cc:515] T 8c76d1b509f04ab1a10b83cff675c11e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.343814 1707 leader_election.cc:290] T 8c76d1b509f04ab1a10b83cff675c11e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
W20251025 07:52:34.349640 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:46231, user_credentials={real_user=slave}} blocked reactor thread for 141849us
I20251025 07:52:34.350445 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "467324e2432f4caca148951329caa292" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:34.350692 776 raft_consensus.cc:3060] T 467324e2432f4caca148951329caa292 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.350783 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f26667f053a4f16b5da330c0b5e40f9" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:34.351019 774 raft_consensus.cc:2468] T 2f26667f053a4f16b5da330c0b5e40f9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:34.351441 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6897405bbc1c47a4ab0aa3af80e60e3d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:34.351701 773 raft_consensus.cc:2468] T 6897405bbc1c47a4ab0aa3af80e60e3d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:34.351768 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27651312cc2e42a8bfcd245f5d301679" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:34.351997 912 raft_consensus.cc:2468] T 27651312cc2e42a8bfcd245f5d301679 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:34.352218 847 leader_election.cc:304] T 2f26667f053a4f16b5da330c0b5e40f9 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:34.352320 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c76d1b509f04ab1a10b83cff675c11e" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:34.352391 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c76d1b509f04ab1a10b83cff675c11e" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:34.352551 914 raft_consensus.cc:2468] T 8c76d1b509f04ab1a10b83cff675c11e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:34.352903 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c76d1b509f04ab1a10b83cff675c11e" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:34.352840 847 leader_election.cc:304] T 6897405bbc1c47a4ab0aa3af80e60e3d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:34.353112 773 raft_consensus.cc:2393] T 8c76d1b509f04ab1a10b83cff675c11e P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3aea2869ff814cc7b142012faa274382 in current term 1: Already voted for candidate 23eeffd3de254ce18db5febfde971c6b in this term.
I20251025 07:52:34.353333 1167 tablet_bootstrap.cc:492] T 7f29b64704e147aa9b0b5f6cda860f09 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:34.353456 710 leader_election.cc:304] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:34.353533 1167 ts_tablet_manager.cc:1403] T 7f29b64704e147aa9b0b5f6cda860f09 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.048s user 0.000s sys 0.017s
I20251025 07:52:34.353781 776 raft_consensus.cc:2468] T 467324e2432f4caca148951329caa292 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:34.354012 1670 raft_consensus.cc:2804] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:34.354161 1670 raft_consensus.cc:493] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:34.354311 1670 raft_consensus.cc:3060] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.354636 1167 raft_consensus.cc:359] T 7f29b64704e147aa9b0b5f6cda860f09 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.354884 1167 raft_consensus.cc:385] T 7f29b64704e147aa9b0b5f6cda860f09 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.354980 1167 raft_consensus.cc:740] T 7f29b64704e147aa9b0b5f6cda860f09 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.355288 1167 consensus_queue.cc:260] T 7f29b64704e147aa9b0b5f6cda860f09 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.356179 1167 ts_tablet_manager.cc:1434] T 7f29b64704e147aa9b0b5f6cda860f09 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251025 07:52:34.356595 1167 tablet_bootstrap.cc:492] T b17d2a9ad6be49d29f2413666fba7e69 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:34.358691 1670 raft_consensus.cc:515] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.359167 1167 tablet_bootstrap.cc:654] T b17d2a9ad6be49d29f2413666fba7e69 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.360757 1670 leader_election.cc:290] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:34.362468 1711 raft_consensus.cc:2804] T 2f26667f053a4f16b5da330c0b5e40f9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:34.364261 1711 raft_consensus.cc:493] T 2f26667f053a4f16b5da330c0b5e40f9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:34.364534 1711 raft_consensus.cc:3060] T 2f26667f053a4f16b5da330c0b5e40f9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.368301 1711 raft_consensus.cc:515] T 2f26667f053a4f16b5da330c0b5e40f9 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:34.377893 711 connection.cc:537] server connection from 127.0.0.1:48830 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251025 07:52:34.378103 1167 tablet_bootstrap.cc:492] T b17d2a9ad6be49d29f2413666fba7e69 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:34.378280 1167 ts_tablet_manager.cc:1403] T b17d2a9ad6be49d29f2413666fba7e69 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20251025 07:52:34.379271 1167 raft_consensus.cc:359] T b17d2a9ad6be49d29f2413666fba7e69 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:34.379595 1167 raft_consensus.cc:385] T b17d2a9ad6be49d29f2413666fba7e69 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.379698 1167 raft_consensus.cc:740] T b17d2a9ad6be49d29f2413666fba7e69 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.379957 1167 consensus_queue.cc:260] T b17d2a9ad6be49d29f2413666fba7e69 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
W20251025 07:52:34.380030 1101 connection.cc:537] client connection to 127.0.104.61:42203 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251025 07:52:34.380546 1167 ts_tablet_manager.cc:1434] T b17d2a9ad6be49d29f2413666fba7e69 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:34.380857 1167 tablet_bootstrap.cc:492] T 4f25c9caa05b4da4bd56a8a21fde423c P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
W20251025 07:52:34.381080 1101 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.104.61:42203)
W20251025 07:52:34.381920 1101 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.61:42203)
W20251025 07:52:34.382215 1101 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.104.61:42203)
I20251025 07:52:34.382750 1167 tablet_bootstrap.cc:654] T 4f25c9caa05b4da4bd56a8a21fde423c P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:34.383565 1101 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.104.61:42203)
W20251025 07:52:34.386951 709 connection.cc:537] client connection to 127.0.104.61:42203 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 07:52:34.387630 1349 connection.cc:537] server connection from 127.0.0.1:47008 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 07:52:34.391006 1341 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.61:42203, user_credentials={real_user=slave}} blocked reactor thread for 79966.6us
W20251025 07:52:34.391608 1341 connection.cc:700] client connection to 127.0.104.61:42203 send error: Network error: sendmsg error: Connection reset by peer (error 104)
I20251025 07:52:34.334112 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c76d1b509f04ab1a10b83cff675c11e" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:34.343739 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c76d1b509f04ab1a10b83cff675c11e" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:34.395766 1047 raft_consensus.cc:3060] T 8c76d1b509f04ab1a10b83cff675c11e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.360227 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27651312cc2e42a8bfcd245f5d301679" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:34.377846 983 connection.cc:537] server connection from 127.0.0.1:47408 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 07:52:34.386533 982 connection.cc:537] client connection to 127.0.104.61:42203 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 07:52:34.398451 712 leader_election.cc:343] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:34.398600 1167 tablet_bootstrap.cc:492] T 4f25c9caa05b4da4bd56a8a21fde423c P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:34.398765 1167 ts_tablet_manager.cc:1403] T 4f25c9caa05b4da4bd56a8a21fde423c P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.004s
W20251025 07:52:34.399149 439 connection.cc:537] client connection to 127.0.104.61:42203 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 07:52:34.399291 439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 7 similar messages]
I20251025 07:52:34.400475 1047 raft_consensus.cc:2468] T 8c76d1b509f04ab1a10b83cff675c11e P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:34.401041 712 leader_election.cc:304] T 8c76d1b509f04ab1a10b83cff675c11e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:34.402412 1670 raft_consensus.cc:2804] T 8c76d1b509f04ab1a10b83cff675c11e P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:34.402849 1670 raft_consensus.cc:697] T 8c76d1b509f04ab1a10b83cff675c11e P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:34.403296 1670 consensus_queue.cc:237] T 8c76d1b509f04ab1a10b83cff675c11e P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.407972 1167 raft_consensus.cc:359] T 4f25c9caa05b4da4bd56a8a21fde423c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:34.408228 1167 raft_consensus.cc:385] T 4f25c9caa05b4da4bd56a8a21fde423c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.408319 1167 raft_consensus.cc:740] T 4f25c9caa05b4da4bd56a8a21fde423c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.408589 1167 consensus_queue.cc:260] T 4f25c9caa05b4da4bd56a8a21fde423c P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:34.409096 1167 ts_tablet_manager.cc:1434] T 4f25c9caa05b4da4bd56a8a21fde423c P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20251025 07:52:34.409474 1675 leader_election.cc:290] T fab67b56eaa44b54886c17181d8c52b2 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:34.410274 1341 leader_election.cc:336] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8603bddd8a084ff7a73a25641768dfa9 (127.0.104.61:42203): Network error: sendmsg error: Connection reset by peer (error 104)
I20251025 07:52:34.410650 1341 leader_election.cc:304] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b4a999d754c941ec850c143e4e24c407; no voters: 8603bddd8a084ff7a73a25641768dfa9, b2dfbd0b44a441c9be9254f660974944
W20251025 07:52:34.410892 1734 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 07:52:34.397456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:42203 (local address 127.0.104.3:55967)
1025 07:52:34.410466 (+ 13010us) negotiation.cc:107] Waiting for socket to connect
1025 07:52:34.410569 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:42203: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":12874,"thread_start_us":12784,"threads_started":1}
I20251025 07:52:34.411494 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fab67b56eaa44b54886c17181d8c52b2" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:34.411679 776 raft_consensus.cc:3060] T fab67b56eaa44b54886c17181d8c52b2 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:34.411728 439 leader_election.cc:336] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8603bddd8a084ff7a73a25641768dfa9 (127.0.104.61:42203): Network error: Client connection negotiation failed: client connection to 127.0.104.61:42203: connect: Connection refused (error 111)
W20251025 07:52:34.412132 439 consensus_peers.cc:597] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 -> Peer 8603bddd8a084ff7a73a25641768dfa9 (127.0.104.61:42203): Couldn't send request to peer 8603bddd8a084ff7a73a25641768dfa9. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.61:42203: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20251025 07:52:34.412282 439 leader_election.cc:336] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8603bddd8a084ff7a73a25641768dfa9 (127.0.104.61:42203): Network error: Client connection negotiation failed: client connection to 127.0.104.61:42203: connect: Connection refused (error 111)
I20251025 07:52:34.414949 776 raft_consensus.cc:2468] T fab67b56eaa44b54886c17181d8c52b2 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:34.415248 1167 tablet_bootstrap.cc:492] T a6ddefa1738f4ff98586900c344359d2 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:34.415447 982 leader_election.cc:304] T fab67b56eaa44b54886c17181d8c52b2 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:34.417073 1167 tablet_bootstrap.cc:654] T a6ddefa1738f4ff98586900c344359d2 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.428140 1612 raft_consensus.cc:2804] T fab67b56eaa44b54886c17181d8c52b2 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:34.428360 1612 raft_consensus.cc:697] T fab67b56eaa44b54886c17181d8c52b2 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:34.428673 1612 consensus_queue.cc:237] T fab67b56eaa44b54886c17181d8c52b2 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:34.428929 710 leader_election.cc:343] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 5cba9c05dd424cc788febc614a34819a
I20251025 07:52:34.429082 710 leader_election.cc:304] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:34.429729 1671 raft_consensus.cc:493] T d37758f606344127b3db74c87babe69c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:34.429891 1671 raft_consensus.cc:515] T d37758f606344127b3db74c87babe69c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.430253 1670 raft_consensus.cc:2749] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:34.430605 1671 leader_election.cc:290] T d37758f606344127b3db74c87babe69c P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:34.431584 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d37758f606344127b3db74c87babe69c" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:34.431823 776 raft_consensus.cc:2468] T d37758f606344127b3db74c87babe69c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:34.432247 982 leader_election.cc:304] T d37758f606344127b3db74c87babe69c P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:34.432566 1671 raft_consensus.cc:2804] T d37758f606344127b3db74c87babe69c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:34.432684 1671 raft_consensus.cc:493] T d37758f606344127b3db74c87babe69c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:34.432812 1671 raft_consensus.cc:3060] T d37758f606344127b3db74c87babe69c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.436034 1671 raft_consensus.cc:515] T d37758f606344127b3db74c87babe69c P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.437564 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d37758f606344127b3db74c87babe69c" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:34.437780 776 raft_consensus.cc:3060] T d37758f606344127b3db74c87babe69c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.440907 776 raft_consensus.cc:2468] T d37758f606344127b3db74c87babe69c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:34.442991 1671 leader_election.cc:290] T d37758f606344127b3db74c87babe69c P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:34.443384 1671 raft_consensus.cc:493] T 35ba9d23262248cda6fdf03c51be11a4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:34.443785 1671 raft_consensus.cc:515] T 35ba9d23262248cda6fdf03c51be11a4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:34.445752 1671 leader_election.cc:290] T 35ba9d23262248cda6fdf03c51be11a4 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:34.447427 982 leader_election.cc:304] T d37758f606344127b3db74c87babe69c P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:34.447710 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "35ba9d23262248cda6fdf03c51be11a4" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:34.447952 776 raft_consensus.cc:2468] T 35ba9d23262248cda6fdf03c51be11a4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:34.448256 1671 raft_consensus.cc:2804] T d37758f606344127b3db74c87babe69c P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:34.448601 1671 raft_consensus.cc:697] T d37758f606344127b3db74c87babe69c P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:34.448957 1671 consensus_queue.cc:237] T d37758f606344127b3db74c87babe69c P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.450151 982 leader_election.cc:304] T 35ba9d23262248cda6fdf03c51be11a4 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:34.450548 1569 raft_consensus.cc:2804] T 35ba9d23262248cda6fdf03c51be11a4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:34.450654 1569 raft_consensus.cc:493] T 35ba9d23262248cda6fdf03c51be11a4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:34.450781 1569 raft_consensus.cc:3060] T 35ba9d23262248cda6fdf03c51be11a4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.454452 1569 raft_consensus.cc:515] T 35ba9d23262248cda6fdf03c51be11a4 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:34.455695 1569 leader_election.cc:290] T 35ba9d23262248cda6fdf03c51be11a4 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:34.455859 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "35ba9d23262248cda6fdf03c51be11a4" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:34.456126 776 raft_consensus.cc:3060] T 35ba9d23262248cda6fdf03c51be11a4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.388317 1712 raft_consensus.cc:2804] T 6897405bbc1c47a4ab0aa3af80e60e3d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:34.395035 1166 tablet_bootstrap.cc:492] T a6ddefa1738f4ff98586900c344359d2 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:34.410046 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fab67b56eaa44b54886c17181d8c52b2" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:34.458397 1712 raft_consensus.cc:493] T 6897405bbc1c47a4ab0aa3af80e60e3d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:34.458526 913 raft_consensus.cc:3060] T fab67b56eaa44b54886c17181d8c52b2 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.458572 1166 ts_tablet_manager.cc:1403] T a6ddefa1738f4ff98586900c344359d2 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.254s user 0.035s sys 0.047s
I20251025 07:52:34.458678 1712 raft_consensus.cc:3060] T 6897405bbc1c47a4ab0aa3af80e60e3d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.459630 1166 raft_consensus.cc:359] T a6ddefa1738f4ff98586900c344359d2 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.459888 1166 raft_consensus.cc:385] T a6ddefa1738f4ff98586900c344359d2 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.459986 1166 raft_consensus.cc:740] T a6ddefa1738f4ff98586900c344359d2 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.460310 1166 consensus_queue.cc:260] T a6ddefa1738f4ff98586900c344359d2 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.460989 1166 ts_tablet_manager.cc:1434] T a6ddefa1738f4ff98586900c344359d2 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:34.461380 1166 tablet_bootstrap.cc:492] T 887e7c7a5d0e4aaf88f26246306df19c P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:34.461915 913 raft_consensus.cc:2468] T fab67b56eaa44b54886c17181d8c52b2 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:34.461850 1712 raft_consensus.cc:515] T 6897405bbc1c47a4ab0aa3af80e60e3d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.431250 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d37758f606344127b3db74c87babe69c" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:34.462101 914 raft_consensus.cc:2468] T d37758f606344127b3db74c87babe69c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:34.430153 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27651312cc2e42a8bfcd245f5d301679" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:34.462471 911 raft_consensus.cc:3060] T 27651312cc2e42a8bfcd245f5d301679 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.463456 1712 leader_election.cc:290] T 6897405bbc1c47a4ab0aa3af80e60e3d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:34.463730 1166 tablet_bootstrap.cc:654] T 887e7c7a5d0e4aaf88f26246306df19c P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.463835 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6897405bbc1c47a4ab0aa3af80e60e3d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:34.464437 850 leader_election.cc:343] T 6897405bbc1c47a4ab0aa3af80e60e3d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:34.464681 1670 raft_consensus.cc:493] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:34.464852 1670 raft_consensus.cc:515] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.465474 911 raft_consensus.cc:2468] T 27651312cc2e42a8bfcd245f5d301679 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:34.445384 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "35ba9d23262248cda6fdf03c51be11a4" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:34.465739 910 raft_consensus.cc:2468] T 35ba9d23262248cda6fdf03c51be11a4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:34.456027 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "35ba9d23262248cda6fdf03c51be11a4" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:34.465832 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02e4c4d6ffb34cecaf17beb5ba197e81" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:34.465971 909 raft_consensus.cc:3060] T 35ba9d23262248cda6fdf03c51be11a4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.466017 1047 raft_consensus.cc:2468] T 02e4c4d6ffb34cecaf17beb5ba197e81 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:34.466701 712 leader_election.cc:304] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:34.466998 1441 raft_consensus.cc:2804] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:34.467111 1441 raft_consensus.cc:493] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:34.467239 1441 raft_consensus.cc:3060] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.467651 1670 leader_election.cc:290] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:34.430001 847 connection.cc:537] client connection to 127.0.104.61:42203 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 07:52:34.468086 847 connection.cc:537] server connection from 127.0.0.1:56590 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251025 07:52:34.468528 847 leader_election.cc:304] T 8c76d1b509f04ab1a10b83cff675c11e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:34.468869 909 raft_consensus.cc:2468] T 35ba9d23262248cda6fdf03c51be11a4 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:34.437281 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d37758f606344127b3db74c87babe69c" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:34.469112 912 raft_consensus.cc:3060] T d37758f606344127b3db74c87babe69c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.469491 710 leader_election.cc:304] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:34.470263 1670 raft_consensus.cc:2804] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:34.470513 1670 raft_consensus.cc:697] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:34.470677 1441 raft_consensus.cc:515] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.470804 1670 consensus_queue.cc:237] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.471454 1441 leader_election.cc:290] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:34.471803 912 raft_consensus.cc:2468] T d37758f606344127b3db74c87babe69c P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:34.472139 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02e4c4d6ffb34cecaf17beb5ba197e81" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:34.472308 1047 raft_consensus.cc:3060] T 02e4c4d6ffb34cecaf17beb5ba197e81 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.473891 776 raft_consensus.cc:2468] T 35ba9d23262248cda6fdf03c51be11a4 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:34.474385 982 leader_election.cc:304] T 35ba9d23262248cda6fdf03c51be11a4 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:34.474778 1675 raft_consensus.cc:2804] T 35ba9d23262248cda6fdf03c51be11a4 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:34.476181 1675 raft_consensus.cc:697] T 35ba9d23262248cda6fdf03c51be11a4 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:34.476601 1675 consensus_queue.cc:237] T 35ba9d23262248cda6fdf03c51be11a4 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:34.476868 1047 raft_consensus.cc:2468] T 02e4c4d6ffb34cecaf17beb5ba197e81 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:34.477437 712 leader_election.cc:304] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:34.477761 1441 raft_consensus.cc:2804] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:34.477968 1441 raft_consensus.cc:697] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:34.478331 1441 consensus_queue.cc:237] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.478575 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6897405bbc1c47a4ab0aa3af80e60e3d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:34.478910 775 raft_consensus.cc:3060] T 6897405bbc1c47a4ab0aa3af80e60e3d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.481961 1561 raft_consensus.cc:3060] T 8c76d1b509f04ab1a10b83cff675c11e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.495040 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f26667f053a4f16b5da330c0b5e40f9" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:34.495687 850 leader_election.cc:343] T 2f26667f053a4f16b5da330c0b5e40f9 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:34.497191 1732 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 07:52:34.388461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:42203 (local address 127.0.104.1:35587)
1025 07:52:34.496677 (+108216us) negotiation.cc:107] Waiting for socket to connect
1025 07:52:34.496878 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:42203: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":108070,"thread_start_us":107937,"threads_started":1}
I20251025 07:52:34.499619 775 raft_consensus.cc:2468] T 6897405bbc1c47a4ab0aa3af80e60e3d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:34.500173 1711 leader_election.cc:290] T 2f26667f053a4f16b5da330c0b5e40f9 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:34.502761 1675 raft_consensus.cc:493] T 2784bbfb97d5466f9ec856add7d0b29a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:34.502960 1675 raft_consensus.cc:515] T 2784bbfb97d5466f9ec856add7d0b29a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.503767 1675 leader_election.cc:290] T 2784bbfb97d5466f9ec856add7d0b29a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:34.504390 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2784bbfb97d5466f9ec856add7d0b29a" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:34.504639 913 raft_consensus.cc:2468] T 2784bbfb97d5466f9ec856add7d0b29a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:34.505162 983 leader_election.cc:304] T 2784bbfb97d5466f9ec856add7d0b29a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:34.506199 1169 tablet_bootstrap.cc:492] T 27651312cc2e42a8bfcd245f5d301679 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:34.506352 1169 ts_tablet_manager.cc:1403] T 27651312cc2e42a8bfcd245f5d301679 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.204s user 0.024s sys 0.016s
I20251025 07:52:34.507210 1169 raft_consensus.cc:359] T 27651312cc2e42a8bfcd245f5d301679 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.507405 1169 raft_consensus.cc:385] T 27651312cc2e42a8bfcd245f5d301679 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.507508 1169 raft_consensus.cc:740] T 27651312cc2e42a8bfcd245f5d301679 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.510869 1738 raft_consensus.cc:2749] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 07:52:34.511212 1612 raft_consensus.cc:2804] T 2784bbfb97d5466f9ec856add7d0b29a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:34.511803 1612 raft_consensus.cc:493] T 2784bbfb97d5466f9ec856add7d0b29a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:34.512277 1612 raft_consensus.cc:3060] T 2784bbfb97d5466f9ec856add7d0b29a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.517768 1612 raft_consensus.cc:515] T 2784bbfb97d5466f9ec856add7d0b29a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.522612 1725 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2dfbd0b44a441c9be9254f660974944. Latest consensus state: current_term: 2 leader_uuid: "b2dfbd0b44a441c9be9254f660974944" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } } }
I20251025 07:52:34.523180 1725 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 07:52:34.524402 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02e4c4d6ffb34cecaf17beb5ba197e81" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:34.524682 913 raft_consensus.cc:2468] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:34.525868 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02e4c4d6ffb34cecaf17beb5ba197e81" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:34.526021 913 raft_consensus.cc:3060] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.526877 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2784bbfb97d5466f9ec856add7d0b29a" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:34.527041 912 raft_consensus.cc:3060] T 2784bbfb97d5466f9ec856add7d0b29a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.530000 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2784bbfb97d5466f9ec856add7d0b29a" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:34.530068 913 raft_consensus.cc:2468] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:34.530263 776 raft_consensus.cc:3060] T 2784bbfb97d5466f9ec856add7d0b29a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.533109 776 raft_consensus.cc:2468] T 2784bbfb97d5466f9ec856add7d0b29a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:34.533578 912 raft_consensus.cc:2468] T 2784bbfb97d5466f9ec856add7d0b29a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:34.534066 983 leader_election.cc:304] T 2784bbfb97d5466f9ec856add7d0b29a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:34.522362 1169 consensus_queue.cc:260] T 27651312cc2e42a8bfcd245f5d301679 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.534413 1675 raft_consensus.cc:2804] T 2784bbfb97d5466f9ec856add7d0b29a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:34.534968 1169 ts_tablet_manager.cc:1434] T 27651312cc2e42a8bfcd245f5d301679 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20251025 07:52:34.535275 1169 tablet_bootstrap.cc:492] T 6897405bbc1c47a4ab0aa3af80e60e3d P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:34.537143 1169 tablet_bootstrap.cc:654] T 6897405bbc1c47a4ab0aa3af80e60e3d P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.539345 1612 leader_election.cc:290] T 2784bbfb97d5466f9ec856add7d0b29a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:34.545190 1731 negotiation.cc:337] Failed RPC negotiation. Trace:
1025 07:52:34.384884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:42203 (local address 127.0.0.1:40806)
1025 07:52:34.456848 (+ 71964us) negotiation.cc:107] Waiting for socket to connect
1025 07:52:34.457075 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:42203: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":71759,"thread_start_us":71620,"threads_started":1}
I20251025 07:52:34.553329 1675 raft_consensus.cc:697] T 2784bbfb97d5466f9ec856add7d0b29a P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:34.553825 1675 consensus_queue.cc:237] T 2784bbfb97d5466f9ec856add7d0b29a P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.554750 1169 tablet_bootstrap.cc:492] T 6897405bbc1c47a4ab0aa3af80e60e3d P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:34.554934 1169 ts_tablet_manager.cc:1403] T 6897405bbc1c47a4ab0aa3af80e60e3d P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.001s
I20251025 07:52:34.558451 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f26667f053a4f16b5da330c0b5e40f9" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:34.558676 776 raft_consensus.cc:3060] T 2f26667f053a4f16b5da330c0b5e40f9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.560264 847 leader_election.cc:304] T 6897405bbc1c47a4ab0aa3af80e60e3d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:34.560923 1707 raft_consensus.cc:2804] T 6897405bbc1c47a4ab0aa3af80e60e3d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:34.561411 1707 raft_consensus.cc:697] T 6897405bbc1c47a4ab0aa3af80e60e3d P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:34.561728 776 raft_consensus.cc:2468] T 2f26667f053a4f16b5da330c0b5e40f9 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:34.561728 1707 consensus_queue.cc:237] T 6897405bbc1c47a4ab0aa3af80e60e3d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.557575 1169 raft_consensus.cc:359] T 6897405bbc1c47a4ab0aa3af80e60e3d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.563200 1169 raft_consensus.cc:385] T 6897405bbc1c47a4ab0aa3af80e60e3d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.563297 1169 raft_consensus.cc:740] T 6897405bbc1c47a4ab0aa3af80e60e3d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.563576 1169 consensus_queue.cc:260] T 6897405bbc1c47a4ab0aa3af80e60e3d P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.564169 1169 ts_tablet_manager.cc:1434] T 6897405bbc1c47a4ab0aa3af80e60e3d P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20251025 07:52:34.564543 1169 tablet_bootstrap.cc:492] T 4f25c9caa05b4da4bd56a8a21fde423c P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:34.566445 1169 tablet_bootstrap.cc:654] T 4f25c9caa05b4da4bd56a8a21fde423c P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.567713 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2784bbfb97d5466f9ec856add7d0b29a" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:34.568621 1561 raft_consensus.cc:2749] T 8c76d1b509f04ab1a10b83cff675c11e P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:34.568915 1658 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "b2dfbd0b44a441c9be9254f660974944" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } } }
I20251025 07:52:34.569173 1658 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [sys.catalog]: This master's current role is: LEADER
I20251025 07:52:34.571872 775 raft_consensus.cc:2376] T 2784bbfb97d5466f9ec856add7d0b29a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1. Re-sending same reply.
I20251025 07:52:34.577216 1167 tablet_bootstrap.cc:492] T a6ddefa1738f4ff98586900c344359d2 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:34.577416 1167 ts_tablet_manager.cc:1403] T a6ddefa1738f4ff98586900c344359d2 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.168s user 0.002s sys 0.017s
I20251025 07:52:34.578322 1167 raft_consensus.cc:359] T a6ddefa1738f4ff98586900c344359d2 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.578544 1167 raft_consensus.cc:385] T a6ddefa1738f4ff98586900c344359d2 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.578634 1167 raft_consensus.cc:740] T a6ddefa1738f4ff98586900c344359d2 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.578850 1167 consensus_queue.cc:260] T a6ddefa1738f4ff98586900c344359d2 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.579265 1166 tablet_bootstrap.cc:492] T 887e7c7a5d0e4aaf88f26246306df19c P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:34.579813 1167 ts_tablet_manager.cc:1434] T a6ddefa1738f4ff98586900c344359d2 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251025 07:52:34.580137 1167 tablet_bootstrap.cc:492] T fa77085e81c7440d9e255c44d1e80c26 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:34.581118 1166 ts_tablet_manager.cc:1403] T 887e7c7a5d0e4aaf88f26246306df19c P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.119s user 0.035s sys 0.023s
I20251025 07:52:34.582269 1166 raft_consensus.cc:359] T 887e7c7a5d0e4aaf88f26246306df19c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:34.582469 847 consensus_peers.cc:597] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382 -> Peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Couldn't send request to peer 23eeffd3de254ce18db5febfde971c6b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 627a10a0714f4be8bdbced503fd6388d. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:34.582968 1167 tablet_bootstrap.cc:654] T fa77085e81c7440d9e255c44d1e80c26 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.583281 1169 tablet_bootstrap.cc:492] T 4f25c9caa05b4da4bd56a8a21fde423c P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:34.583467 1169 ts_tablet_manager.cc:1403] T 4f25c9caa05b4da4bd56a8a21fde423c P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20251025 07:52:34.584408 1169 raft_consensus.cc:359] T 4f25c9caa05b4da4bd56a8a21fde423c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:34.584630 1169 raft_consensus.cc:385] T 4f25c9caa05b4da4bd56a8a21fde423c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.584748 1169 raft_consensus.cc:740] T 4f25c9caa05b4da4bd56a8a21fde423c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.582561 1166 raft_consensus.cc:385] T 887e7c7a5d0e4aaf88f26246306df19c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.584988 1166 raft_consensus.cc:740] T 887e7c7a5d0e4aaf88f26246306df19c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.585012 1169 consensus_queue.cc:260] T 4f25c9caa05b4da4bd56a8a21fde423c P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:34.585455 1166 consensus_queue.cc:260] T 887e7c7a5d0e4aaf88f26246306df19c P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.586606 1166 ts_tablet_manager.cc:1434] T 887e7c7a5d0e4aaf88f26246306df19c P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251025 07:52:34.587044 1166 tablet_bootstrap.cc:492] T 0b21b7da1e2f47829c3f6cdaee689e8a P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:34.591151 1167 tablet_bootstrap.cc:492] T fa77085e81c7440d9e255c44d1e80c26 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:34.591336 1167 ts_tablet_manager.cc:1403] T fa77085e81c7440d9e255c44d1e80c26 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20251025 07:52:34.591646 847 leader_election.cc:304] T 2f26667f053a4f16b5da330c0b5e40f9 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:34.592887 1167 raft_consensus.cc:359] T fa77085e81c7440d9e255c44d1e80c26 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.593127 1167 raft_consensus.cc:385] T fa77085e81c7440d9e255c44d1e80c26 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.593227 1167 raft_consensus.cc:740] T fa77085e81c7440d9e255c44d1e80c26 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.593508 1167 consensus_queue.cc:260] T fa77085e81c7440d9e255c44d1e80c26 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.594027 1218 raft_consensus.cc:2804] T 2f26667f053a4f16b5da330c0b5e40f9 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:34.594206 1218 raft_consensus.cc:697] T 2f26667f053a4f16b5da330c0b5e40f9 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:34.594393 1167 ts_tablet_manager.cc:1434] T fa77085e81c7440d9e255c44d1e80c26 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251025 07:52:34.594472 1218 consensus_queue.cc:237] T 2f26667f053a4f16b5da330c0b5e40f9 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.594727 1167 tablet_bootstrap.cc:492] T 0b21b7da1e2f47829c3f6cdaee689e8a P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:34.595980 1169 ts_tablet_manager.cc:1434] T 4f25c9caa05b4da4bd56a8a21fde423c P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20251025 07:52:34.597074 1167 tablet_bootstrap.cc:654] T 0b21b7da1e2f47829c3f6cdaee689e8a P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.599611 1169 tablet_bootstrap.cc:492] T 2f26667f053a4f16b5da330c0b5e40f9 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:34.602818 1169 tablet_bootstrap.cc:654] T 2f26667f053a4f16b5da330c0b5e40f9 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.603510 1167 tablet_bootstrap.cc:492] T 0b21b7da1e2f47829c3f6cdaee689e8a P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:34.603655 1167 ts_tablet_manager.cc:1403] T 0b21b7da1e2f47829c3f6cdaee689e8a P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20251025 07:52:34.604566 1167 raft_consensus.cc:359] T 0b21b7da1e2f47829c3f6cdaee689e8a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.604768 1167 raft_consensus.cc:385] T 0b21b7da1e2f47829c3f6cdaee689e8a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.604877 1167 raft_consensus.cc:740] T 0b21b7da1e2f47829c3f6cdaee689e8a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.605099 1167 consensus_queue.cc:260] T 0b21b7da1e2f47829c3f6cdaee689e8a P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.605913 1167 ts_tablet_manager.cc:1434] T 0b21b7da1e2f47829c3f6cdaee689e8a P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:34.606259 1167 tablet_bootstrap.cc:492] T 887e7c7a5d0e4aaf88f26246306df19c P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:34.608342 1167 tablet_bootstrap.cc:654] T 887e7c7a5d0e4aaf88f26246306df19c P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.613282 1169 tablet_bootstrap.cc:492] T 2f26667f053a4f16b5da330c0b5e40f9 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:34.613492 1169 ts_tablet_manager.cc:1403] T 2f26667f053a4f16b5da330c0b5e40f9 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.002s
I20251025 07:52:34.614491 1169 raft_consensus.cc:359] T 2f26667f053a4f16b5da330c0b5e40f9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.614729 1169 raft_consensus.cc:385] T 2f26667f053a4f16b5da330c0b5e40f9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.614802 1169 raft_consensus.cc:740] T 2f26667f053a4f16b5da330c0b5e40f9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.615068 1169 consensus_queue.cc:260] T 2f26667f053a4f16b5da330c0b5e40f9 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.615423 1561 raft_consensus.cc:493] T b772025852794a49b072f37cccbd3291 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:34.615653 1561 raft_consensus.cc:515] T b772025852794a49b072f37cccbd3291 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:34.617537 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b772025852794a49b072f37cccbd3291" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:34.617791 1047 raft_consensus.cc:2468] T b772025852794a49b072f37cccbd3291 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:34.618306 850 leader_election.cc:304] T b772025852794a49b072f37cccbd3291 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:34.618574 1169 ts_tablet_manager.cc:1434] T 2f26667f053a4f16b5da330c0b5e40f9 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20251025 07:52:34.618595 1218 raft_consensus.cc:2804] T b772025852794a49b072f37cccbd3291 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:34.618706 1218 raft_consensus.cc:493] T b772025852794a49b072f37cccbd3291 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:34.618788 1218 raft_consensus.cc:3060] T b772025852794a49b072f37cccbd3291 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.620775 1561 leader_election.cc:290] T b772025852794a49b072f37cccbd3291 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:34.621060 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b772025852794a49b072f37cccbd3291" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:34.621325 775 raft_consensus.cc:2468] T b772025852794a49b072f37cccbd3291 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:34.622643 1725 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "b2dfbd0b44a441c9be9254f660974944" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } } }
I20251025 07:52:34.622938 1725 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 07:52:34.623793 1218 raft_consensus.cc:515] T b772025852794a49b072f37cccbd3291 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:34.624717 1169 tablet_bootstrap.cc:492] T 7f29b64704e147aa9b0b5f6cda860f09 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:34.624903 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b772025852794a49b072f37cccbd3291" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:34.625080 1047 raft_consensus.cc:3060] T b772025852794a49b072f37cccbd3291 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.626639 1169 tablet_bootstrap.cc:654] T 7f29b64704e147aa9b0b5f6cda860f09 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.628175 1047 raft_consensus.cc:2468] T b772025852794a49b072f37cccbd3291 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:34.628559 850 leader_election.cc:304] T b772025852794a49b072f37cccbd3291 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:34.628881 1218 leader_election.cc:290] T b772025852794a49b072f37cccbd3291 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:34.629045 1218 raft_consensus.cc:2804] T b772025852794a49b072f37cccbd3291 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:34.629168 1218 raft_consensus.cc:697] T b772025852794a49b072f37cccbd3291 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:34.629441 1218 consensus_queue.cc:237] T b772025852794a49b072f37cccbd3291 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:34.629606 1166 tablet_bootstrap.cc:654] T 0b21b7da1e2f47829c3f6cdaee689e8a P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.632670 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b772025852794a49b072f37cccbd3291" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:34.632951 775 raft_consensus.cc:3060] T b772025852794a49b072f37cccbd3291 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.635974 776 raft_consensus.cc:3060] T 7e869c65771f4339a546dfeebb227c62 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.645539 775 raft_consensus.cc:2468] T b772025852794a49b072f37cccbd3291 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:34.646564 1166 tablet_bootstrap.cc:492] T 0b21b7da1e2f47829c3f6cdaee689e8a P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:34.646881 1166 ts_tablet_manager.cc:1403] T 0b21b7da1e2f47829c3f6cdaee689e8a P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.060s user 0.003s sys 0.001s
I20251025 07:52:34.648537 1167 tablet_bootstrap.cc:492] T 887e7c7a5d0e4aaf88f26246306df19c P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:34.648713 1167 ts_tablet_manager.cc:1403] T 887e7c7a5d0e4aaf88f26246306df19c P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.043s user 0.005s sys 0.009s
I20251025 07:52:34.651917 1167 raft_consensus.cc:359] T 887e7c7a5d0e4aaf88f26246306df19c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.652179 1167 raft_consensus.cc:385] T 887e7c7a5d0e4aaf88f26246306df19c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.652251 1167 raft_consensus.cc:740] T 887e7c7a5d0e4aaf88f26246306df19c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.652489 1167 consensus_queue.cc:260] T 887e7c7a5d0e4aaf88f26246306df19c P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.654563 1167 ts_tablet_manager.cc:1434] T 887e7c7a5d0e4aaf88f26246306df19c P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251025 07:52:34.655452 1167 tablet_bootstrap.cc:492] T f0bdaa8cfd764f1b83f47881353d26b6 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:34.656002 1166 raft_consensus.cc:359] T 0b21b7da1e2f47829c3f6cdaee689e8a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.656288 1166 raft_consensus.cc:385] T 0b21b7da1e2f47829c3f6cdaee689e8a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.656402 1166 raft_consensus.cc:740] T 0b21b7da1e2f47829c3f6cdaee689e8a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.656795 1166 consensus_queue.cc:260] T 0b21b7da1e2f47829c3f6cdaee689e8a P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.657456 1166 ts_tablet_manager.cc:1434] T 0b21b7da1e2f47829c3f6cdaee689e8a P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.010s user 0.000s sys 0.001s
I20251025 07:52:34.657959 1166 tablet_bootstrap.cc:492] T 3775a2e08d4e40bd9f57e4cda759d535 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:34.661119 1166 tablet_bootstrap.cc:654] T 3775a2e08d4e40bd9f57e4cda759d535 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.661276 1167 tablet_bootstrap.cc:654] T f0bdaa8cfd764f1b83f47881353d26b6 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.670452 1169 tablet_bootstrap.cc:492] T 7f29b64704e147aa9b0b5f6cda860f09 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:34.670624 1169 ts_tablet_manager.cc:1403] T 7f29b64704e147aa9b0b5f6cda860f09 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.046s user 0.005s sys 0.006s
I20251025 07:52:34.671056 1749 raft_consensus.cc:493] T 27651312cc2e42a8bfcd245f5d301679 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:34.671221 1749 raft_consensus.cc:515] T 27651312cc2e42a8bfcd245f5d301679 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.672062 1749 leader_election.cc:290] T 27651312cc2e42a8bfcd245f5d301679 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:34.672977 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27651312cc2e42a8bfcd245f5d301679" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:34.673321 912 raft_consensus.cc:2393] T 27651312cc2e42a8bfcd245f5d301679 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 23eeffd3de254ce18db5febfde971c6b in this term.
I20251025 07:52:34.673600 1169 raft_consensus.cc:359] T 7f29b64704e147aa9b0b5f6cda860f09 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.673916 1169 raft_consensus.cc:385] T 7f29b64704e147aa9b0b5f6cda860f09 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.674060 1169 raft_consensus.cc:740] T 7f29b64704e147aa9b0b5f6cda860f09 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.674443 1169 consensus_queue.cc:260] T 7f29b64704e147aa9b0b5f6cda860f09 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.675791 1667 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251025 07:52:34.681450 1169 ts_tablet_manager.cc:1434] T 7f29b64704e147aa9b0b5f6cda860f09 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20251025 07:52:34.689075 1169 tablet_bootstrap.cc:492] T a6ddefa1738f4ff98586900c344359d2 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:34.693774 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27651312cc2e42a8bfcd245f5d301679" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:34.694561 982 leader_election.cc:304] T 27651312cc2e42a8bfcd245f5d301679 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:34.695067 1749 raft_consensus.cc:3060] T 27651312cc2e42a8bfcd245f5d301679 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.698747 1749 raft_consensus.cc:2749] T 27651312cc2e42a8bfcd245f5d301679 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:34.700158 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_550c910e06364276ae5c4548afa39e6c [id=03294f958266430589e89052d6c729df]
I20251025 07:52:34.701046 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]
I20251025 07:52:34.701893 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de]
I20251025 07:52:34.702179 1169 tablet_bootstrap.cc:654] T a6ddefa1738f4ff98586900c344359d2 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.705073 1167 tablet_bootstrap.cc:492] T f0bdaa8cfd764f1b83f47881353d26b6 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:34.705265 1167 ts_tablet_manager.cc:1403] T f0bdaa8cfd764f1b83f47881353d26b6 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.001s
I20251025 07:52:34.705861 1218 raft_consensus.cc:493] T 887e7c7a5d0e4aaf88f26246306df19c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:34.706097 1218 raft_consensus.cc:515] T 887e7c7a5d0e4aaf88f26246306df19c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.706288 1167 raft_consensus.cc:359] T f0bdaa8cfd764f1b83f47881353d26b6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.706543 1167 raft_consensus.cc:385] T f0bdaa8cfd764f1b83f47881353d26b6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.706653 1167 raft_consensus.cc:740] T f0bdaa8cfd764f1b83f47881353d26b6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.706887 1218 leader_election.cc:290] T 887e7c7a5d0e4aaf88f26246306df19c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:34.707454 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "887e7c7a5d0e4aaf88f26246306df19c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:34.707602 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "887e7c7a5d0e4aaf88f26246306df19c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:34.707676 776 raft_consensus.cc:2468] T 887e7c7a5d0e4aaf88f26246306df19c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:34.708277 847 leader_election.cc:304] T 887e7c7a5d0e4aaf88f26246306df19c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
W20251025 07:52:34.708714 850 leader_election.cc:343] T 887e7c7a5d0e4aaf88f26246306df19c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:34.709189 1218 raft_consensus.cc:2804] T 887e7c7a5d0e4aaf88f26246306df19c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:34.709306 1218 raft_consensus.cc:493] T 887e7c7a5d0e4aaf88f26246306df19c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:34.709439 1218 raft_consensus.cc:3060] T 887e7c7a5d0e4aaf88f26246306df19c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.712870 1169 tablet_bootstrap.cc:492] T a6ddefa1738f4ff98586900c344359d2 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:34.713042 1169 ts_tablet_manager.cc:1403] T a6ddefa1738f4ff98586900c344359d2 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.007s
I20251025 07:52:34.713784 1169 raft_consensus.cc:359] T a6ddefa1738f4ff98586900c344359d2 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.713995 1169 raft_consensus.cc:385] T a6ddefa1738f4ff98586900c344359d2 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.714072 1169 raft_consensus.cc:740] T a6ddefa1738f4ff98586900c344359d2 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.714216 1218 raft_consensus.cc:515] T 887e7c7a5d0e4aaf88f26246306df19c P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.714432 1169 consensus_queue.cc:260] T a6ddefa1738f4ff98586900c344359d2 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.715710 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "887e7c7a5d0e4aaf88f26246306df19c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:34.715948 776 raft_consensus.cc:3060] T 887e7c7a5d0e4aaf88f26246306df19c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.720284 1561 raft_consensus.cc:493] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:34.720466 1561 raft_consensus.cc:515] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:34.720896 1169 ts_tablet_manager.cc:1434] T a6ddefa1738f4ff98586900c344359d2 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20251025 07:52:34.720896 1670 raft_consensus.cc:493] T b17d2a9ad6be49d29f2413666fba7e69 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:34.721040 1670 raft_consensus.cc:515] T b17d2a9ad6be49d29f2413666fba7e69 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:34.721174 1561 leader_election.cc:290] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:34.721714 1670 leader_election.cc:290] T b17d2a9ad6be49d29f2413666fba7e69 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:34.721952 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b17d2a9ad6be49d29f2413666fba7e69" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:34.722241 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b17d2a9ad6be49d29f2413666fba7e69" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:34.722434 775 raft_consensus.cc:2468] T b17d2a9ad6be49d29f2413666fba7e69 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
W20251025 07:52:34.722508 850 leader_election.cc:343] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:34.722540 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b17d2a9ad6be49d29f2413666fba7e69" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:34.723280 712 leader_election.cc:343] T b17d2a9ad6be49d29f2413666fba7e69 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:34.723373 847 leader_election.cc:304] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:34.724025 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b17d2a9ad6be49d29f2413666fba7e69" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:34.724217 912 raft_consensus.cc:2468] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:34.724560 1561 raft_consensus.cc:2804] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:34.724668 1561 raft_consensus.cc:493] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:34.724750 1561 raft_consensus.cc:3060] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.725656 1169 tablet_bootstrap.cc:492] T fa77085e81c7440d9e255c44d1e80c26 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:34.725739 710 leader_election.cc:304] T b17d2a9ad6be49d29f2413666fba7e69 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:34.726176 1670 raft_consensus.cc:2804] T b17d2a9ad6be49d29f2413666fba7e69 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:34.726285 1670 raft_consensus.cc:493] T b17d2a9ad6be49d29f2413666fba7e69 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:34.726367 1670 raft_consensus.cc:3060] T b17d2a9ad6be49d29f2413666fba7e69 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.727694 1561 raft_consensus.cc:515] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:34.727816 1169 tablet_bootstrap.cc:654] T fa77085e81c7440d9e255c44d1e80c26 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.728749 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b17d2a9ad6be49d29f2413666fba7e69" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:34.730321 850 leader_election.cc:343] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:34.730507 1561 leader_election.cc:290] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:34.731120 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b17d2a9ad6be49d29f2413666fba7e69" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:34.730948 1670 raft_consensus.cc:515] T b17d2a9ad6be49d29f2413666fba7e69 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:34.731564 775 raft_consensus.cc:2393] T b17d2a9ad6be49d29f2413666fba7e69 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3aea2869ff814cc7b142012faa274382 in current term 1: Already voted for candidate 23eeffd3de254ce18db5febfde971c6b in this term.
I20251025 07:52:34.731602 776 raft_consensus.cc:2468] T 887e7c7a5d0e4aaf88f26246306df19c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:34.732169 847 leader_election.cc:304] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:34.732498 1561 raft_consensus.cc:2749] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251025 07:52:34.732263 1218 leader_election.cc:290] T 887e7c7a5d0e4aaf88f26246306df19c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:34.732672 847 leader_election.cc:304] T 887e7c7a5d0e4aaf88f26246306df19c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:34.732817 1167 consensus_queue.cc:260] T f0bdaa8cfd764f1b83f47881353d26b6 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.733059 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "887e7c7a5d0e4aaf88f26246306df19c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:34.735179 850 leader_election.cc:343] T 887e7c7a5d0e4aaf88f26246306df19c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:34.737825 1218 raft_consensus.cc:2804] T 887e7c7a5d0e4aaf88f26246306df19c P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:34.738041 1218 raft_consensus.cc:697] T 887e7c7a5d0e4aaf88f26246306df19c P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:34.738535 1218 consensus_queue.cc:237] T 887e7c7a5d0e4aaf88f26246306df19c P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.739879 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b17d2a9ad6be49d29f2413666fba7e69" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:34.740118 912 raft_consensus.cc:2393] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 23eeffd3de254ce18db5febfde971c6b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:34.740072 1670 leader_election.cc:290] T b17d2a9ad6be49d29f2413666fba7e69 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:34.740801 1167 ts_tablet_manager.cc:1434] T f0bdaa8cfd764f1b83f47881353d26b6 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.035s user 0.003s sys 0.000s
I20251025 07:52:34.745445 1167 tablet_bootstrap.cc:492] T 2209089667dd4937abd96615f6929449 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:34.746312 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b17d2a9ad6be49d29f2413666fba7e69" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:34.746955 712 leader_election.cc:343] T b17d2a9ad6be49d29f2413666fba7e69 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:34.748943 712 leader_election.cc:304] T b17d2a9ad6be49d29f2413666fba7e69 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:34.749330 1670 raft_consensus.cc:2749] T b17d2a9ad6be49d29f2413666fba7e69 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251025 07:52:34.751071 1167 tablet_bootstrap.cc:654] T 2209089667dd4937abd96615f6929449 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.753561 1670 raft_consensus.cc:493] T 7f29b64704e147aa9b0b5f6cda860f09 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:34.753738 1670 raft_consensus.cc:515] T 7f29b64704e147aa9b0b5f6cda860f09 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.754431 1670 leader_election.cc:290] T 7f29b64704e147aa9b0b5f6cda860f09 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:34.754878 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f29b64704e147aa9b0b5f6cda860f09" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:34.755112 1046 raft_consensus.cc:2468] T 7f29b64704e147aa9b0b5f6cda860f09 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:34.755707 712 leader_election.cc:304] T 7f29b64704e147aa9b0b5f6cda860f09 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:34.756014 1670 raft_consensus.cc:2804] T 7f29b64704e147aa9b0b5f6cda860f09 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:34.756116 1670 raft_consensus.cc:493] T 7f29b64704e147aa9b0b5f6cda860f09 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:34.756224 1670 raft_consensus.cc:3060] T 7f29b64704e147aa9b0b5f6cda860f09 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.758195 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f29b64704e147aa9b0b5f6cda860f09" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:34.758423 912 raft_consensus.cc:2468] T 7f29b64704e147aa9b0b5f6cda860f09 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:34.759110 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_380ca630f4144f40a04084d2ef8600e3 [id=06b6a39af6764e44972d57ddd722f838]
I20251025 07:52:34.760275 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_92cbcd8230f249a8b2821f69a81b41f1 [id=08e53121e86d405dbbd02a4605695fec]
I20251025 07:52:34.760192 1670 raft_consensus.cc:515] T 7f29b64704e147aa9b0b5f6cda860f09 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.761525 1169 tablet_bootstrap.cc:492] T fa77085e81c7440d9e255c44d1e80c26 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:34.761673 1169 ts_tablet_manager.cc:1403] T fa77085e81c7440d9e255c44d1e80c26 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.003s
I20251025 07:52:34.761675 1670 leader_election.cc:290] T 7f29b64704e147aa9b0b5f6cda860f09 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:34.762399 1169 raft_consensus.cc:359] T fa77085e81c7440d9e255c44d1e80c26 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.762563 1169 raft_consensus.cc:385] T fa77085e81c7440d9e255c44d1e80c26 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.762641 1169 raft_consensus.cc:740] T fa77085e81c7440d9e255c44d1e80c26 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.762835 1169 consensus_queue.cc:260] T fa77085e81c7440d9e255c44d1e80c26 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.762849 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f29b64704e147aa9b0b5f6cda860f09" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:34.763089 912 raft_consensus.cc:3060] T 7f29b64704e147aa9b0b5f6cda860f09 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.763293 1169 ts_tablet_manager.cc:1434] T fa77085e81c7440d9e255c44d1e80c26 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:34.763597 1169 tablet_bootstrap.cc:492] T 0b21b7da1e2f47829c3f6cdaee689e8a P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:34.765179 1169 tablet_bootstrap.cc:654] T 0b21b7da1e2f47829c3f6cdaee689e8a P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.767204 912 raft_consensus.cc:2468] T 7f29b64704e147aa9b0b5f6cda860f09 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:34.767767 710 leader_election.cc:304] T 7f29b64704e147aa9b0b5f6cda860f09 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:34.768092 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_5237c35dbfa7429e803d5008a8b3487c [id=0d597ecab6f54d1ba6cb5bd3436a9db8]
I20251025 07:52:34.768139 1670 raft_consensus.cc:2804] T 7f29b64704e147aa9b0b5f6cda860f09 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:34.768301 1670 raft_consensus.cc:697] T 7f29b64704e147aa9b0b5f6cda860f09 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:34.768574 1670 consensus_queue.cc:237] T 7f29b64704e147aa9b0b5f6cda860f09 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.769124 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_c43ac1adc76d4ccf819411ec76775932 [id=106e3728b0b846178eb48ccbf076e24e]
I20251025 07:52:34.769361 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f29b64704e147aa9b0b5f6cda860f09" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:34.769543 1046 raft_consensus.cc:3060] T 7f29b64704e147aa9b0b5f6cda860f09 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.769614 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_bc1398c7c53846da95b511e6a442e416 [id=10e2dcf046e241a9b881adabc51defbc]
I20251025 07:52:34.770795 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_19a3326e4da345c1bbeb2cf083d28e24 [id=11c088570f6940b1895d1691465c10b7]
I20251025 07:52:34.771803 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_6fd078e1af7a41f8a24caebf5e12e2b2 [id=14ce70a07f554566bf85c2e37c80fa65]
I20251025 07:52:34.772454 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_cd12897d1fc5454ab07cdc13fefb6978 [id=1655781ab7a046ad8e6dcfacaf2db1b1]
I20251025 07:52:34.773093 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_eafaca066a2548a686050fed8c380553 [id=17525d0c9b664dcdadbf42e85da234ca]
I20251025 07:52:34.774148 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_78724078b4574f35879fd2db388fc4f4 [id=187bde879b044237ada58158eb000123]
I20251025 07:52:34.775496 1167 tablet_bootstrap.cc:492] T 2209089667dd4937abd96615f6929449 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:34.775750 1167 ts_tablet_manager.cc:1403] T 2209089667dd4937abd96615f6929449 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20251025 07:52:34.776818 1167 raft_consensus.cc:359] T 2209089667dd4937abd96615f6929449 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.777132 1167 raft_consensus.cc:385] T 2209089667dd4937abd96615f6929449 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.777281 1167 raft_consensus.cc:740] T 2209089667dd4937abd96615f6929449 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.777571 1046 raft_consensus.cc:2468] T 7f29b64704e147aa9b0b5f6cda860f09 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:34.777653 1167 consensus_queue.cc:260] T 2209089667dd4937abd96615f6929449 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.778734 1167 ts_tablet_manager.cc:1434] T 2209089667dd4937abd96615f6929449 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251025 07:52:34.778649 1166 tablet_bootstrap.cc:492] T 3775a2e08d4e40bd9f57e4cda759d535 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:34.779263 1166 ts_tablet_manager.cc:1403] T 3775a2e08d4e40bd9f57e4cda759d535 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.121s user 0.031s sys 0.023s
I20251025 07:52:34.779578 1167 tablet_bootstrap.cc:492] T 3775a2e08d4e40bd9f57e4cda759d535 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:34.780823 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_cd9190d6cb714bdbb763dd934b33c10a [id=1b5db2f5053c4cb3a3b40eb109d5d5d9]
I20251025 07:52:34.781826 1169 tablet_bootstrap.cc:492] T 0b21b7da1e2f47829c3f6cdaee689e8a P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:34.782156 1169 ts_tablet_manager.cc:1403] T 0b21b7da1e2f47829c3f6cdaee689e8a P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.001s
I20251025 07:52:34.782963 1167 tablet_bootstrap.cc:654] T 3775a2e08d4e40bd9f57e4cda759d535 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.783470 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_2c8bad53187a49858df0c2bde3485e61 [id=1d22c44df6d24fed9c7009b7c7bd5a70]
I20251025 07:52:34.783567 1169 raft_consensus.cc:359] T 0b21b7da1e2f47829c3f6cdaee689e8a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.784207 1169 raft_consensus.cc:385] T 0b21b7da1e2f47829c3f6cdaee689e8a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.784407 1169 raft_consensus.cc:740] T 0b21b7da1e2f47829c3f6cdaee689e8a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.785041 1169 consensus_queue.cc:260] T 0b21b7da1e2f47829c3f6cdaee689e8a P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.786095 1169 ts_tablet_manager.cc:1434] T 0b21b7da1e2f47829c3f6cdaee689e8a P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20251025 07:52:34.786190 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_cf3a2c0772674c2a8ff5fee551a93a0f [id=1ffca9f550ce498296d664517e9fba86]
I20251025 07:52:34.786720 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_f24cd210ae3342e68c69aa6670e3e4b0 [id=22cab1e94aa544fa828350f9a70f8555]
I20251025 07:52:34.787042 1169 tablet_bootstrap.cc:492] T 887e7c7a5d0e4aaf88f26246306df19c P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:34.787544 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_68399b04a61f453da82d016afd85a32d [id=254e184d61d3414a8b460b4f95c408a9]
I20251025 07:52:34.788048 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_1f11da1b07914daabda9c8e29a9230fe [id=27557bb332dd49bb8f386c294bb12a69]
I20251025 07:52:34.788574 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_9f6b2f95721f488c86fde259a7cf95ec [id=277005e0d596465293065585a7555f4b]
I20251025 07:52:34.789081 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_6c5a24747ba14352b0285a41ec2985a4 [id=28bbaad474164a8c8c97a76bb8081791]
I20251025 07:52:34.789670 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f]
I20251025 07:52:34.790892 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_4eaf06c0720744ff8b2fc8866cf05e2b [id=2b66e5f736de408bbd711b88361e1c7b]
I20251025 07:52:34.793151 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5]
I20251025 07:52:34.793735 1166 raft_consensus.cc:359] T 3775a2e08d4e40bd9f57e4cda759d535 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.794219 1166 raft_consensus.cc:385] T 3775a2e08d4e40bd9f57e4cda759d535 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.794337 1166 raft_consensus.cc:740] T 3775a2e08d4e40bd9f57e4cda759d535 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.794732 1169 tablet_bootstrap.cc:654] T 887e7c7a5d0e4aaf88f26246306df19c P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.795094 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_00b3b394e5ed46faaac0202afe54b437 [id=304ef5bba5ed4e198622b8c8dc17f43a]
I20251025 07:52:34.795809 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_29aeaf089ca24270b3f21c2e08d61634 [id=3294d42240024d25ab2487e75640ddd0]
I20251025 07:52:34.797706 1167 tablet_bootstrap.cc:492] T 3775a2e08d4e40bd9f57e4cda759d535 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:34.797888 1167 ts_tablet_manager.cc:1403] T 3775a2e08d4e40bd9f57e4cda759d535 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.005s
I20251025 07:52:34.798311 1441 raft_consensus.cc:493] T fa77085e81c7440d9e255c44d1e80c26 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:34.798463 1441 raft_consensus.cc:515] T fa77085e81c7440d9e255c44d1e80c26 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.799115 1441 leader_election.cc:290] T fa77085e81c7440d9e255c44d1e80c26 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:34.799042 1167 raft_consensus.cc:359] T 3775a2e08d4e40bd9f57e4cda759d535 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.799250 1167 raft_consensus.cc:385] T 3775a2e08d4e40bd9f57e4cda759d535 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.799357 1167 raft_consensus.cc:740] T 3775a2e08d4e40bd9f57e4cda759d535 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.799599 1167 consensus_queue.cc:260] T 3775a2e08d4e40bd9f57e4cda759d535 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.799824 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa77085e81c7440d9e255c44d1e80c26" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:34.800029 1046 raft_consensus.cc:2468] T fa77085e81c7440d9e255c44d1e80c26 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:34.800588 1167 ts_tablet_manager.cc:1434] T 3775a2e08d4e40bd9f57e4cda759d535 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20251025 07:52:34.801195 712 leader_election.cc:304] T fa77085e81c7440d9e255c44d1e80c26 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:34.802222 1441 raft_consensus.cc:2804] T fa77085e81c7440d9e255c44d1e80c26 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
W20251025 07:52:34.802318 983 consensus_peers.cc:597] T 21e52928580c4762b0ae7152490ef955 P db42be7ad3c741a0aa427f27e245993b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21e52928580c4762b0ae7152490ef955. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:34.802353 1441 raft_consensus.cc:493] T fa77085e81c7440d9e255c44d1e80c26 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:34.802466 1441 raft_consensus.cc:3060] T fa77085e81c7440d9e255c44d1e80c26 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.806053 1441 raft_consensus.cc:515] T fa77085e81c7440d9e255c44d1e80c26 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.800920 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa77085e81c7440d9e255c44d1e80c26" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:34.807236 1166 consensus_queue.cc:260] T 3775a2e08d4e40bd9f57e4cda759d535 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.807898 1166 ts_tablet_manager.cc:1434] T 3775a2e08d4e40bd9f57e4cda759d535 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20251025 07:52:34.807204 912 raft_consensus.cc:2468] T fa77085e81c7440d9e255c44d1e80c26 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:34.809429 1441 leader_election.cc:290] T fa77085e81c7440d9e255c44d1e80c26 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:34.809844 1167 tablet_bootstrap.cc:492] T 8ff73ffc42434f74a4949f156771d5a4 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:34.811452 1166 tablet_bootstrap.cc:492] T 2209089667dd4937abd96615f6929449 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:34.813429 1166 tablet_bootstrap.cc:654] T 2209089667dd4937abd96615f6929449 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.814616 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa77085e81c7440d9e255c44d1e80c26" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:34.814848 913 raft_consensus.cc:3060] T fa77085e81c7440d9e255c44d1e80c26 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.815074 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa77085e81c7440d9e255c44d1e80c26" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:34.815287 1046 raft_consensus.cc:3060] T fa77085e81c7440d9e255c44d1e80c26 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.825932 1749 consensus_queue.cc:1048] T a13bf4d7eb1440f7a28b85cb1bf99667 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:34.827173 1046 raft_consensus.cc:2468] T fa77085e81c7440d9e255c44d1e80c26 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:34.827359 1670 raft_consensus.cc:493] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:34.827534 1670 raft_consensus.cc:515] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.828425 913 raft_consensus.cc:2468] T fa77085e81c7440d9e255c44d1e80c26 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:34.828833 1169 tablet_bootstrap.cc:492] T 887e7c7a5d0e4aaf88f26246306df19c P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:34.829332 1169 ts_tablet_manager.cc:1403] T 887e7c7a5d0e4aaf88f26246306df19c P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.000s
I20251025 07:52:34.828146 712 leader_election.cc:304] T fa77085e81c7440d9e255c44d1e80c26 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:34.829663 1670 leader_election.cc:290] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:34.830471 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7fbae8be73bf4e18b5a5c4ec4cdebc12" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:34.830628 1169 raft_consensus.cc:359] T 887e7c7a5d0e4aaf88f26246306df19c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.830691 1046 raft_consensus.cc:2468] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:34.830799 1169 raft_consensus.cc:385] T 887e7c7a5d0e4aaf88f26246306df19c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.830900 1169 raft_consensus.cc:740] T 887e7c7a5d0e4aaf88f26246306df19c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.831035 1167 tablet_bootstrap.cc:654] T 8ff73ffc42434f74a4949f156771d5a4 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.831125 1169 consensus_queue.cc:260] T 887e7c7a5d0e4aaf88f26246306df19c P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.831284 1670 raft_consensus.cc:2804] T fa77085e81c7440d9e255c44d1e80c26 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:34.831465 1670 raft_consensus.cc:697] T fa77085e81c7440d9e255c44d1e80c26 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:34.831665 1169 ts_tablet_manager.cc:1434] T 887e7c7a5d0e4aaf88f26246306df19c P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:34.831730 1670 consensus_queue.cc:237] T fa77085e81c7440d9e255c44d1e80c26 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.832058 1169 tablet_bootstrap.cc:492] T b17d2a9ad6be49d29f2413666fba7e69 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:34.833752 1169 tablet_bootstrap.cc:654] T b17d2a9ad6be49d29f2413666fba7e69 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.835492 1166 tablet_bootstrap.cc:492] T 2209089667dd4937abd96615f6929449 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:34.835741 1166 ts_tablet_manager.cc:1403] T 2209089667dd4937abd96615f6929449 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.004s
I20251025 07:52:34.836608 1166 raft_consensus.cc:359] T 2209089667dd4937abd96615f6929449 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.836869 1166 raft_consensus.cc:385] T 2209089667dd4937abd96615f6929449 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.836977 1166 raft_consensus.cc:740] T 2209089667dd4937abd96615f6929449 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.837291 1166 consensus_queue.cc:260] T 2209089667dd4937abd96615f6929449 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.837862 1166 ts_tablet_manager.cc:1434] T 2209089667dd4937abd96615f6929449 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251025 07:52:34.838238 712 leader_election.cc:304] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:34.838469 1166 tablet_bootstrap.cc:492] T 8ff73ffc42434f74a4949f156771d5a4 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:34.838577 1441 raft_consensus.cc:2804] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:34.838708 1441 raft_consensus.cc:493] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:34.838835 1441 raft_consensus.cc:3060] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.840082 1166 tablet_bootstrap.cc:654] T 8ff73ffc42434f74a4949f156771d5a4 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.848614 1675 consensus_queue.cc:1048] T 08da9d2687704725aacd11cd92c6a9f9 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:34.850788 1441 raft_consensus.cc:515] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.851557 1441 leader_election.cc:290] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:34.852025 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7fbae8be73bf4e18b5a5c4ec4cdebc12" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:34.852250 1046 raft_consensus.cc:3060] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.852550 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7fbae8be73bf4e18b5a5c4ec4cdebc12" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:34.852766 913 raft_consensus.cc:2468] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:34.855047 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7fbae8be73bf4e18b5a5c4ec4cdebc12" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:34.855435 913 raft_consensus.cc:3060] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.859340 1046 raft_consensus.cc:2468] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:34.861620 712 leader_election.cc:304] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:34.862088 1441 raft_consensus.cc:2804] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:34.862459 1441 raft_consensus.cc:697] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:34.864370 1441 consensus_queue.cc:237] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.865011 913 raft_consensus.cc:2468] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:34.868271 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_bcd5afa1072f47898d67504fc72b84c4 [id=3d6f0426d32f481ba8152eaacb298784]
I20251025 07:52:34.870064 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_d55201c0dabc4134b687cfbd805119e3 [id=3f7923f091ac46e0bc20bf7401b5458d]
I20251025 07:52:34.871238 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_2b9b772d4c234ac286bef3658482e490 [id=409fab60f9d744798e6f92b868f0f7c2]
I20251025 07:52:34.872361 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_7eb8424d5192485da1a2d67bfdb2bbbf [id=40c29df99590408cbd5c583d4fc741eb]
I20251025 07:52:34.872671 1675 raft_consensus.cc:493] T a6ddefa1738f4ff98586900c344359d2 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:34.872880 1675 raft_consensus.cc:515] T a6ddefa1738f4ff98586900c344359d2 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.873647 1675 leader_election.cc:290] T a6ddefa1738f4ff98586900c344359d2 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:34.874953 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6ddefa1738f4ff98586900c344359d2" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:34.875277 776 raft_consensus.cc:2468] T a6ddefa1738f4ff98586900c344359d2 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:34.875775 982 leader_election.cc:304] T a6ddefa1738f4ff98586900c344359d2 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:34.876188 1675 raft_consensus.cc:2804] T a6ddefa1738f4ff98586900c344359d2 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:34.876291 1675 raft_consensus.cc:493] T a6ddefa1738f4ff98586900c344359d2 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:34.876413 1675 raft_consensus.cc:3060] T a6ddefa1738f4ff98586900c344359d2 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.877954 1749 raft_consensus.cc:493] T 6897405bbc1c47a4ab0aa3af80e60e3d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:34.878100 1749 raft_consensus.cc:515] T 6897405bbc1c47a4ab0aa3af80e60e3d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.878899 1749 leader_election.cc:290] T 6897405bbc1c47a4ab0aa3af80e60e3d P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:34.879669 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6897405bbc1c47a4ab0aa3af80e60e3d" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:34.882817 1675 raft_consensus.cc:515] T a6ddefa1738f4ff98586900c344359d2 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.884038 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6ddefa1738f4ff98586900c344359d2" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:34.884275 912 raft_consensus.cc:3060] T a6ddefa1738f4ff98586900c344359d2 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.888535 912 raft_consensus.cc:2468] T a6ddefa1738f4ff98586900c344359d2 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:34.888870 1675 leader_election.cc:290] T a6ddefa1738f4ff98586900c344359d2 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:34.889061 983 leader_election.cc:304] T a6ddefa1738f4ff98586900c344359d2 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:34.889286 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6ddefa1738f4ff98586900c344359d2" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:34.889451 1675 raft_consensus.cc:2804] T a6ddefa1738f4ff98586900c344359d2 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:34.889735 1675 raft_consensus.cc:697] T a6ddefa1738f4ff98586900c344359d2 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:34.890158 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6897405bbc1c47a4ab0aa3af80e60e3d" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:34.890237 1675 consensus_queue.cc:237] T a6ddefa1738f4ff98586900c344359d2 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.891338 913 raft_consensus.cc:2376] T a6ddefa1738f4ff98586900c344359d2 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1. Re-sending same reply.
I20251025 07:52:34.892995 776 raft_consensus.cc:2393] T 6897405bbc1c47a4ab0aa3af80e60e3d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:34.893494 982 leader_election.cc:304] T 6897405bbc1c47a4ab0aa3af80e60e3d P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:34.894057 1749 raft_consensus.cc:3060] T 6897405bbc1c47a4ab0aa3af80e60e3d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.897990 1749 raft_consensus.cc:2749] T 6897405bbc1c47a4ab0aa3af80e60e3d P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:34.902907 1749 consensus_queue.cc:1048] T a13bf4d7eb1440f7a28b85cb1bf99667 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:34.915364 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_3292d47465544af0970067fb567e7ede [id=41fc0b99786749259c0a0059dec97ea1]
I20251025 07:52:34.916079 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_3709b01b0a9a48499f5bd229e3e9f31a [id=433d4cda7efb41349209f3e82cc41e3a]
I20251025 07:52:34.917410 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_ab6798879a554075b3deaa059ad077c7 [id=45325c188eac4d34bb016d01de091f87]
I20251025 07:52:34.918004 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_1431c97720de4d5a9a92bd2d574eb904 [id=484c28bccff0448bbca48cb7eaaa55df]
I20251025 07:52:34.919469 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_48712d8a17e04509888897a9cf2b4e30 [id=488b550bbdef476c95e8bad42bd4b4dd]
I20251025 07:52:34.925333 1169 tablet_bootstrap.cc:492] T b17d2a9ad6be49d29f2413666fba7e69 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:34.925535 1169 ts_tablet_manager.cc:1403] T b17d2a9ad6be49d29f2413666fba7e69 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.094s user 0.021s sys 0.016s
I20251025 07:52:34.926750 1169 raft_consensus.cc:359] T b17d2a9ad6be49d29f2413666fba7e69 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:34.926851 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6ddefa1738f4ff98586900c344359d2" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:34.927035 1169 raft_consensus.cc:385] T b17d2a9ad6be49d29f2413666fba7e69 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.927181 1169 raft_consensus.cc:740] T b17d2a9ad6be49d29f2413666fba7e69 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.927554 773 raft_consensus.cc:3060] T a6ddefa1738f4ff98586900c344359d2 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.927495 1169 consensus_queue.cc:260] T b17d2a9ad6be49d29f2413666fba7e69 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:34.927601 1441 raft_consensus.cc:493] T f3c87a83082b421cb4be546ac4ce7ad8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:34.927579 1659 raft_consensus.cc:493] T 2209089667dd4937abd96615f6929449 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:34.928102 1169 ts_tablet_manager.cc:1434] T b17d2a9ad6be49d29f2413666fba7e69 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:34.928433 1169 tablet_bootstrap.cc:492] T 3775a2e08d4e40bd9f57e4cda759d535 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:34.928439 1441 raft_consensus.cc:515] T f3c87a83082b421cb4be546ac4ce7ad8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.930439 1169 tablet_bootstrap.cc:654] T 3775a2e08d4e40bd9f57e4cda759d535 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.931581 1659 raft_consensus.cc:515] T 2209089667dd4937abd96615f6929449 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.932183 1659 leader_election.cc:290] T 2209089667dd4937abd96615f6929449 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:34.932230 1166 tablet_bootstrap.cc:492] T 8ff73ffc42434f74a4949f156771d5a4 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
W20251025 07:52:34.932324 710 consensus_peers.cc:597] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac0ea298bc114c56a8093e12def21ce1. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:34.932365 1166 ts_tablet_manager.cc:1403] T 8ff73ffc42434f74a4949f156771d5a4 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.094s user 0.010s sys 0.025s
I20251025 07:52:34.933279 1166 raft_consensus.cc:359] T 8ff73ffc42434f74a4949f156771d5a4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:34.933550 710 leader_election.cc:343] T f3c87a83082b421cb4be546ac4ce7ad8 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: f3c87a83082b421cb4be546ac4ce7ad8
I20251025 07:52:34.933600 1166 raft_consensus.cc:385] T 8ff73ffc42434f74a4949f156771d5a4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.933804 1166 raft_consensus.cc:740] T 8ff73ffc42434f74a4949f156771d5a4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.934075 1166 consensus_queue.cc:260] T 8ff73ffc42434f74a4949f156771d5a4 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.934232 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2209089667dd4937abd96615f6929449" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:34.934533 913 raft_consensus.cc:2468] T 2209089667dd4937abd96615f6929449 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:34.935107 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3c87a83082b421cb4be546ac4ce7ad8" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:34.935423 1166 ts_tablet_manager.cc:1434] T 8ff73ffc42434f74a4949f156771d5a4 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251025 07:52:34.935508 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2209089667dd4937abd96615f6929449" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:34.936362 1166 tablet_bootstrap.cc:492] T f0bdaa8cfd764f1b83f47881353d26b6 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:34.936796 1441 leader_election.cc:290] T f3c87a83082b421cb4be546ac4ce7ad8 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:34.937040 710 leader_election.cc:304] T 2209089667dd4937abd96615f6929449 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:34.937533 712 leader_election.cc:304] T f3c87a83082b421cb4be546ac4ce7ad8 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:34.937963 712 leader_election.cc:343] T 2209089667dd4937abd96615f6929449 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:34.938218 1441 raft_consensus.cc:2804] T 2209089667dd4937abd96615f6929449 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:34.939193 1166 tablet_bootstrap.cc:654] T f0bdaa8cfd764f1b83f47881353d26b6 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.940444 1441 raft_consensus.cc:493] T 2209089667dd4937abd96615f6929449 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:34.940560 1441 raft_consensus.cc:3060] T 2209089667dd4937abd96615f6929449 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.941586 1169 tablet_bootstrap.cc:492] T 3775a2e08d4e40bd9f57e4cda759d535 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:34.941715 1169 ts_tablet_manager.cc:1403] T 3775a2e08d4e40bd9f57e4cda759d535 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.001s
I20251025 07:52:34.942457 1169 raft_consensus.cc:359] T 3775a2e08d4e40bd9f57e4cda759d535 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.942625 1169 raft_consensus.cc:385] T 3775a2e08d4e40bd9f57e4cda759d535 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.942719 1169 raft_consensus.cc:740] T 3775a2e08d4e40bd9f57e4cda759d535 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.942920 1169 consensus_queue.cc:260] T 3775a2e08d4e40bd9f57e4cda759d535 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.943039 1659 raft_consensus.cc:2749] T f3c87a83082b421cb4be546ac4ce7ad8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:34.943457 1169 ts_tablet_manager.cc:1434] T 3775a2e08d4e40bd9f57e4cda759d535 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:34.943851 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3]
I20251025 07:52:34.944060 1169 tablet_bootstrap.cc:492] T 2209089667dd4937abd96615f6929449 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:34.946123 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_bf5cb8a733934f4ca29ecc9d473e4f2c [id=4bb4741007834a24aca34e608365ea03]
I20251025 07:52:34.946847 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_ea269532aff34ca1803fc1c5d5ad3641 [id=4c34b64b299f4162a53866cf01c5298f]
I20251025 07:52:34.947361 1166 tablet_bootstrap.cc:492] T f0bdaa8cfd764f1b83f47881353d26b6 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:34.947301 1441 raft_consensus.cc:515] T 2209089667dd4937abd96615f6929449 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.947660 1166 ts_tablet_manager.cc:1403] T f0bdaa8cfd764f1b83f47881353d26b6 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20251025 07:52:34.948266 1441 leader_election.cc:290] T 2209089667dd4937abd96615f6929449 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:34.948331 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_c26454d3c928434ab0dae167abcc2ff2 [id=4cd0c9a010af47939a90da1132b9e6be]
I20251025 07:52:34.948527 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2209089667dd4937abd96615f6929449" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:34.948750 913 raft_consensus.cc:3060] T 2209089667dd4937abd96615f6929449 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:34.948824 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_c78d2d5f287d45ab9318f5c8a4c1e7ae [id=4d099e6134f548b49dec1897f6660560]
I20251025 07:52:34.949318 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_e317f8d728da4c76886f67d0995ab7f4 [id=4d2121dea4904850aecb65da5dd2d8f9]
I20251025 07:52:34.949791 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]
I20251025 07:52:34.951041 1167 tablet_bootstrap.cc:492] T 8ff73ffc42434f74a4949f156771d5a4 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:34.951850 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_a437a2623d704e3ea3c5b8c65c88b936 [id=51b12903e6024e1a9adc2f9fa7400937]
I20251025 07:52:34.952139 913 raft_consensus.cc:2468] T 2209089667dd4937abd96615f6929449 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:34.952662 710 leader_election.cc:304] T 2209089667dd4937abd96615f6929449 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:34.953492 1166 raft_consensus.cc:359] T f0bdaa8cfd764f1b83f47881353d26b6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.953778 1166 raft_consensus.cc:385] T f0bdaa8cfd764f1b83f47881353d26b6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.953897 1166 raft_consensus.cc:740] T f0bdaa8cfd764f1b83f47881353d26b6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.954265 1166 consensus_queue.cc:260] T f0bdaa8cfd764f1b83f47881353d26b6 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.954686 1169 tablet_bootstrap.cc:654] T 2209089667dd4937abd96615f6929449 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.955498 1166 ts_tablet_manager.cc:1434] T f0bdaa8cfd764f1b83f47881353d26b6 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251025 07:52:34.957474 1166 tablet_bootstrap.cc:492] T 8956ad28b35d45f28c683ac055bba441 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:34.959056 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2209089667dd4937abd96615f6929449" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:34.959200 1166 tablet_bootstrap.cc:654] T 8956ad28b35d45f28c683ac055bba441 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.966827 1167 ts_tablet_manager.cc:1403] T 8ff73ffc42434f74a4949f156771d5a4 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.142s user 0.003s sys 0.007s
I20251025 07:52:34.966893 773 raft_consensus.cc:2468] T a6ddefa1738f4ff98586900c344359d2 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:34.967223 1166 tablet_bootstrap.cc:492] T 8956ad28b35d45f28c683ac055bba441 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
W20251025 07:52:34.967176 712 leader_election.cc:343] T 2209089667dd4937abd96615f6929449 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:34.967356 1166 ts_tablet_manager.cc:1403] T 8956ad28b35d45f28c683ac055bba441 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20251025 07:52:34.966841 1670 raft_consensus.cc:2804] T 2209089667dd4937abd96615f6929449 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:34.968195 1166 raft_consensus.cc:359] T 8956ad28b35d45f28c683ac055bba441 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.968354 1166 raft_consensus.cc:385] T 8956ad28b35d45f28c683ac055bba441 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.968436 1166 raft_consensus.cc:740] T 8956ad28b35d45f28c683ac055bba441 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.968636 1166 consensus_queue.cc:260] T 8956ad28b35d45f28c683ac055bba441 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.969226 1166 ts_tablet_manager.cc:1434] T 8956ad28b35d45f28c683ac055bba441 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20251025 07:52:34.969487 983 consensus_peers.cc:597] T 08da9d2687704725aacd11cd92c6a9f9 P db42be7ad3c741a0aa427f27e245993b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08da9d2687704725aacd11cd92c6a9f9. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:34.971753 1670 raft_consensus.cc:697] T 2209089667dd4937abd96615f6929449 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:34.972298 1670 consensus_queue.cc:237] T 2209089667dd4937abd96615f6929449 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.973515 1167 raft_consensus.cc:359] T 8ff73ffc42434f74a4949f156771d5a4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.974560 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_b8551dc7ed8c48228c5d1fec6a1e8b0c [id=52d442791ab94896b1e377af37ab6154]
I20251025 07:52:34.975736 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_5ecc1936fc8447669d0602cfd6133c6b [id=551028aad5c246378c7acb8e443853a7]
I20251025 07:52:34.977952 1167 raft_consensus.cc:385] T 8ff73ffc42434f74a4949f156771d5a4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:34.979493 1167 raft_consensus.cc:740] T 8ff73ffc42434f74a4949f156771d5a4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:34.979787 1166 tablet_bootstrap.cc:492] T cb824bd780524832b2b60f5618a39da5 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:34.981256 1167 consensus_queue.cc:260] T 8ff73ffc42434f74a4949f156771d5a4 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:34.981797 1166 tablet_bootstrap.cc:654] T cb824bd780524832b2b60f5618a39da5 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.982368 1167 ts_tablet_manager.cc:1434] T 8ff73ffc42434f74a4949f156771d5a4 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20251025 07:52:34.982785 1167 tablet_bootstrap.cc:492] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:34.985047 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_8f502389fe564ef3a4e7e23819a8a6e8 [id=56f4e62bc2354befa7d7374e07b103de]
I20251025 07:52:34.991031 1167 tablet_bootstrap.cc:654] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:34.991360 1441 consensus_queue.cc:1048] T fd9b4be139d347f4a6a60767029c342a P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:35.004562 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_a3f50508f4b047b6837c67248417b21c [id=572bc5595be947468e24f2a9144f12a4]
I20251025 07:52:35.005220 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_dad8e9c26a9a41e2bcf3fa1db9f5b087 [id=579250c3c8fb4e9783e388a53ba35a6c]
I20251025 07:52:35.005781 1670 consensus_queue.cc:1048] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:35.009117 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_4145d1c9c33c4351a89976c5c4c18d75 [id=58190b46780d49eb9f429626ddce531c]
I20251025 07:52:35.012534 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_fd5720d69b084df1b36485b7890ac6c2 [id=59840a567e5244e8a9024cbad42471e4]
I20251025 07:52:35.015189 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_51f2b43545db41029568a0321a37e139 [id=5a499fdacb8047dc9ce9bed92d7dd233]
I20251025 07:52:35.016625 1166 tablet_bootstrap.cc:492] T cb824bd780524832b2b60f5618a39da5 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:35.017340 1167 tablet_bootstrap.cc:492] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:35.017513 1167 ts_tablet_manager.cc:1403] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.002s
I20251025 07:52:35.017624 1169 tablet_bootstrap.cc:492] T 2209089667dd4937abd96615f6929449 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:35.018004 1169 ts_tablet_manager.cc:1403] T 2209089667dd4937abd96615f6929449 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.074s user 0.019s sys 0.001s
I20251025 07:52:35.018581 1167 raft_consensus.cc:359] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.018774 1167 raft_consensus.cc:385] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:35.018873 1167 raft_consensus.cc:740] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:35.019043 1169 raft_consensus.cc:359] T 2209089667dd4937abd96615f6929449 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.019134 1167 consensus_queue.cc:260] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.019333 1169 raft_consensus.cc:385] T 2209089667dd4937abd96615f6929449 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:35.019454 1169 raft_consensus.cc:740] T 2209089667dd4937abd96615f6929449 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:35.019824 1169 consensus_queue.cc:260] T 2209089667dd4937abd96615f6929449 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.020411 1169 ts_tablet_manager.cc:1434] T 2209089667dd4937abd96615f6929449 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:35.020946 1169 tablet_bootstrap.cc:492] T 8ff73ffc42434f74a4949f156771d5a4 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:35.022972 1169 tablet_bootstrap.cc:654] T 8ff73ffc42434f74a4949f156771d5a4 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:35.023212 1166 ts_tablet_manager.cc:1403] T cb824bd780524832b2b60f5618a39da5 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.016s
I20251025 07:52:35.024101 1166 raft_consensus.cc:359] T cb824bd780524832b2b60f5618a39da5 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.024540 1166 raft_consensus.cc:385] T cb824bd780524832b2b60f5618a39da5 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:35.024672 1166 raft_consensus.cc:740] T cb824bd780524832b2b60f5618a39da5 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:35.024974 1166 consensus_queue.cc:260] T cb824bd780524832b2b60f5618a39da5 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.025408 1659 consensus_queue.cc:1048] T fd9b4be139d347f4a6a60767029c342a P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:35.025652 1166 ts_tablet_manager.cc:1434] T cb824bd780524832b2b60f5618a39da5 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:35.027037 1166 tablet_bootstrap.cc:492] T bef0ef23d90d480d9c99aef21d7271fd P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:35.027037 1167 ts_tablet_manager.cc:1434] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251025 07:52:35.028247 1167 tablet_bootstrap.cc:492] T cb824bd780524832b2b60f5618a39da5 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:35.029016 1166 tablet_bootstrap.cc:654] T bef0ef23d90d480d9c99aef21d7271fd P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:35.029657 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_d647b0fa975743998d7f49c36abe9c96 [id=5b435831d8254ae8b126218a39e3a362]
I20251025 07:52:35.030215 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3]
I20251025 07:52:35.031589 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_d5ce3dfc91174e268c17c2d5b3f93b86 [id=607c24e5c581467db1eac04c5165450d]
I20251025 07:52:35.032078 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_6eb139482bc940989a606a0d8745d54f [id=616b3a99164f47fbb918c4e91d0c2e14]
I20251025 07:52:35.032822 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_974f1ee455d7488593e5c496cc012099 [id=630c274339664b9db18ab4ef7c29c16f]
I20251025 07:52:35.033308 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_2bcb67fcbc85452faaadfa683987951c [id=63e72692d0b6442396189a2f37170815]
I20251025 07:52:35.034811 1167 tablet_bootstrap.cc:654] T cb824bd780524832b2b60f5618a39da5 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:35.038341 1166 tablet_bootstrap.cc:492] T bef0ef23d90d480d9c99aef21d7271fd P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:35.038718 1166 ts_tablet_manager.cc:1403] T bef0ef23d90d480d9c99aef21d7271fd P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.002s
I20251025 07:52:35.039929 1166 raft_consensus.cc:359] T bef0ef23d90d480d9c99aef21d7271fd P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.041757 1169 tablet_bootstrap.cc:492] T 8ff73ffc42434f74a4949f156771d5a4 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:35.041966 1169 ts_tablet_manager.cc:1403] T 8ff73ffc42434f74a4949f156771d5a4 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.007s
I20251025 07:52:35.042886 1169 raft_consensus.cc:359] T 8ff73ffc42434f74a4949f156771d5a4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.043124 1169 raft_consensus.cc:385] T 8ff73ffc42434f74a4949f156771d5a4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:35.043256 1169 raft_consensus.cc:740] T 8ff73ffc42434f74a4949f156771d5a4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:35.043558 1169 consensus_queue.cc:260] T 8ff73ffc42434f74a4949f156771d5a4 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.044134 1169 ts_tablet_manager.cc:1434] T 8ff73ffc42434f74a4949f156771d5a4 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:35.045113 1169 tablet_bootstrap.cc:492] T f0bdaa8cfd764f1b83f47881353d26b6 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:35.045228 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_73279bdee234454e9def34441da9579a [id=6430d82b82c042469a77ea9f4dad15fc]
I20251025 07:52:35.047008 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_0ebd49dea97c4b0790902d42d65e749f [id=68d232ecc08c4f8db462fc0ad620e9be]
I20251025 07:52:35.047250 1166 raft_consensus.cc:385] T bef0ef23d90d480d9c99aef21d7271fd P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:35.047456 1166 raft_consensus.cc:740] T bef0ef23d90d480d9c99aef21d7271fd P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:35.047758 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_3f081b7bb4e1420c8121ea4115439f3d [id=696a1285fbfe4038a8b1b38708c8ea49]
I20251025 07:52:35.047881 1166 consensus_queue.cc:260] T bef0ef23d90d480d9c99aef21d7271fd P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.052292 1167 tablet_bootstrap.cc:492] T cb824bd780524832b2b60f5618a39da5 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:35.052469 1167 ts_tablet_manager.cc:1403] T cb824bd780524832b2b60f5618a39da5 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20251025 07:52:35.053085 1169 tablet_bootstrap.cc:654] T f0bdaa8cfd764f1b83f47881353d26b6 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:35.052518 1166 ts_tablet_manager.cc:1434] T bef0ef23d90d480d9c99aef21d7271fd P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20251025 07:52:35.053318 1167 raft_consensus.cc:359] T cb824bd780524832b2b60f5618a39da5 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.053637 1166 tablet_bootstrap.cc:492] T 1d261e432cf5403ea162129706bc4d4d P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:35.054394 1167 raft_consensus.cc:385] T cb824bd780524832b2b60f5618a39da5 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:35.054502 1167 raft_consensus.cc:740] T cb824bd780524832b2b60f5618a39da5 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:35.054776 1167 consensus_queue.cc:260] T cb824bd780524832b2b60f5618a39da5 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.055660 1167 ts_tablet_manager.cc:1434] T cb824bd780524832b2b60f5618a39da5 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251025 07:52:35.056502 1167 tablet_bootstrap.cc:492] T 87268e27540c4ec58ce1e62df819f19d P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:35.056533 1166 tablet_bootstrap.cc:654] T 1d261e432cf5403ea162129706bc4d4d P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:35.063473 1167 tablet_bootstrap.cc:654] T 87268e27540c4ec58ce1e62df819f19d P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:35.067375 1725 raft_consensus.cc:493] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b2dfbd0b44a441c9be9254f660974944)
I20251025 07:52:35.067538 1725 raft_consensus.cc:515] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } }
I20251025 07:52:35.068270 1725 leader_election.cc:290] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b2dfbd0b44a441c9be9254f660974944 (127.0.104.62:37421), 8603bddd8a084ff7a73a25641768dfa9 (127.0.104.61:42203)
I20251025 07:52:35.069031 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_173708dfd0274b34a7db6fa2b239a8b0 [id=6bc5ca21f8c54249a5150f0fea2686c7]
I20251025 07:52:35.070955 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_646f48a21c194af3a1611e5e12b2e11b [id=6f395892b09240fea02482b3199cfb5e]
I20251025 07:52:35.071734 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_3353358c7aab4407880b823e1500b6d3 [id=70586b36140141d9b7579a32df907ae5]
I20251025 07:52:35.074088 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c]
I20251025 07:52:35.080324 1441 raft_consensus.cc:493] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:35.080539 1441 raft_consensus.cc:515] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.081159 1441 leader_election.cc:290] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:35.081282 1749 consensus_queue.cc:1048] T 21e52928580c4762b0ae7152490ef955 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:35.082048 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bef0ef23d90d480d9c99aef21d7271fd" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:35.082288 912 raft_consensus.cc:2468] T bef0ef23d90d480d9c99aef21d7271fd P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:35.082713 710 leader_election.cc:304] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:35.083096 1441 raft_consensus.cc:2804] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:35.084061 1169 tablet_bootstrap.cc:492] T f0bdaa8cfd764f1b83f47881353d26b6 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:35.084245 1169 ts_tablet_manager.cc:1403] T f0bdaa8cfd764f1b83f47881353d26b6 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.003s
I20251025 07:52:35.085248 1441 raft_consensus.cc:493] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:35.085180 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bef0ef23d90d480d9c99aef21d7271fd" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:35.085400 1441 raft_consensus.cc:3060] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:35.086496 712 leader_election.cc:343] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:35.087608 1749 raft_consensus.cc:493] T 2209089667dd4937abd96615f6929449 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:35.087776 1749 raft_consensus.cc:515] T 2209089667dd4937abd96615f6929449 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.088366 1749 leader_election.cc:290] T 2209089667dd4937abd96615f6929449 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:35.088963 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2209089667dd4937abd96615f6929449" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:35.089319 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2209089667dd4937abd96615f6929449" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:35.089310 1441 raft_consensus.cc:515] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.089535 912 raft_consensus.cc:2393] T 2209089667dd4937abd96615f6929449 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 23eeffd3de254ce18db5febfde971c6b in this term.
I20251025 07:52:35.085208 1169 raft_consensus.cc:359] T f0bdaa8cfd764f1b83f47881353d26b6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.090274 1169 raft_consensus.cc:385] T f0bdaa8cfd764f1b83f47881353d26b6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:35.090407 1169 raft_consensus.cc:740] T f0bdaa8cfd764f1b83f47881353d26b6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:35.090409 1441 leader_election.cc:290] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:35.090696 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bef0ef23d90d480d9c99aef21d7271fd" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:35.090674 1169 consensus_queue.cc:260] T f0bdaa8cfd764f1b83f47881353d26b6 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.090902 912 raft_consensus.cc:3060] T bef0ef23d90d480d9c99aef21d7271fd P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.091291 1169 ts_tablet_manager.cc:1434] T f0bdaa8cfd764f1b83f47881353d26b6 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251025 07:52:35.094007 1169 tablet_bootstrap.cc:492] T cb824bd780524832b2b60f5618a39da5 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:35.095105 983 leader_election.cc:304] T 2209089667dd4937abd96615f6929449 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:35.095849 1169 tablet_bootstrap.cc:654] T cb824bd780524832b2b60f5618a39da5 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:35.097432 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bef0ef23d90d480d9c99aef21d7271fd" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:35.097988 1749 raft_consensus.cc:3060] T 2209089667dd4937abd96615f6929449 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:35.103020 983 consensus_peers.cc:597] T f3c87a83082b421cb4be546ac4ce7ad8 P db42be7ad3c741a0aa427f27e245993b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3c87a83082b421cb4be546ac4ce7ad8. This is attempt 1: this message will repeat every 5th retry.
W20251025 07:52:35.102958 712 leader_election.cc:343] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:35.104144 912 raft_consensus.cc:2468] T bef0ef23d90d480d9c99aef21d7271fd P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:35.105643 710 leader_election.cc:304] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:35.102069 1749 raft_consensus.cc:2749] T 2209089667dd4937abd96615f6929449 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:35.106160 1441 raft_consensus.cc:2804] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:35.106328 1441 raft_consensus.cc:697] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:35.106606 1441 consensus_queue.cc:237] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.114876 1167 tablet_bootstrap.cc:492] T 87268e27540c4ec58ce1e62df819f19d P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:35.115166 1167 ts_tablet_manager.cc:1403] T 87268e27540c4ec58ce1e62df819f19d P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.001s
I20251025 07:52:35.116009 1169 tablet_bootstrap.cc:492] T cb824bd780524832b2b60f5618a39da5 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:35.116338 1169 ts_tablet_manager.cc:1403] T cb824bd780524832b2b60f5618a39da5 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.014s
I20251025 07:52:35.116819 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_5261ed9b1f29420eab473df75f903987 [id=771a5d1dc57e4dffbfe4bf757f57719c]
I20251025 07:52:35.117136 1167 raft_consensus.cc:359] T 87268e27540c4ec58ce1e62df819f19d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.117372 1167 raft_consensus.cc:385] T 87268e27540c4ec58ce1e62df819f19d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:35.118175 1167 raft_consensus.cc:740] T 87268e27540c4ec58ce1e62df819f19d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:35.118495 1169 raft_consensus.cc:359] T cb824bd780524832b2b60f5618a39da5 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.118871 1169 raft_consensus.cc:385] T cb824bd780524832b2b60f5618a39da5 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:35.119107 1169 raft_consensus.cc:740] T cb824bd780524832b2b60f5618a39da5 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
W20251025 07:52:35.119155 1341 leader_election.cc:336] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8603bddd8a084ff7a73a25641768dfa9 (127.0.104.61:42203): Network error: Client connection negotiation failed: client connection to 127.0.104.61:42203: connect: Connection refused (error 111)
I20251025 07:52:35.119541 1169 consensus_queue.cc:260] T cb824bd780524832b2b60f5618a39da5 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.120455 1169 ts_tablet_manager.cc:1434] T cb824bd780524832b2b60f5618a39da5 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20251025 07:52:35.121011 1169 tablet_bootstrap.cc:492] T be5ab0db6ac34c2c9a370445e251eb46 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:35.121570 1166 tablet_bootstrap.cc:492] T 1d261e432cf5403ea162129706bc4d4d P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:35.121739 1166 ts_tablet_manager.cc:1403] T 1d261e432cf5403ea162129706bc4d4d P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.068s user 0.015s sys 0.006s
I20251025 07:52:35.122710 1166 raft_consensus.cc:359] T 1d261e432cf5403ea162129706bc4d4d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.122917 1166 raft_consensus.cc:385] T 1d261e432cf5403ea162129706bc4d4d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:35.122992 1166 raft_consensus.cc:740] T 1d261e432cf5403ea162129706bc4d4d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:35.123252 1166 consensus_queue.cc:260] T 1d261e432cf5403ea162129706bc4d4d P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.123636 1167 consensus_queue.cc:260] T 87268e27540c4ec58ce1e62df819f19d P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.124809 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]
I20251025 07:52:35.125272 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_ba4b8cc7b930482690b7e192e0edff12 [id=7944225d3ce7439b881bfb9289d9f040]
I20251025 07:52:35.129846 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_3cd469e1b1414bc7ad4a122483cb7755 [id=7b4db2984274499c98f6189d753f7856]
I20251025 07:52:35.130366 1169 tablet_bootstrap.cc:654] T be5ab0db6ac34c2c9a370445e251eb46 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:35.130479 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_438f67f3dc62465999a917986922b3ed [id=7c7fcd265e624d7fab74661db505d9c3]
I20251025 07:52:35.131122 773 raft_consensus.cc:3060] T f3c87a83082b421cb4be546ac4ce7ad8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.132210 1218 consensus_queue.cc:1048] T 467324e2432f4caca148951329caa292 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:35.135660 1569 consensus_queue.cc:1048] T f3c87a83082b421cb4be546ac4ce7ad8 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:35.137223 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]
I20251025 07:52:35.140112 1166 ts_tablet_manager.cc:1434] T 1d261e432cf5403ea162129706bc4d4d P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.018s user 0.000s sys 0.003s
I20251025 07:52:35.141321 1167 ts_tablet_manager.cc:1434] T 87268e27540c4ec58ce1e62df819f19d P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
I20251025 07:52:35.144492 1166 tablet_bootstrap.cc:492] T 87268e27540c4ec58ce1e62df819f19d P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:35.144680 1167 tablet_bootstrap.cc:492] T 1d261e432cf5403ea162129706bc4d4d P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:35.151648 1167 tablet_bootstrap.cc:654] T 1d261e432cf5403ea162129706bc4d4d P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:35.151582 1166 tablet_bootstrap.cc:654] T 87268e27540c4ec58ce1e62df819f19d P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:35.153764 1169 tablet_bootstrap.cc:492] T be5ab0db6ac34c2c9a370445e251eb46 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:35.154109 1169 ts_tablet_manager.cc:1403] T be5ab0db6ac34c2c9a370445e251eb46 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.006s
I20251025 07:52:35.155560 1169 raft_consensus.cc:359] T be5ab0db6ac34c2c9a370445e251eb46 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.156116 1169 raft_consensus.cc:385] T be5ab0db6ac34c2c9a370445e251eb46 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:35.156311 1169 raft_consensus.cc:740] T be5ab0db6ac34c2c9a370445e251eb46 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:35.156953 1169 consensus_queue.cc:260] T be5ab0db6ac34c2c9a370445e251eb46 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.160668 1169 ts_tablet_manager.cc:1434] T be5ab0db6ac34c2c9a370445e251eb46 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20251025 07:52:35.162163 1169 tablet_bootstrap.cc:492] T bef0ef23d90d480d9c99aef21d7271fd P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:35.173214 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_2ddc7ef45af34002a5db135b33ec855e [id=7e34aeb0c3ea4fc5ba6a9077a79335d1]
I20251025 07:52:35.175055 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_7f5a5345567a48d09dedce1df9d63a3b [id=7eaeb32957a946998e90658207639ed7]
I20251025 07:52:35.175140 1169 tablet_bootstrap.cc:654] T bef0ef23d90d480d9c99aef21d7271fd P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:35.176800 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_590ce35533c7442390ce8991bcfa80fb [id=7fd21f23b9ca483b8044392285400801]
I20251025 07:52:35.181094 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_088aad2a46b041bfa29ed8e6929760b1 [id=834669c54dae449790818684fd8659bc]
I20251025 07:52:35.182837 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_a55d3aa69b804240b042e7b94ca493d5 [id=83841f9dd7b34f27a36830af24022ab9]
I20251025 07:52:35.184751 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_ac4beca0c3bd4cf29565faf5323d9ff7 [id=857bcd1dbe2e477bbc0d77f79a58f7f7]
I20251025 07:52:35.188738 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_054db0ca31bd42ebb5d61ec81db8146b [id=85f4e4639f224541a0dec677ff5e62d5]
I20251025 07:52:35.189740 1561 consensus_queue.cc:1048] T 467324e2432f4caca148951329caa292 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:35.192296 1561 raft_consensus.cc:493] T 0b21b7da1e2f47829c3f6cdaee689e8a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:35.192459 1561 raft_consensus.cc:515] T 0b21b7da1e2f47829c3f6cdaee689e8a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.192612 1167 tablet_bootstrap.cc:492] T 1d261e432cf5403ea162129706bc4d4d P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:35.192835 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_68c19f91d7094d5ab0ab4f77f74a937e [id=867342b181604482af90da2976ef21a0]
I20251025 07:52:35.192915 1167 ts_tablet_manager.cc:1403] T 1d261e432cf5403ea162129706bc4d4d P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.009s
I20251025 07:52:35.193195 1561 leader_election.cc:290] T 0b21b7da1e2f47829c3f6cdaee689e8a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:35.194262 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_4d3dff3c6fda405f84600e93d3818380 [id=86fcd91f390d4661bcbb6d43190e6f2f]
I20251025 07:52:35.194468 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0b21b7da1e2f47829c3f6cdaee689e8a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:35.195266 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0b21b7da1e2f47829c3f6cdaee689e8a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:35.195317 774 raft_consensus.cc:2468] T 0b21b7da1e2f47829c3f6cdaee689e8a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:35.195500 1046 raft_consensus.cc:2468] T 0b21b7da1e2f47829c3f6cdaee689e8a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:35.195569 1167 raft_consensus.cc:359] T 1d261e432cf5403ea162129706bc4d4d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.196235 850 leader_election.cc:304] T 0b21b7da1e2f47829c3f6cdaee689e8a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:35.196717 1707 raft_consensus.cc:2804] T 0b21b7da1e2f47829c3f6cdaee689e8a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:35.196821 1707 raft_consensus.cc:493] T 0b21b7da1e2f47829c3f6cdaee689e8a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:35.196933 1707 raft_consensus.cc:3060] T 0b21b7da1e2f47829c3f6cdaee689e8a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.197909 1169 tablet_bootstrap.cc:492] T bef0ef23d90d480d9c99aef21d7271fd P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:35.198076 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_e244b2d2db74449aabd10a8b28e4afd2 [id=8bb9f6cbc77241f0b64c7ab15d1fe266]
I20251025 07:52:35.198222 1169 ts_tablet_manager.cc:1403] T bef0ef23d90d480d9c99aef21d7271fd P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.037s user 0.007s sys 0.002s
I20251025 07:52:35.198817 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_8d0e78b0bb8d433f9ed8e49d24d7e95d [id=8be005a6c0ba404da6ebed5da21f7498]
I20251025 07:52:35.199561 1169 raft_consensus.cc:359] T bef0ef23d90d480d9c99aef21d7271fd P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.199821 1169 raft_consensus.cc:385] T bef0ef23d90d480d9c99aef21d7271fd P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:35.199956 1169 raft_consensus.cc:740] T bef0ef23d90d480d9c99aef21d7271fd P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:35.200277 1169 consensus_queue.cc:260] T bef0ef23d90d480d9c99aef21d7271fd P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.200877 1707 raft_consensus.cc:515] T 0b21b7da1e2f47829c3f6cdaee689e8a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.201112 1169 ts_tablet_manager.cc:1434] T bef0ef23d90d480d9c99aef21d7271fd P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251025 07:52:35.201648 1169 tablet_bootstrap.cc:492] T 8956ad28b35d45f28c683ac055bba441 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:35.201804 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525]
I20251025 07:52:35.203460 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_4976d4f5d05540258d144751d129bdb8 [id=8cde66661cf749919839da894ed4dc05]
I20251025 07:52:35.203927 1707 leader_election.cc:290] T 0b21b7da1e2f47829c3f6cdaee689e8a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:35.204486 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0b21b7da1e2f47829c3f6cdaee689e8a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:35.204670 1046 raft_consensus.cc:3060] T 0b21b7da1e2f47829c3f6cdaee689e8a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.206523 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0b21b7da1e2f47829c3f6cdaee689e8a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:35.206830 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_ea39b0e2e2204f38b1c9b1d3346a1e36 [id=9170ed658b754ac8ad50f060c442f27c]
I20251025 07:52:35.207653 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_7955f4594e0e4f7c803f728a43c229ca [id=95040fbb66f94779b7721e2209d82a65]
I20251025 07:52:35.207942 1046 raft_consensus.cc:2468] T 0b21b7da1e2f47829c3f6cdaee689e8a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:35.208523 850 leader_election.cc:304] T 0b21b7da1e2f47829c3f6cdaee689e8a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:35.208940 1707 raft_consensus.cc:2804] T 0b21b7da1e2f47829c3f6cdaee689e8a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:35.209093 1707 raft_consensus.cc:697] T 0b21b7da1e2f47829c3f6cdaee689e8a P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:35.209362 1707 consensus_queue.cc:237] T 0b21b7da1e2f47829c3f6cdaee689e8a P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.212010 1167 raft_consensus.cc:385] T 1d261e432cf5403ea162129706bc4d4d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:35.212159 1167 raft_consensus.cc:740] T 1d261e432cf5403ea162129706bc4d4d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:35.212496 1167 consensus_queue.cc:260] T 1d261e432cf5403ea162129706bc4d4d P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.213189 1167 ts_tablet_manager.cc:1434] T 1d261e432cf5403ea162129706bc4d4d P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.020s user 0.003s sys 0.000s
I20251025 07:52:35.213608 1167 tablet_bootstrap.cc:492] T 71432a6848944a0da06ef9b81f160dc4 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:35.213920 774 raft_consensus.cc:3060] T 0b21b7da1e2f47829c3f6cdaee689e8a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.216145 1167 tablet_bootstrap.cc:654] T 71432a6848944a0da06ef9b81f160dc4 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:35.217695 1749 raft_consensus.cc:493] T 3775a2e08d4e40bd9f57e4cda759d535 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:35.217869 1749 raft_consensus.cc:515] T 3775a2e08d4e40bd9f57e4cda759d535 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.217938 774 raft_consensus.cc:2468] T 0b21b7da1e2f47829c3f6cdaee689e8a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:35.218590 1749 leader_election.cc:290] T 3775a2e08d4e40bd9f57e4cda759d535 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:35.219219 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3775a2e08d4e40bd9f57e4cda759d535" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:35.219383 1169 tablet_bootstrap.cc:654] T 8956ad28b35d45f28c683ac055bba441 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:35.219444 912 raft_consensus.cc:2468] T 3775a2e08d4e40bd9f57e4cda759d535 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:35.220237 983 leader_election.cc:304] T 3775a2e08d4e40bd9f57e4cda759d535 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:35.220253 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3775a2e08d4e40bd9f57e4cda759d535" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:35.220489 776 raft_consensus.cc:2468] T 3775a2e08d4e40bd9f57e4cda759d535 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:35.221236 1749 raft_consensus.cc:2804] T 3775a2e08d4e40bd9f57e4cda759d535 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:35.221344 1749 raft_consensus.cc:493] T 3775a2e08d4e40bd9f57e4cda759d535 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:35.221431 1749 raft_consensus.cc:3060] T 3775a2e08d4e40bd9f57e4cda759d535 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.221818 1441 raft_consensus.cc:493] T cb824bd780524832b2b60f5618a39da5 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:35.221972 1441 raft_consensus.cc:515] T cb824bd780524832b2b60f5618a39da5 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.222752 1441 leader_election.cc:290] T cb824bd780524832b2b60f5618a39da5 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:35.224174 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb824bd780524832b2b60f5618a39da5" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:35.224368 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb824bd780524832b2b60f5618a39da5" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:35.224414 912 raft_consensus.cc:2468] T cb824bd780524832b2b60f5618a39da5 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:35.224566 1046 raft_consensus.cc:2468] T cb824bd780524832b2b60f5618a39da5 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:35.224964 712 leader_election.cc:304] T cb824bd780524832b2b60f5618a39da5 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:35.225081 1749 raft_consensus.cc:515] T 3775a2e08d4e40bd9f57e4cda759d535 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.225359 1441 raft_consensus.cc:2804] T cb824bd780524832b2b60f5618a39da5 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:35.225476 1441 raft_consensus.cc:493] T cb824bd780524832b2b60f5618a39da5 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:35.225579 1441 raft_consensus.cc:3060] T cb824bd780524832b2b60f5618a39da5 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.226374 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3775a2e08d4e40bd9f57e4cda759d535" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:35.226609 776 raft_consensus.cc:3060] T 3775a2e08d4e40bd9f57e4cda759d535 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.229205 1441 raft_consensus.cc:515] T cb824bd780524832b2b60f5618a39da5 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.229598 1561 raft_consensus.cc:493] T 1d261e432cf5403ea162129706bc4d4d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:35.229754 1561 raft_consensus.cc:515] T 1d261e432cf5403ea162129706bc4d4d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.230414 1561 leader_election.cc:290] T 1d261e432cf5403ea162129706bc4d4d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:35.230979 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d261e432cf5403ea162129706bc4d4d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:35.231156 774 raft_consensus.cc:2468] T 1d261e432cf5403ea162129706bc4d4d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:35.231195 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3775a2e08d4e40bd9f57e4cda759d535" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:35.231417 912 raft_consensus.cc:3060] T 3775a2e08d4e40bd9f57e4cda759d535 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.231585 847 leader_election.cc:304] T 1d261e432cf5403ea162129706bc4d4d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:35.231616 1441 leader_election.cc:290] T cb824bd780524832b2b60f5618a39da5 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:35.231806 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_d5ae5bc2cb804015ba50415cd3f62847 [id=951b49e5f0ca41488c2e92be79288158]
I20251025 07:52:35.231902 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d261e432cf5403ea162129706bc4d4d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:35.232084 1561 raft_consensus.cc:2804] T 1d261e432cf5403ea162129706bc4d4d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:35.232175 1561 raft_consensus.cc:493] T 1d261e432cf5403ea162129706bc4d4d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:35.232276 1561 raft_consensus.cc:3060] T 1d261e432cf5403ea162129706bc4d4d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.232391 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_350383f0e98947eba77186e74ad88781 [id=96b0e47c666c4d8f982741aea7b17ae7]
W20251025 07:52:35.232409 850 leader_election.cc:343] T 1d261e432cf5403ea162129706bc4d4d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:35.233081 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_4ea985df4c5b415c9570c64506a98cc4 [id=98d2b7797e33485e8872054aa2eb8ec9]
I20251025 07:52:35.234294 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb824bd780524832b2b60f5618a39da5" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:35.234450 1048 raft_consensus.cc:3060] T cb824bd780524832b2b60f5618a39da5 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.233647 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_3f291c48b0144f0a8ddb1b51ebbe31cd [id=98db26b2e7bf409dbfe8d201ae701df7]
I20251025 07:52:35.235714 1561 raft_consensus.cc:515] T 1d261e432cf5403ea162129706bc4d4d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.236145 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_6f5fbeb82b024bfe95511811dcc51406 [id=9c2312d7095f407d9d30f224cdce66c6]
I20251025 07:52:35.236545 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_750ecb905839434fa9a1aa72f87394be [id=9ce3ba77c56a4ef09cb3b40ea0e488f6]
I20251025 07:52:35.236636 1167 tablet_bootstrap.cc:492] T 71432a6848944a0da06ef9b81f160dc4 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:35.236792 1167 ts_tablet_manager.cc:1403] T 71432a6848944a0da06ef9b81f160dc4 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.003s
I20251025 07:52:35.237059 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d261e432cf5403ea162129706bc4d4d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:35.237144 1048 raft_consensus.cc:2468] T cb824bd780524832b2b60f5618a39da5 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:35.237227 774 raft_consensus.cc:3060] T 1d261e432cf5403ea162129706bc4d4d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.237607 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb824bd780524832b2b60f5618a39da5" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:35.237673 1167 raft_consensus.cc:359] T 71432a6848944a0da06ef9b81f160dc4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:35.237797 913 raft_consensus.cc:3060] T cb824bd780524832b2b60f5618a39da5 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.237850 1167 raft_consensus.cc:385] T 71432a6848944a0da06ef9b81f160dc4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:35.237946 1167 raft_consensus.cc:740] T 71432a6848944a0da06ef9b81f160dc4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:35.238185 1167 consensus_queue.cc:260] T 71432a6848944a0da06ef9b81f160dc4 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:35.238248 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d261e432cf5403ea162129706bc4d4d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:35.238911 1167 ts_tablet_manager.cc:1434] T 71432a6848944a0da06ef9b81f160dc4 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:35.239362 1167 tablet_bootstrap.cc:492] T c59f6e6eb9594a8e83f109a09b034045 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:35.240248 774 raft_consensus.cc:2468] T 1d261e432cf5403ea162129706bc4d4d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:35.240586 913 raft_consensus.cc:2468] T cb824bd780524832b2b60f5618a39da5 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:35.240797 847 leader_election.cc:304] T 1d261e432cf5403ea162129706bc4d4d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:35.240968 712 leader_election.cc:304] T cb824bd780524832b2b60f5618a39da5 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:35.241181 1218 raft_consensus.cc:2804] T 1d261e432cf5403ea162129706bc4d4d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:35.241250 1167 tablet_bootstrap.cc:654] T c59f6e6eb9594a8e83f109a09b034045 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:35.242617 912 raft_consensus.cc:2468] T 3775a2e08d4e40bd9f57e4cda759d535 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:35.243150 983 leader_election.cc:304] T 3775a2e08d4e40bd9f57e4cda759d535 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:35.243484 1612 raft_consensus.cc:2804] T 3775a2e08d4e40bd9f57e4cda759d535 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:35.243661 1612 raft_consensus.cc:697] T 3775a2e08d4e40bd9f57e4cda759d535 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:35.243960 1612 consensus_queue.cc:237] T 3775a2e08d4e40bd9f57e4cda759d535 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.244169 1169 tablet_bootstrap.cc:492] T 8956ad28b35d45f28c683ac055bba441 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:35.244372 1169 ts_tablet_manager.cc:1403] T 8956ad28b35d45f28c683ac055bba441 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.043s user 0.005s sys 0.000s
I20251025 07:52:35.245167 1169 raft_consensus.cc:359] T 8956ad28b35d45f28c683ac055bba441 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.245371 1169 raft_consensus.cc:385] T 8956ad28b35d45f28c683ac055bba441 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:35.245442 1169 raft_consensus.cc:740] T 8956ad28b35d45f28c683ac055bba441 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:35.245656 1169 consensus_queue.cc:260] T 8956ad28b35d45f28c683ac055bba441 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.245884 776 raft_consensus.cc:2468] T 3775a2e08d4e40bd9f57e4cda759d535 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:35.246776 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_07b5304bacea445c92e5dde5059008aa [id=9d6d15d545584125aaaf2d51aed95f2c]
I20251025 07:52:35.247139 1749 leader_election.cc:290] T 3775a2e08d4e40bd9f57e4cda759d535 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:35.247323 1659 raft_consensus.cc:2804] T cb824bd780524832b2b60f5618a39da5 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:35.247479 1659 raft_consensus.cc:697] T cb824bd780524832b2b60f5618a39da5 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:35.247886 1659 consensus_queue.cc:237] T cb824bd780524832b2b60f5618a39da5 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:35.248951 850 leader_election.cc:343] T 1d261e432cf5403ea162129706bc4d4d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:35.249192 1670 consensus_queue.cc:1048] T 8c76d1b509f04ab1a10b83cff675c11e P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:35.251343 1561 leader_election.cc:290] T 1d261e432cf5403ea162129706bc4d4d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:35.256711 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_2cd70245aa3d4809a0647e7fa499f62f [id=a003268ec3da425989fa4041699de4d3]
I20251025 07:52:35.257714 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_c45e312143074d47be4fac483ee16136 [id=a003a6f6574b4cde88d30928eaa7f4ec]
I20251025 07:52:35.262599 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_157e52b1e07f4356afc68bb3ec5e4eb0 [id=a0facec9e7314c59b2175f0196869d73]
I20251025 07:52:35.263068 1169 ts_tablet_manager.cc:1434] T 8956ad28b35d45f28c683ac055bba441 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20251025 07:52:35.263196 1749 consensus_queue.cc:1048] T fab67b56eaa44b54886c17181d8c52b2 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:35.263293 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRBHlYt/build/asan/bin/kudu
/tmp/dist-test-taskRBHlYt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:42203
--webserver_interface=127.0.104.61
--webserver_port=45311
--builtin_ntp_servers=127.0.104.20:40091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:37421,127.0.104.61:42203,127.0.104.60:37611
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251025 07:52:35.265522 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_9fa69b10b5e842639fc0b0a58cd9b6a5 [id=a253ce0253bd4da4814741c482193d8f]
I20251025 07:52:35.265894 1169 tablet_bootstrap.cc:492] T 1d261e432cf5403ea162129706bc4d4d P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:35.267795 1169 tablet_bootstrap.cc:654] T 1d261e432cf5403ea162129706bc4d4d P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:35.269595 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_d67fb372b12e40d082d119e53b9ebc3e [id=a4f956a8e567440cbb5ad5214a0aa10a]
I20251025 07:52:35.270823 1749 consensus_queue.cc:1048] T d37758f606344127b3db74c87babe69c P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:35.270932 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_23d4d4a3c90c47ea8230c39a51138418 [id=a7070a0269fc4fc99447ab671eb226be]
I20251025 07:52:35.271689 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_23cacc46bec647adbc62b457f73487cf [id=a8b081495f564c10aa6a05e5eed80488]
I20251025 07:52:35.272301 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_7ce70fea0fe04c52a62033afb8eef977 [id=a8b2a16fd037435893c8f60e92fd1627]
I20251025 07:52:35.272904 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_1de0b151cf4c4c10aca35e45623caf2c [id=a947f21398254a53bc6668c4f8d39e53]
I20251025 07:52:35.273697 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_91898f2a551144d18df0309c827855bf [id=aaa734f6549741768b0156bdf5cc25af]
I20251025 07:52:35.262393 1218 raft_consensus.cc:697] T 1d261e432cf5403ea162129706bc4d4d P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:35.276077 1167 tablet_bootstrap.cc:492] T c59f6e6eb9594a8e83f109a09b034045 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:35.276270 1167 ts_tablet_manager.cc:1403] T c59f6e6eb9594a8e83f109a09b034045 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.002s
I20251025 07:52:35.277177 1167 raft_consensus.cc:359] T c59f6e6eb9594a8e83f109a09b034045 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.277417 1167 raft_consensus.cc:385] T c59f6e6eb9594a8e83f109a09b034045 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:35.277509 1167 raft_consensus.cc:740] T c59f6e6eb9594a8e83f109a09b034045 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:35.277761 1167 consensus_queue.cc:260] T c59f6e6eb9594a8e83f109a09b034045 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.279381 1167 ts_tablet_manager.cc:1434] T c59f6e6eb9594a8e83f109a09b034045 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251025 07:52:35.280736 1167 tablet_bootstrap.cc:492] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:35.283041 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_87d38832f85e4965bfab1a077990114f [id=ad812862698d421184ebf501ce79b7be]
I20251025 07:52:35.283550 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_6140b1f390084256848ae1b746059268 [id=adaaf2bcd09f4c779aee11bb7cd4ba15]
I20251025 07:52:35.284332 1167 tablet_bootstrap.cc:654] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:35.285382 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_74277c3abac14953b2eb73c850e3e727 [id=afdcd9fdb01f4cad85633186bed13682]
I20251025 07:52:35.286198 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e]
I20251025 07:52:35.287765 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_2e41b219b43c4746bd24f0cb1e41547f [id=b087a7c6648d46a3b7d410cf8cc81e2c]
I20251025 07:52:35.288503 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_ab87db32c7024d709eee5476d53f17ce [id=b1458cb701e14086a5a7fd82db235571]
I20251025 07:52:35.289242 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84]
I20251025 07:52:35.289991 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_19354444047b451bbf22bafcf9ae185d [id=b3725485d48c41a1952091d028a783c0]
I20251025 07:52:35.291404 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7]
I20251025 07:52:35.292738 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_45f85cb4514d4ec3b06cff95dd4a70f6 [id=b7073684e63d4875841df3dd5171bcc2]
I20251025 07:52:35.295359 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_4e81dfac4c7742c7986eb24e6d54a124 [id=b87e43a3dc3c4cd98ee3f44964e8092f]
I20251025 07:52:35.295449 1169 tablet_bootstrap.cc:492] T 1d261e432cf5403ea162129706bc4d4d P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:35.295620 1169 ts_tablet_manager.cc:1403] T 1d261e432cf5403ea162129706bc4d4d P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.006s
I20251025 07:52:35.296701 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_309913c03a4d46019d35478363f149e8 [id=bad99c3045fc4b938e20d2dcfdd0b1bc]
I20251025 07:52:35.297541 1169 raft_consensus.cc:359] T 1d261e432cf5403ea162129706bc4d4d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.297765 1169 raft_consensus.cc:385] T 1d261e432cf5403ea162129706bc4d4d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:35.297885 1169 raft_consensus.cc:740] T 1d261e432cf5403ea162129706bc4d4d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:35.298169 1169 consensus_queue.cc:260] T 1d261e432cf5403ea162129706bc4d4d P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.298804 1167 tablet_bootstrap.cc:492] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:35.298871 1169 ts_tablet_manager.cc:1434] T 1d261e432cf5403ea162129706bc4d4d P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251025 07:52:35.299062 1167 ts_tablet_manager.cc:1403] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.001s
I20251025 07:52:35.299181 1169 tablet_bootstrap.cc:492] T 71432a6848944a0da06ef9b81f160dc4 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:35.300393 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_3d163f45f94f450ebf0447897bdf41a6 [id=bb316a7e000748ca902a9bfb494802d7]
I20251025 07:52:35.301122 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_45faec266a9b4b1b9f5ef3ac915e3bbc [id=bc377ac4bb164a80b9ebce289a790326]
I20251025 07:52:35.301399 1169 tablet_bootstrap.cc:654] T 71432a6848944a0da06ef9b81f160dc4 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:35.301982 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]
I20251025 07:52:35.303325 1167 raft_consensus.cc:359] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.303589 1167 raft_consensus.cc:385] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:35.303728 1167 raft_consensus.cc:740] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:35.304004 1167 consensus_queue.cc:260] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.304628 1167 ts_tablet_manager.cc:1434] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251025 07:52:35.304986 1167 tablet_bootstrap.cc:492] T c4610b0c62e741eca28c21f63cc3de81 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:35.302603 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_a6c869949b744bc98efb735b0e73e7aa [id=bf77b5ff6f1048dd996d948437f9ff5e]
I20251025 07:52:35.307214 1167 tablet_bootstrap.cc:654] T c4610b0c62e741eca28c21f63cc3de81 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:35.308705 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_7d2010ece331457fa18a0f13ac5cbdfa [id=c0e60203ceb3445ca63dc5f8f81c37b3]
I20251025 07:52:35.309785 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_32b54e82cbc84d39a0020826f8737d62 [id=c28b0b6acd61490caecfdaab803a9d92]
I20251025 07:52:35.310652 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_dde55dddf5d14de084003c22da08f6cd [id=c3f37f5469ab4d60a7597994c11357cf]
I20251025 07:52:35.313905 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_99fd36369f9a4e0fb14f234c98635203 [id=c8059ab6cec146efa631d06799d969a9]
I20251025 07:52:35.316154 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_17e5895bc86c441ba014d094c1bf3893 [id=c8c37f4cecb94242b3cea6e61a2c1cc4]
I20251025 07:52:35.317405 1166 tablet_bootstrap.cc:492] T 87268e27540c4ec58ce1e62df819f19d P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:35.317600 1166 ts_tablet_manager.cc:1403] T 87268e27540c4ec58ce1e62df819f19d P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.173s user 0.002s sys 0.029s
I20251025 07:52:35.318674 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_f16b7a3b7d91423f8502d73e4234ab4a [id=cc7db0a7eaea4caa8c17281a8079c372]
I20251025 07:52:35.319999 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_da4ed35a8ed24427b6f4499fd1cc203c [id=cdbb76e674fc41cfb0f86d678a3f4f0b]
I20251025 07:52:35.321138 1218 consensus_queue.cc:237] T 1d261e432cf5403ea162129706bc4d4d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.323030 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_d5c65dce8b1b4214ba519d964012bfc0 [id=d4be274b36a845a99a4b96b8846248fd]
I20251025 07:52:35.326786 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_208cd3f3c07744ed92b1260ebb401310 [id=d6c5598979e84e36a3bf1996f37f9c28]
I20251025 07:52:35.333580 774 raft_consensus.cc:3060] T 4037dee385d04e85bb78c8e77cb00379 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.334525 1829 consensus_queue.cc:1048] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:35.335615 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_dfc7f1d4bf8b48bebbb00313419c3fd8 [id=d82b3e1317434cf08a0d51e8a9427224]
I20251025 07:52:35.337194 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_a0665a4cb38b40109df3991cb2f71ed6 [id=dbd92b3210f24fc2b8146dcfcb18833a]
I20251025 07:52:35.338167 1612 consensus_queue.cc:1048] T d37758f606344127b3db74c87babe69c P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20251025 07:52:35.339938 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_bfcdc02fe0dc48a0ae793ecd15f50a70 [id=dec6d4b5bbe34710aed2d981e185f35e]
I20251025 07:52:35.342590 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_4791672658f043f6b97b9320a531ebb5 [id=dfd20cf9c3e541c4abe23d59e0aa7f2a]
I20251025 07:52:35.345417 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_332b308220f64c6dbb5f29711d8463b3 [id=e041946c093b4c789ce175c736cc230b]
I20251025 07:52:35.348371 1167 tablet_bootstrap.cc:492] T c4610b0c62e741eca28c21f63cc3de81 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:35.348770 1167 ts_tablet_manager.cc:1403] T c4610b0c62e741eca28c21f63cc3de81 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.044s user 0.006s sys 0.007s
I20251025 07:52:35.350522 1167 raft_consensus.cc:359] T c4610b0c62e741eca28c21f63cc3de81 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.351998 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_ddc8580fd1674f9ca61d528f22c51c94 [id=e33f00d4d5844ebba7ea3e200e154eeb]
I20251025 07:52:35.352553 1167 raft_consensus.cc:385] T c4610b0c62e741eca28c21f63cc3de81 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:35.352806 1167 raft_consensus.cc:740] T c4610b0c62e741eca28c21f63cc3de81 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:35.353368 1167 consensus_queue.cc:260] T c4610b0c62e741eca28c21f63cc3de81 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.354363 1167 ts_tablet_manager.cc:1434] T c4610b0c62e741eca28c21f63cc3de81 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20251025 07:52:35.356441 1169 tablet_bootstrap.cc:492] T 71432a6848944a0da06ef9b81f160dc4 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:35.356654 1169 ts_tablet_manager.cc:1403] T 71432a6848944a0da06ef9b81f160dc4 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.058s user 0.009s sys 0.000s
I20251025 07:52:35.357282 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_56a34b2e5e744f94a5a955e4a796498a [id=e35831cc8e8d47139dd92b95e78c6aee]
I20251025 07:52:35.357621 1169 raft_consensus.cc:359] T 71432a6848944a0da06ef9b81f160dc4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:35.357872 1169 raft_consensus.cc:385] T 71432a6848944a0da06ef9b81f160dc4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:35.357955 1169 raft_consensus.cc:740] T 71432a6848944a0da06ef9b81f160dc4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:35.358103 1569 raft_consensus.cc:493] T f0bdaa8cfd764f1b83f47881353d26b6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:35.358263 1569 raft_consensus.cc:515] T f0bdaa8cfd764f1b83f47881353d26b6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.358325 1169 consensus_queue.cc:260] T 71432a6848944a0da06ef9b81f160dc4 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:35.358489 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_915572e49894471d8933816077123d6c [id=e372ca4384644779a6b37f389f549dec]
I20251025 07:52:35.358807 1169 ts_tablet_manager.cc:1434] T 71432a6848944a0da06ef9b81f160dc4 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:35.358945 1569 leader_election.cc:290] T f0bdaa8cfd764f1b83f47881353d26b6 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:35.359141 1169 tablet_bootstrap.cc:492] T 87268e27540c4ec58ce1e62df819f19d P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:35.359401 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_bd168da90c704b7987fc731f5ab28045 [id=e7683b98787e4d55b81303f4b58e4a6b]
I20251025 07:52:35.360001 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_e386bc02f89041418ff3ddf8ac224eea [id=e800c96844e843b7a6fd3145907e5503]
I20251025 07:52:35.360924 1169 tablet_bootstrap.cc:654] T 87268e27540c4ec58ce1e62df819f19d P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:35.362601 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0bdaa8cfd764f1b83f47881353d26b6" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:35.362828 776 raft_consensus.cc:2468] T f0bdaa8cfd764f1b83f47881353d26b6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:35.365077 982 leader_election.cc:304] T f0bdaa8cfd764f1b83f47881353d26b6 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:35.365345 1671 raft_consensus.cc:2804] T f0bdaa8cfd764f1b83f47881353d26b6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:35.365460 1671 raft_consensus.cc:493] T f0bdaa8cfd764f1b83f47881353d26b6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:35.365541 1671 raft_consensus.cc:3060] T f0bdaa8cfd764f1b83f47881353d26b6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.367630 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_4b1446ea61a9442fb3e397dfe2c5fb03 [id=e9652929eaab46cab42bf8d6896577ed]
I20251025 07:52:35.368131 1167 tablet_bootstrap.cc:492] T 8956ad28b35d45f28c683ac055bba441 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:35.368402 1671 raft_consensus.cc:515] T f0bdaa8cfd764f1b83f47881353d26b6 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.368685 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_0b0bd6b71b92454fb8cdf967ffd8349e [id=e9b21659de7d4276aa1f9effb7076eb4]
I20251025 07:52:35.369629 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_8129e15bf5c549bd853c04b227a7ef26 [id=e9e8547061ba4ac987d0aa7616c7f3ea]
I20251025 07:52:35.370582 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_92c0120dd29348618040150dedb14c43 [id=ea892976886644d8aa025ae2edef718d]
I20251025 07:52:35.371728 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_9bde97409538452a99917ba1e4ed04b3 [id=eb909ab8da1d45e28008f447066bea71]
I20251025 07:52:35.372040 1829 consensus_queue.cc:1048] T 8c76d1b509f04ab1a10b83cff675c11e P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251025 07:52:35.372900 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0bdaa8cfd764f1b83f47881353d26b6" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:35.373124 776 raft_consensus.cc:3060] T f0bdaa8cfd764f1b83f47881353d26b6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.373937 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_34b975902f884c658093a51baa54b095 [id=ec0cc315ecba4988a97d68f00ff22697]
I20251025 07:52:35.374970 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_3b10c71d2c1b4bd2849e0cad9ec767d7 [id=ee09c3f16a194bbf97dd2768b5231c06]
I20251025 07:52:35.375128 1671 leader_election.cc:290] T f0bdaa8cfd764f1b83f47881353d26b6 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:35.376170 1167 tablet_bootstrap.cc:654] T 8956ad28b35d45f28c683ac055bba441 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:35.378021 776 raft_consensus.cc:2468] T f0bdaa8cfd764f1b83f47881353d26b6 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:35.378615 982 leader_election.cc:304] T f0bdaa8cfd764f1b83f47881353d26b6 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:35.379078 1671 raft_consensus.cc:2804] T f0bdaa8cfd764f1b83f47881353d26b6 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:35.379371 1671 raft_consensus.cc:697] T f0bdaa8cfd764f1b83f47881353d26b6 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:35.379696 1671 consensus_queue.cc:237] T f0bdaa8cfd764f1b83f47881353d26b6 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.381960 1569 consensus_queue.cc:1048] T 35ba9d23262248cda6fdf03c51be11a4 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:35.396034 1829 raft_consensus.cc:493] T 4f25c9caa05b4da4bd56a8a21fde423c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:35.396239 1829 raft_consensus.cc:515] T 4f25c9caa05b4da4bd56a8a21fde423c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:35.397084 1829 leader_election.cc:290] T 4f25c9caa05b4da4bd56a8a21fde423c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:35.398319 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f25c9caa05b4da4bd56a8a21fde423c" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:35.398561 1047 raft_consensus.cc:2468] T 4f25c9caa05b4da4bd56a8a21fde423c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:35.399176 712 leader_election.cc:304] T 4f25c9caa05b4da4bd56a8a21fde423c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:35.399609 1829 raft_consensus.cc:2804] T 4f25c9caa05b4da4bd56a8a21fde423c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:35.399744 1829 raft_consensus.cc:493] T 4f25c9caa05b4da4bd56a8a21fde423c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:35.399834 1829 raft_consensus.cc:3060] T 4f25c9caa05b4da4bd56a8a21fde423c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.398068 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_76158f8cf82e4cdb9a90580e46aaea48 [id=f249f30497bc450c887d659721469de8]
I20251025 07:52:35.405884 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_6aaf0d6ed7554bd4afa602cdca717ce2 [id=f32a2592b45047bda447d79e36427e31]
I20251025 07:52:35.405905 1561 consensus_queue.cc:1048] T 4037dee385d04e85bb78c8e77cb00379 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:35.406467 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_fcddb31989ee4236a4981c12779b3f5d [id=f3794fa081364666b944c412415bdba0]
I20251025 07:52:35.406888 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_58bd2fb570eb4cfebf3ce98477e23525 [id=f4668f4d18304a619724e52d76d6c915]
I20251025 07:52:35.407298 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_10e57b42f97544208675708b1e028846 [id=f5c1be203e2b4831a025871cf8a2ef17]
I20251025 07:52:35.407722 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236]
I20251025 07:52:35.408097 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_ac3211ef7b0941d2ad4db2fab62231b2 [id=f721af9327cd4e00a20fc1bfebd9e17b]
I20251025 07:52:35.410732 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_dbface3af7f6413d9060b881447d34b3 [id=f74f5abb40284e08a990f9caaa828636]
W20251025 07:52:35.410995 850 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:45417, user_credentials={real_user=slave}} blocked reactor thread for 94462.1us
I20251025 07:52:35.411304 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]
I20251025 07:52:35.411818 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_ab4b1a5a7abb46fd978c91bb6d74840e [id=fb21a45b86ab446e8c3dae4512a9598f]
I20251025 07:52:35.412683 1167 tablet_bootstrap.cc:492] T 8956ad28b35d45f28c683ac055bba441 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:35.412689 1671 raft_consensus.cc:493] T 71432a6848944a0da06ef9b81f160dc4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:35.412832 1167 ts_tablet_manager.cc:1403] T 8956ad28b35d45f28c683ac055bba441 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.003s
I20251025 07:52:35.412871 1671 raft_consensus.cc:515] T 71432a6848944a0da06ef9b81f160dc4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:35.413144 1670 raft_consensus.cc:493] T 8ff73ffc42434f74a4949f156771d5a4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:35.413300 1670 raft_consensus.cc:515] T 8ff73ffc42434f74a4949f156771d5a4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.413697 1167 raft_consensus.cc:359] T 8956ad28b35d45f28c683ac055bba441 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.413870 1167 raft_consensus.cc:385] T 8956ad28b35d45f28c683ac055bba441 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:35.413918 1670 leader_election.cc:290] T 8ff73ffc42434f74a4949f156771d5a4 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:35.413975 1167 raft_consensus.cc:740] T 8956ad28b35d45f28c683ac055bba441 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:35.414160 1671 leader_election.cc:290] T 71432a6848944a0da06ef9b81f160dc4 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:35.414302 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "71432a6848944a0da06ef9b81f160dc4" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:35.414486 1167 consensus_queue.cc:260] T 8956ad28b35d45f28c683ac055bba441 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.414502 774 raft_consensus.cc:2468] T 71432a6848944a0da06ef9b81f160dc4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:35.414999 982 leader_election.cc:304] T 71432a6848944a0da06ef9b81f160dc4 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:35.415079 1167 ts_tablet_manager.cc:1434] T 8956ad28b35d45f28c683ac055bba441 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:35.415398 1167 tablet_bootstrap.cc:492] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:35.416601 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8ff73ffc42434f74a4949f156771d5a4" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:35.416812 1047 raft_consensus.cc:2468] T 8ff73ffc42434f74a4949f156771d5a4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:35.417210 712 leader_election.cc:304] T 8ff73ffc42434f74a4949f156771d5a4 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:35.417553 1670 raft_consensus.cc:2804] T 8ff73ffc42434f74a4949f156771d5a4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:35.417608 1167 tablet_bootstrap.cc:654] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:35.417659 1670 raft_consensus.cc:493] T 8ff73ffc42434f74a4949f156771d5a4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:35.417713 1671 raft_consensus.cc:2804] T 71432a6848944a0da06ef9b81f160dc4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:35.417793 1670 raft_consensus.cc:3060] T 8ff73ffc42434f74a4949f156771d5a4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.417814 1671 raft_consensus.cc:493] T 71432a6848944a0da06ef9b81f160dc4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:35.417925 1671 raft_consensus.cc:3060] T 71432a6848944a0da06ef9b81f160dc4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.421098 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_e90d7f5ff13b40878e5b625c97792c7d [id=fc8180c09d7b4e74b2e161f7acb8fe0e]
I20251025 07:52:35.421296 1670 raft_consensus.cc:515] T 8ff73ffc42434f74a4949f156771d5a4 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.421599 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_d49fa3011bf146b5a0dcb1c2d4557138 [id=fc927d99526b494db7c23ccf3e94fa02]
I20251025 07:52:35.422204 1671 raft_consensus.cc:515] T 71432a6848944a0da06ef9b81f160dc4 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:35.422751 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8ff73ffc42434f74a4949f156771d5a4" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:35.422930 1047 raft_consensus.cc:3060] T 8ff73ffc42434f74a4949f156771d5a4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.422982 1670 leader_election.cc:290] T 8ff73ffc42434f74a4949f156771d5a4 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:35.423484 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "71432a6848944a0da06ef9b81f160dc4" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:35.423666 774 raft_consensus.cc:3060] T 71432a6848944a0da06ef9b81f160dc4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.425932 1441 consensus_queue.cc:1048] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:35.426111 1047 raft_consensus.cc:2468] T 8ff73ffc42434f74a4949f156771d5a4 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:35.426790 712 leader_election.cc:304] T 8ff73ffc42434f74a4949f156771d5a4 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:35.427047 774 raft_consensus.cc:2468] T 71432a6848944a0da06ef9b81f160dc4 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:35.427237 1441 raft_consensus.cc:2804] T 8ff73ffc42434f74a4949f156771d5a4 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:35.427407 1441 raft_consensus.cc:697] T 8ff73ffc42434f74a4949f156771d5a4 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:35.427666 1441 consensus_queue.cc:237] T 8ff73ffc42434f74a4949f156771d5a4 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.428377 1667 catalog_manager.cc:679] Loaded metadata for table default.Table_3b25f9f08fa04ed5a0d2a758b95e8bbc [id=ff4ac044250d489d9826b92531214409]
I20251025 07:52:35.428545 982 leader_election.cc:304] T 71432a6848944a0da06ef9b81f160dc4 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:35.428865 1612 raft_consensus.cc:2804] T 71432a6848944a0da06ef9b81f160dc4 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:35.429028 1612 raft_consensus.cc:697] T 71432a6848944a0da06ef9b81f160dc4 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:35.429323 1612 consensus_queue.cc:237] T 71432a6848944a0da06ef9b81f160dc4 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:35.432370 1829 raft_consensus.cc:515] T 4f25c9caa05b4da4bd56a8a21fde423c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:35.435194 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f25c9caa05b4da4bd56a8a21fde423c" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:35.435428 1046 raft_consensus.cc:3060] T 4f25c9caa05b4da4bd56a8a21fde423c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.437938 1671 leader_election.cc:290] T 71432a6848944a0da06ef9b81f160dc4 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:35.439503 1046 raft_consensus.cc:2468] T 4f25c9caa05b4da4bd56a8a21fde423c P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:35.440161 712 leader_election.cc:304] T 4f25c9caa05b4da4bd56a8a21fde423c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:35.440788 1670 raft_consensus.cc:2804] T 4f25c9caa05b4da4bd56a8a21fde423c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:35.441321 1670 raft_consensus.cc:697] T 4f25c9caa05b4da4bd56a8a21fde423c P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:35.441663 1670 consensus_queue.cc:237] T 4f25c9caa05b4da4bd56a8a21fde423c P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:35.445046 1829 leader_election.cc:290] T 4f25c9caa05b4da4bd56a8a21fde423c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:35.451974 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0bdaa8cfd764f1b83f47881353d26b6" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:35.452765 1707 consensus_queue.cc:1048] T 4037dee385d04e85bb78c8e77cb00379 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.038s
I20251025 07:52:35.455062 1169 tablet_bootstrap.cc:492] T 87268e27540c4ec58ce1e62df819f19d P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:35.455343 1169 ts_tablet_manager.cc:1403] T 87268e27540c4ec58ce1e62df819f19d P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.096s user 0.006s sys 0.003s
I20251025 07:52:35.456926 1169 raft_consensus.cc:359] T 87268e27540c4ec58ce1e62df819f19d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.457244 1169 raft_consensus.cc:385] T 87268e27540c4ec58ce1e62df819f19d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:35.457394 1169 raft_consensus.cc:740] T 87268e27540c4ec58ce1e62df819f19d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:35.457794 1169 consensus_queue.cc:260] T 87268e27540c4ec58ce1e62df819f19d P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.463802 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f25c9caa05b4da4bd56a8a21fde423c" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:35.464757 1166 raft_consensus.cc:359] T 87268e27540c4ec58ce1e62df819f19d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.464986 1166 raft_consensus.cc:385] T 87268e27540c4ec58ce1e62df819f19d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:35.465945 1839 raft_consensus.cc:493] T f0bdaa8cfd764f1b83f47881353d26b6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:35.467159 1667 tablet_loader.cc:96] loaded metadata for tablet 0063a0776b2247afaa8be82f6eff26f1 (table default.Table_2c8bad53187a49858df0c2bde3485e61 [id=1d22c44df6d24fed9c7009b7c7bd5a70])
I20251025 07:52:35.467615 914 raft_consensus.cc:2468] T 4f25c9caa05b4da4bd56a8a21fde423c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:35.468351 1166 raft_consensus.cc:740] T 87268e27540c4ec58ce1e62df819f19d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:35.468187 1839 raft_consensus.cc:515] T f0bdaa8cfd764f1b83f47881353d26b6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.470465 1671 consensus_queue.cc:1048] T 35ba9d23262248cda6fdf03c51be11a4 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:35.470975 1667 tablet_loader.cc:96] loaded metadata for tablet 01f82980559b4662bdc09786ed429ddc (table default.Table_072e0092adb64ef0ac6834dc18688ccf [id=1d4cbe4a4f38445297ceeebb8ffdaa8e])
I20251025 07:52:35.472373 1667 tablet_loader.cc:96] loaded metadata for tablet 0325d462441349ef81841f1b0cab2a20 (table default.Table_750ecb905839434fa9a1aa72f87394be [id=9ce3ba77c56a4ef09cb3b40ea0e488f6])
I20251025 07:52:35.473578 1667 tablet_loader.cc:96] loaded metadata for tablet 04449cc97f404a19b6a1b20dea074f95 (table default.Table_332b308220f64c6dbb5f29711d8463b3 [id=e041946c093b4c789ce175c736cc230b])
I20251025 07:52:35.474507 1667 tablet_loader.cc:96] loaded metadata for tablet 04d27d54fdb7483b869408af65ff85c9 (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2])
I20251025 07:52:35.475646 1667 tablet_loader.cc:96] loaded metadata for tablet 0540d6a10cf04fa89756623e9aaaeb68 (table default.Table_f16b7a3b7d91423f8502d73e4234ab4a [id=cc7db0a7eaea4caa8c17281a8079c372])
I20251025 07:52:35.476352 1169 ts_tablet_manager.cc:1434] T 87268e27540c4ec58ce1e62df819f19d P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.020s user 0.003s sys 0.000s
I20251025 07:52:35.477429 1667 tablet_loader.cc:96] loaded metadata for tablet 0551e5d694894e8fbeb66b5e0c1cd75b (table default.Table_2cd70245aa3d4809a0647e7fa499f62f [id=a003268ec3da425989fa4041699de4d3])
I20251025 07:52:35.481483 1169 tablet_bootstrap.cc:492] T f3001314b56345f5a70314777cae0125 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:35.482529 1667 tablet_loader.cc:96] loaded metadata for tablet 0582ee42ab564c07ae3facfc5935617c (table default.Table_6aaf0d6ed7554bd4afa602cdca717ce2 [id=f32a2592b45047bda447d79e36427e31])
I20251025 07:52:35.483445 1667 tablet_loader.cc:96] loaded metadata for tablet 062c0f3e51a3487880cea373f8cf1f54 (table default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de])
I20251025 07:52:35.484508 1667 tablet_loader.cc:96] loaded metadata for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e])
I20251025 07:52:35.485574 1667 tablet_loader.cc:96] loaded metadata for tablet 07898860aa594a05a211fc89768cb76d (table default.Table_92cbcd8230f249a8b2821f69a81b41f1 [id=08e53121e86d405dbbd02a4605695fec])
I20251025 07:52:35.486796 1667 tablet_loader.cc:96] loaded metadata for tablet 09898735796541b0a8f3be22cdb6cc30 (table default.Table_bfcdc02fe0dc48a0ae793ecd15f50a70 [id=dec6d4b5bbe34710aed2d981e185f35e])
I20251025 07:52:35.496376 1612 consensus_queue.cc:1048] T fab67b56eaa44b54886c17181d8c52b2 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:35.496820 1671 consensus_queue.cc:1048] T 2784bbfb97d5466f9ec856add7d0b29a P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:35.498843 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0bdaa8cfd764f1b83f47881353d26b6" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:35.499116 1048 raft_consensus.cc:3060] T 2f26667f053a4f16b5da330c0b5e40f9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.507239 910 raft_consensus.cc:2468] T f0bdaa8cfd764f1b83f47881353d26b6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:35.509266 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0bdaa8cfd764f1b83f47881353d26b6" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:35.509568 774 raft_consensus.cc:2393] T f0bdaa8cfd764f1b83f47881353d26b6 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3aea2869ff814cc7b142012faa274382 in current term 1: Already voted for candidate db42be7ad3c741a0aa427f27e245993b in this term.
I20251025 07:52:35.498337 1166 consensus_queue.cc:260] T 87268e27540c4ec58ce1e62df819f19d P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.498303 906 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8ff73ffc42434f74a4949f156771d5a4" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:35.513334 1707 consensus_queue.cc:1048] T 2f26667f053a4f16b5da330c0b5e40f9 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20251025 07:52:35.513731 1667 tablet_loader.cc:96] loaded metadata for tablet 0b134710cf254908876c9763582f82d6 (table default.Table_68399b04a61f453da82d016afd85a32d [id=254e184d61d3414a8b460b4f95c408a9])
I20251025 07:52:35.514223 1167 tablet_bootstrap.cc:492] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:35.507145 904 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "71432a6848944a0da06ef9b81f160dc4" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:35.514432 1167 ts_tablet_manager.cc:1403] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.099s user 0.012s sys 0.013s
I20251025 07:52:35.514533 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0bdaa8cfd764f1b83f47881353d26b6" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:35.514761 908 raft_consensus.cc:3060] T f0bdaa8cfd764f1b83f47881353d26b6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.515326 847 leader_election.cc:304] T f0bdaa8cfd764f1b83f47881353d26b6 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:35.515431 1167 raft_consensus.cc:359] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.515628 1167 raft_consensus.cc:385] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:35.515693 1167 raft_consensus.cc:740] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:35.515771 1667 tablet_loader.cc:96] loaded metadata for tablet 0b47d1b0e6a342888135d7e4e5fa1f43 (table default.Table_c78d2d5f287d45ab9318f5c8a4c1e7ae [id=4d099e6134f548b49dec1897f6660560])
I20251025 07:52:35.515944 1167 consensus_queue.cc:260] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:35.516384 983 leader_election.cc:343] T 71432a6848944a0da06ef9b81f160dc4 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:35.516666 1667 tablet_loader.cc:96] loaded metadata for tablet 1068f566477e4c6db7dde566fff4c2fb (table default.Table_7ce70fea0fe04c52a62033afb8eef977 [id=a8b2a16fd037435893c8f60e92fd1627])
I20251025 07:52:35.516860 1167 ts_tablet_manager.cc:1434] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:35.517192 1167 tablet_bootstrap.cc:492] T 9979cae195e849908677d78c6c33be71 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:35.517592 1667 tablet_loader.cc:96] loaded metadata for tablet 11bfe0c87ca941a3846462c8a4512d83 (table default.Table_00b3b394e5ed46faaac0202afe54b437 [id=304ef5bba5ed4e198622b8c8dc17f43a])
I20251025 07:52:35.519699 1167 tablet_bootstrap.cc:654] T 9979cae195e849908677d78c6c33be71 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:35.520236 1667 tablet_loader.cc:96] loaded metadata for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525])
I20251025 07:52:35.520740 1667 tablet_loader.cc:96] loaded metadata for tablet 143aa3fce9ba4850b000a94054e5dc3a (table default.Table_5261ed9b1f29420eab473df75f903987 [id=771a5d1dc57e4dffbfe4bf757f57719c])
I20251025 07:52:35.521982 1839 leader_election.cc:290] T f0bdaa8cfd764f1b83f47881353d26b6 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:35.522028 1667 tablet_loader.cc:96] loaded metadata for tablet 165575a8b5b54d3c9dbcc1cc93044623 (table default.Table_d55201c0dabc4134b687cfbd805119e3 [id=3f7923f091ac46e0bc20bf7401b5458d])
I20251025 07:52:35.522014 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "71432a6848944a0da06ef9b81f160dc4" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:35.523296 1667 tablet_loader.cc:96] loaded metadata for tablet 19f1b56cdbd24ebe8688cc05fb795d07 (table default.Table_7955f4594e0e4f7c803f728a43c229ca [id=95040fbb66f94779b7721e2209d82a65])
I20251025 07:52:35.525249 1671 raft_consensus.cc:493] T b17d2a9ad6be49d29f2413666fba7e69 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:35.525481 1671 raft_consensus.cc:515] T b17d2a9ad6be49d29f2413666fba7e69 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:35.526229 1671 leader_election.cc:290] T b17d2a9ad6be49d29f2413666fba7e69 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:35.527400 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b17d2a9ad6be49d29f2413666fba7e69" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:35.527006 1667 tablet_loader.cc:96] loaded metadata for tablet 1aeba18be0cb4eb9949aeaf6ad73a812 (table default.Table_bd168da90c704b7987fc731f5ab28045 [id=e7683b98787e4d55b81303f4b58e4a6b])
I20251025 07:52:35.527747 773 raft_consensus.cc:2393] T b17d2a9ad6be49d29f2413666fba7e69 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 23eeffd3de254ce18db5febfde971c6b in this term.
I20251025 07:52:35.528213 1667 tablet_loader.cc:96] loaded metadata for tablet 1b22bd172e0e4f86896e9ecdef666b42 (table default.Table_9bde97409538452a99917ba1e4ed04b3 [id=eb909ab8da1d45e28008f447066bea71])
I20251025 07:52:35.533421 1667 tablet_loader.cc:96] loaded metadata for tablet 1c40b2e06d894a06af9e58285892ea8d (table default.Table_3cd469e1b1414bc7ad4a122483cb7755 [id=7b4db2984274499c98f6189d753f7856])
I20251025 07:52:35.533833 1667 tablet_loader.cc:96] loaded metadata for tablet 1d121fe4cde04b969a5620f1d58288aa (table default.Table_56a34b2e5e744f94a5a955e4a796498a [id=e35831cc8e8d47139dd92b95e78c6aee])
I20251025 07:52:35.535975 1667 tablet_loader.cc:96] loaded metadata for tablet 22c97b3ee92140c68a7e1b93033cb299 (table default.Table_c45e312143074d47be4fac483ee16136 [id=a003a6f6574b4cde88d30928eaa7f4ec])
I20251025 07:52:35.536545 1667 tablet_loader.cc:96] loaded metadata for tablet 22f7db660616457b83b24e947a0b0bae (table default.Table_3d163f45f94f450ebf0447897bdf41a6 [id=bb316a7e000748ca902a9bfb494802d7])
W20251025 07:52:35.537050 983 leader_election.cc:343] T 71432a6848944a0da06ef9b81f160dc4 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:35.542560 1667 tablet_loader.cc:96] loaded metadata for tablet 22fc90c704f64d2f9f4b8c54dee45576 (table default.Table_e317f8d728da4c76886f67d0995ab7f4 [id=4d2121dea4904850aecb65da5dd2d8f9])
I20251025 07:52:35.543315 1667 tablet_loader.cc:96] loaded metadata for tablet 243b8828fc344bacab46d7bab6bd2dec (table default.Table_d647b0fa975743998d7f49c36abe9c96 [id=5b435831d8254ae8b126218a39e3a362])
I20251025 07:52:35.544486 1667 tablet_loader.cc:96] loaded metadata for tablet 288e543bb279410c91c01dd1623e110f (table default.Table_ea269532aff34ca1803fc1c5d5ad3641 [id=4c34b64b299f4162a53866cf01c5298f])
I20251025 07:52:35.545369 1167 tablet_bootstrap.cc:492] T 9979cae195e849908677d78c6c33be71 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:35.554304 1167 ts_tablet_manager.cc:1403] T 9979cae195e849908677d78c6c33be71 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.003s
I20251025 07:52:35.555302 1167 raft_consensus.cc:359] T 9979cae195e849908677d78c6c33be71 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.555585 1167 raft_consensus.cc:385] T 9979cae195e849908677d78c6c33be71 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:35.555722 1167 raft_consensus.cc:740] T 9979cae195e849908677d78c6c33be71 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:35.556314 1667 tablet_loader.cc:96] loaded metadata for tablet 292fe04055954afd8c08ac2636b68191 (table default.Table_9fa69b10b5e842639fc0b0a58cd9b6a5 [id=a253ce0253bd4da4814741c482193d8f])
I20251025 07:52:35.556635 1167 consensus_queue.cc:260] T 9979cae195e849908677d78c6c33be71 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.557416 1167 ts_tablet_manager.cc:1434] T 9979cae195e849908677d78c6c33be71 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251025 07:52:35.558211 1667 tablet_loader.cc:96] loaded metadata for tablet 294d7d20c2f545b8813e632aa058a60c (table default.Table_4eaf06c0720744ff8b2fc8866cf05e2b [id=2b66e5f736de408bbd711b88361e1c7b])
I20251025 07:52:35.558946 1667 tablet_loader.cc:96] loaded metadata for tablet 29dbaf9149134f548a974a5779f7da67 (table default.Table_c43ac1adc76d4ccf819411ec76775932 [id=106e3728b0b846178eb48ccbf076e24e])
I20251025 07:52:35.559335 1167 tablet_bootstrap.cc:492] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:35.561136 1167 tablet_bootstrap.cc:654] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:35.562096 1667 tablet_loader.cc:96] loaded metadata for tablet 2c4789b9d89f42cb844e6cb595e687db (table default.Table_ddc8580fd1674f9ca61d528f22c51c94 [id=e33f00d4d5844ebba7ea3e200e154eeb])
I20251025 07:52:35.562628 1667 tablet_loader.cc:96] loaded metadata for tablet 2c50ec95e48e49b59a76fac4fc862d3d (table default.Table_32b54e82cbc84d39a0020826f8737d62 [id=c28b0b6acd61490caecfdaab803a9d92])
I20251025 07:52:35.564062 1667 tablet_loader.cc:96] loaded metadata for tablet 2d24e3e7a9e4431db609331ad1fd57c9 (table default.Table_2ddc7ef45af34002a5db135b33ec855e [id=7e34aeb0c3ea4fc5ba6a9077a79335d1])
I20251025 07:52:35.564797 1667 tablet_loader.cc:96] loaded metadata for tablet 2e2e6893a48449969b71ee1c08d1960e (table default.Table_19354444047b451bbf22bafcf9ae185d [id=b3725485d48c41a1952091d028a783c0])
I20251025 07:52:35.565320 1667 tablet_loader.cc:96] loaded metadata for tablet 2e87a81c87ec4353a8ad920307809390 (table default.Table_b8551dc7ed8c48228c5d1fec6a1e8b0c [id=52d442791ab94896b1e377af37ab6154])
I20251025 07:52:35.566309 1667 tablet_loader.cc:96] loaded metadata for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7])
I20251025 07:52:35.566773 1667 tablet_loader.cc:96] loaded metadata for tablet 32a143a715cb485cb4501f4e28c68252 (table default.Table_e386bc02f89041418ff3ddf8ac224eea [id=e800c96844e843b7a6fd3145907e5503])
I20251025 07:52:35.567726 1667 tablet_loader.cc:96] loaded metadata for tablet 34c63b69904d499bb0d286941a8303f2 (table default.Table_3709b01b0a9a48499f5bd229e3e9f31a [id=433d4cda7efb41349209f3e82cc41e3a])
I20251025 07:52:35.572263 1667 tablet_loader.cc:96] loaded metadata for tablet 36262ab8c6964dd684c01b4db6219a4f (table default.Table_76158f8cf82e4cdb9a90580e46aaea48 [id=f249f30497bc450c887d659721469de8])
I20251025 07:52:35.573405 1667 tablet_loader.cc:96] loaded metadata for tablet 36c414bc700341188b7d85c14173c3e0 (table default.Table_4e81dfac4c7742c7986eb24e6d54a124 [id=b87e43a3dc3c4cd98ee3f44964e8092f])
I20251025 07:52:35.572860 1825 raft_consensus.cc:493] T 8ff73ffc42434f74a4949f156771d5a4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:35.575699 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b17d2a9ad6be49d29f2413666fba7e69" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:35.576941 1166 ts_tablet_manager.cc:1434] T 87268e27540c4ec58ce1e62df819f19d P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.195s user 0.000s sys 0.002s
I20251025 07:52:35.522181 907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8ff73ffc42434f74a4949f156771d5a4" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:35.580425 1166 tablet_bootstrap.cc:492] T db99a1e68daf4e948a3d66ad276cc5f9 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:35.582376 1441 raft_consensus.cc:493] T 9979cae195e849908677d78c6c33be71 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:35.582540 1441 raft_consensus.cc:515] T 9979cae195e849908677d78c6c33be71 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.583231 1441 leader_election.cc:290] T 9979cae195e849908677d78c6c33be71 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:35.585287 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9979cae195e849908677d78c6c33be71" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:35.585981 712 leader_election.cc:343] T 9979cae195e849908677d78c6c33be71 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:35.574486 1667 tablet_loader.cc:96] loaded metadata for tablet 36c8fcafd5464b4e999276a8062a812a (table default.Table_19a3326e4da345c1bbeb2cf083d28e24 [id=11c088570f6940b1895d1691465c10b7])
I20251025 07:52:35.591264 1167 tablet_bootstrap.cc:492] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:35.579355 1825 raft_consensus.cc:515] T 8ff73ffc42434f74a4949f156771d5a4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.592067 906 raft_consensus.cc:3060] T 8ff73ffc42434f74a4949f156771d5a4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.592272 1667 tablet_loader.cc:96] loaded metadata for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3])
I20251025 07:52:35.592898 1667 tablet_loader.cc:96] loaded metadata for tablet 3aeaa0e9dc594e21a9ddd923e314017d (table default.Table_51f2b43545db41029568a0321a37e139 [id=5a499fdacb8047dc9ce9bed92d7dd233])
I20251025 07:52:35.593719 1167 ts_tablet_manager.cc:1403] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.034s user 0.001s sys 0.007s
I20251025 07:52:35.591960 1825 leader_election.cc:290] T 8ff73ffc42434f74a4949f156771d5a4 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:35.595356 1167 raft_consensus.cc:359] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.595824 903 raft_consensus.cc:2393] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:35.598052 1667 tablet_loader.cc:96] loaded metadata for tablet 3c4970c8a491460e8b4b1c29fde83e4c (table default.Table_915572e49894471d8933816077123d6c [id=e372ca4384644779a6b37f389f549dec])
I20251025 07:52:35.597854 1441 consensus_queue.cc:1048] T 7f29b64704e147aa9b0b5f6cda860f09 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:35.599195 1167 raft_consensus.cc:385] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:35.599509 1167 raft_consensus.cc:740] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:35.599795 1169 tablet_bootstrap.cc:654] T f3001314b56345f5a70314777cae0125 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:35.600169 1167 consensus_queue.cc:260] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.604861 1167 ts_tablet_manager.cc:1434] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20251025 07:52:35.605227 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8ff73ffc42434f74a4949f156771d5a4" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:35.605564 1048 raft_consensus.cc:2393] T 8ff73ffc42434f74a4949f156771d5a4 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3aea2869ff814cc7b142012faa274382 in current term 1: Already voted for candidate 23eeffd3de254ce18db5febfde971c6b in this term.
I20251025 07:52:35.605650 1167 tablet_bootstrap.cc:492] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:35.607434 1167 tablet_bootstrap.cc:654] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:35.608901 1667 tablet_loader.cc:96] loaded metadata for tablet 3d5e0dd14e474fe09993a03851f8b1f0 (table default.Table_cd9190d6cb714bdbb763dd934b33c10a [id=1b5db2f5053c4cb3a3b40eb109d5d5d9])
I20251025 07:52:35.609948 1667 tablet_loader.cc:96] loaded metadata for tablet 3e1ffd02aa384b73835b4361ae3d5467 (table default.Table_cf3a2c0772674c2a8ff5fee551a93a0f [id=1ffca9f550ce498296d664517e9fba86])
I20251025 07:52:35.611294 1667 tablet_loader.cc:96] loaded metadata for tablet 3e79d69d5d004cc395b8e35569e25440 (table default.Table_23d4d4a3c90c47ea8230c39a51138418 [id=a7070a0269fc4fc99447ab671eb226be])
I20251025 07:52:35.616062 1667 tablet_loader.cc:96] loaded metadata for tablet 421db850a64048218eecb9de65e2bdbf (table default.Table_157e52b1e07f4356afc68bb3ec5e4eb0 [id=a0facec9e7314c59b2175f0196869d73])
I20251025 07:52:35.617106 1667 tablet_loader.cc:96] loaded metadata for tablet 435fda144479482fbd4f9f9808d97899 (table default.Table_2e41b219b43c4746bd24f0cb1e41547f [id=b087a7c6648d46a3b7d410cf8cc81e2c])
I20251025 07:52:35.618119 1667 tablet_loader.cc:96] loaded metadata for tablet 437da5ded9b648abb668a8cb4896452c (table default.Table_0ebd49dea97c4b0790902d42d65e749f [id=68d232ecc08c4f8db462fc0ad620e9be])
I20251025 07:52:35.619808 1667 tablet_loader.cc:96] loaded metadata for tablet 4678f7eab5e8497db308e8a38e387618 (table default.Table_bc1398c7c53846da95b511e6a442e416 [id=10e2dcf046e241a9b881adabc51defbc])
I20251025 07:52:35.621809 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8ff73ffc42434f74a4949f156771d5a4" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:35.622056 1860 consensus_queue.cc:1048] T 6897405bbc1c47a4ab0aa3af80e60e3d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20251025 07:52:35.622632 1561 raft_consensus.cc:493] T 87268e27540c4ec58ce1e62df819f19d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:35.622813 1561 raft_consensus.cc:515] T 87268e27540c4ec58ce1e62df819f19d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.623881 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87268e27540c4ec58ce1e62df819f19d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:35.624081 1048 raft_consensus.cc:2468] T 87268e27540c4ec58ce1e62df819f19d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:35.624531 850 leader_election.cc:304] T 87268e27540c4ec58ce1e62df819f19d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:35.624830 1860 raft_consensus.cc:2804] T 87268e27540c4ec58ce1e62df819f19d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:35.624923 1860 raft_consensus.cc:493] T 87268e27540c4ec58ce1e62df819f19d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:35.624992 1860 raft_consensus.cc:3060] T 87268e27540c4ec58ce1e62df819f19d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.625921 1667 tablet_loader.cc:96] loaded metadata for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69])
I20251025 07:52:35.626534 1667 tablet_loader.cc:96] loaded metadata for tablet 4a1ec8c2ed7f41abb588f592f62097d1 (table default.Table_91898f2a551144d18df0309c827855bf [id=aaa734f6549741768b0156bdf5cc25af])
I20251025 07:52:35.626914 1667 tablet_loader.cc:96] loaded metadata for tablet 4b278ce7768349e49dcd1131e404ab34 (table default.Table_309913c03a4d46019d35478363f149e8 [id=bad99c3045fc4b938e20d2dcfdd0b1bc])
I20251025 07:52:35.627254 1667 tablet_loader.cc:96] loaded metadata for tablet 4b28215f61574cf48c447dd01955ac77 (table default.Table_a437a2623d704e3ea3c5b8c65c88b936 [id=51b12903e6024e1a9adc2f9fa7400937])
I20251025 07:52:35.627605 1667 tablet_loader.cc:96] loaded metadata for tablet 4c155aa7fa5b4d68bbab1918b8432372 (table default.Table_6c5a24747ba14352b0285a41ec2985a4 [id=28bbaad474164a8c8c97a76bb8081791])
I20251025 07:52:35.628125 1860 raft_consensus.cc:515] T 87268e27540c4ec58ce1e62df819f19d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.629112 1667 tablet_loader.cc:96] loaded metadata for tablet 509c94f2a422493ea4390ecee576db72 (table default.Table_3f081b7bb4e1420c8121ea4115439f3d [id=696a1285fbfe4038a8b1b38708c8ea49])
I20251025 07:52:35.629310 1860 leader_election.cc:290] T 87268e27540c4ec58ce1e62df819f19d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:35.629694 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87268e27540c4ec58ce1e62df819f19d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:35.629869 1167 tablet_bootstrap.cc:492] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:35.629878 1048 raft_consensus.cc:3060] T 87268e27540c4ec58ce1e62df819f19d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.630020 1167 ts_tablet_manager.cc:1403] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.002s
I20251025 07:52:35.630872 1167 raft_consensus.cc:359] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.631044 1167 raft_consensus.cc:385] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:35.631124 1167 raft_consensus.cc:740] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:35.633028 1048 raft_consensus.cc:2468] T 87268e27540c4ec58ce1e62df819f19d P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:35.633651 850 leader_election.cc:304] T 87268e27540c4ec58ce1e62df819f19d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:35.634073 1860 raft_consensus.cc:2804] T 87268e27540c4ec58ce1e62df819f19d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:35.634891 1860 raft_consensus.cc:697] T 87268e27540c4ec58ce1e62df819f19d P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:35.635250 1860 consensus_queue.cc:237] T 87268e27540c4ec58ce1e62df819f19d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.639362 1167 consensus_queue.cc:260] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.643177 1167 ts_tablet_manager.cc:1434] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20251025 07:52:35.643517 1167 tablet_bootstrap.cc:492] T 896bd7870d87408e8adfdb7a78998111 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:35.645349 1167 tablet_bootstrap.cc:654] T 896bd7870d87408e8adfdb7a78998111 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:35.648619 1825 consensus_queue.cc:1048] T 887e7c7a5d0e4aaf88f26246306df19c P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.044s
I20251025 07:52:35.575436 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f25c9caa05b4da4bd56a8a21fde423c" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:35.650075 902 raft_consensus.cc:3060] T 4f25c9caa05b4da4bd56a8a21fde423c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.650188 1667 tablet_loader.cc:96] loaded metadata for tablet 5361f08f50b745eb98bf02e979d7c71e (table default.Table_a3f50508f4b047b6837c67248417b21c [id=572bc5595be947468e24f2a9144f12a4])
I20251025 07:52:35.651598 1667 tablet_loader.cc:96] loaded metadata for tablet 570201e7bf05406a97c81dde071d894b (table default.Table_646f48a21c194af3a1611e5e12b2e11b [id=6f395892b09240fea02482b3199cfb5e])
I20251025 07:52:35.652248 1667 tablet_loader.cc:96] loaded metadata for tablet 59546dbfac6c413fb33e18ab6b414957 (table default.Table_07b5304bacea445c92e5dde5059008aa [id=9d6d15d545584125aaaf2d51aed95f2c])
I20251025 07:52:35.653192 902 raft_consensus.cc:2468] T 4f25c9caa05b4da4bd56a8a21fde423c P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:35.653347 1667 tablet_loader.cc:96] loaded metadata for tablet 5dab439bc50d4e869548154e1d222ad6 (table default.Table_ab6798879a554075b3deaa059ad077c7 [id=45325c188eac4d34bb016d01de091f87])
I20251025 07:52:35.659489 983 leader_election.cc:304] T b17d2a9ad6be49d29f2413666fba7e69 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:35.659878 1671 raft_consensus.cc:3060] T b17d2a9ad6be49d29f2413666fba7e69 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.660041 1675 raft_consensus.cc:493] T 8956ad28b35d45f28c683ac055bba441 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:35.660189 1675 raft_consensus.cc:515] T 8956ad28b35d45f28c683ac055bba441 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.660880 1675 leader_election.cc:290] T 8956ad28b35d45f28c683ac055bba441 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:35.661521 908 raft_consensus.cc:2468] T f0bdaa8cfd764f1b83f47881353d26b6 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:35.661689 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8956ad28b35d45f28c683ac055bba441" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:35.661907 776 raft_consensus.cc:2468] T 8956ad28b35d45f28c683ac055bba441 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:35.662369 982 leader_election.cc:304] T 8956ad28b35d45f28c683ac055bba441 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:35.662647 1612 raft_consensus.cc:493] T bef0ef23d90d480d9c99aef21d7271fd P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:35.662796 1612 raft_consensus.cc:515] T bef0ef23d90d480d9c99aef21d7271fd P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.663038 1671 raft_consensus.cc:2749] T b17d2a9ad6be49d29f2413666fba7e69 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:35.663311 1671 raft_consensus.cc:2804] T 8956ad28b35d45f28c683ac055bba441 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:35.663401 1671 raft_consensus.cc:493] T 8956ad28b35d45f28c683ac055bba441 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:35.663448 1612 leader_election.cc:290] T bef0ef23d90d480d9c99aef21d7271fd P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:35.664045 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bef0ef23d90d480d9c99aef21d7271fd" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:35.664148 1671 raft_consensus.cc:3060] T 8956ad28b35d45f28c683ac055bba441 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.664398 906 raft_consensus.cc:2468] T 8ff73ffc42434f74a4949f156771d5a4 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:35.665244 1441 raft_consensus.cc:493] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:35.665393 1441 raft_consensus.cc:515] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.666079 1441 leader_election.cc:290] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:35.666875 1667 tablet_loader.cc:96] loaded metadata for tablet 5fc092c2bb2d41ac8c5b4fc07460c8ec (table default.Table_974f1ee455d7488593e5c496cc012099 [id=630c274339664b9db18ab4ef7c29c16f])
I20251025 07:52:35.667500 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db99a1e68daf4e948a3d66ad276cc5f9" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:35.668084 712 leader_election.cc:343] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:35.668743 1667 tablet_loader.cc:96] loaded metadata for tablet 60c60a9ee37f43978910bd067a098bb7 (table default.Table_173708dfd0274b34a7db6fa2b239a8b0 [id=6bc5ca21f8c54249a5150f0fea2686c7])
I20251025 07:52:35.670027 1671 raft_consensus.cc:515] T 8956ad28b35d45f28c683ac055bba441 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.671087 1671 leader_election.cc:290] T 8956ad28b35d45f28c683ac055bba441 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:35.674527 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8956ad28b35d45f28c683ac055bba441" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:35.674652 1441 consensus_queue.cc:1048] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:35.674767 776 raft_consensus.cc:3060] T 8956ad28b35d45f28c683ac055bba441 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.674873 1667 tablet_loader.cc:96] loaded metadata for tablet 627a9cfd398346f2aa34c8932a11b5a8 (table default.Table_2bcb67fcbc85452faaadfa683987951c [id=63e72692d0b6442396189a2f37170815])
I20251025 07:52:35.675268 1667 tablet_loader.cc:96] loaded metadata for tablet 645263a61b654f04985c40d04506c087 (table default.Table_3b25f9f08fa04ed5a0d2a758b95e8bbc [id=ff4ac044250d489d9826b92531214409])
I20251025 07:52:35.678104 776 raft_consensus.cc:2468] T 8956ad28b35d45f28c683ac055bba441 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:35.678622 982 leader_election.cc:304] T 8956ad28b35d45f28c683ac055bba441 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:35.679186 1667 tablet_loader.cc:96] loaded metadata for tablet 66134661be834f7b811d7927ab629812 (table default.Table_590ce35533c7442390ce8991bcfa80fb [id=7fd21f23b9ca483b8044392285400801])
I20251025 07:52:35.679203 1675 raft_consensus.cc:2804] T 8956ad28b35d45f28c683ac055bba441 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:35.679385 1675 raft_consensus.cc:697] T 8956ad28b35d45f28c683ac055bba441 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:35.680272 1667 tablet_loader.cc:96] loaded metadata for tablet 66f0d398e59a4175aa9fe90715124d1a (table default.Table_eafaca066a2548a686050fed8c380553 [id=17525d0c9b664dcdadbf42e85da234ca])
I20251025 07:52:35.680752 1675 consensus_queue.cc:237] T 8956ad28b35d45f28c683ac055bba441 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.682178 1166 tablet_bootstrap.cc:654] T db99a1e68daf4e948a3d66ad276cc5f9 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:35.687266 1667 tablet_loader.cc:96] loaded metadata for tablet 67cf96a7dd3f42e19b86746c87e689f6 (table default.Table_ea39b0e2e2204f38b1c9b1d3346a1e36 [id=9170ed658b754ac8ad50f060c442f27c])
I20251025 07:52:35.688220 1167 tablet_bootstrap.cc:492] T 896bd7870d87408e8adfdb7a78998111 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:35.691969 847 leader_election.cc:304] T 8ff73ffc42434f74a4949f156771d5a4 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:35.692353 1825 raft_consensus.cc:2749] T 8ff73ffc42434f74a4949f156771d5a4 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:35.692461 1667 tablet_loader.cc:96] loaded metadata for tablet 682c1e22d7764dce862d970b33b88cb3 (table default.Table_550c910e06364276ae5c4548afa39e6c [id=03294f958266430589e89052d6c729df])
I20251025 07:52:35.689292 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87268e27540c4ec58ce1e62df819f19d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:35.693133 773 raft_consensus.cc:2468] T 87268e27540c4ec58ce1e62df819f19d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:35.693271 1167 ts_tablet_manager.cc:1403] T 896bd7870d87408e8adfdb7a78998111 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.050s user 0.011s sys 0.000s
I20251025 07:52:35.689664 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87268e27540c4ec58ce1e62df819f19d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:35.693468 774 raft_consensus.cc:3060] T 87268e27540c4ec58ce1e62df819f19d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.694020 1167 raft_consensus.cc:359] T 896bd7870d87408e8adfdb7a78998111 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.694255 1167 raft_consensus.cc:385] T 896bd7870d87408e8adfdb7a78998111 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:35.694336 1167 raft_consensus.cc:740] T 896bd7870d87408e8adfdb7a78998111 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:35.694587 1167 consensus_queue.cc:260] T 896bd7870d87408e8adfdb7a78998111 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.695354 1667 tablet_loader.cc:96] loaded metadata for tablet 6897405bbc1c47a4ab0aa3af80e60e3d (table default.Table_92c0120dd29348618040150dedb14c43 [id=ea892976886644d8aa025ae2edef718d])
I20251025 07:52:35.695768 1667 tablet_loader.cc:96] loaded metadata for tablet 698e9f8de7db4516a811c9eb9143a268 (table default.Table_48712d8a17e04509888897a9cf2b4e30 [id=488b550bbdef476c95e8bad42bd4b4dd])
I20251025 07:52:35.696118 1667 tablet_loader.cc:96] loaded metadata for tablet 6a33cfb2f2de48d4a5e9e39730d4900e (table default.Table_4d3dff3c6fda405f84600e93d3818380 [id=86fcd91f390d4661bcbb6d43190e6f2f])
I20251025 07:52:35.696508 1667 tablet_loader.cc:96] loaded metadata for tablet 6a58931e7cd7454fb583f1c70cd2ad15 (table default.Table_5237c35dbfa7429e803d5008a8b3487c [id=0d597ecab6f54d1ba6cb5bd3436a9db8])
I20251025 07:52:35.696568 774 raft_consensus.cc:2468] T 87268e27540c4ec58ce1e62df819f19d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:35.696856 1667 tablet_loader.cc:96] loaded metadata for tablet 6ce62847e80f48e8807fdbbbdc50803c (table default.Table_dad8e9c26a9a41e2bcf3fa1db9f5b087 [id=579250c3c8fb4e9783e388a53ba35a6c])
I20251025 07:52:35.697218 1667 tablet_loader.cc:96] loaded metadata for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5])
I20251025 07:52:35.700067 1167 ts_tablet_manager.cc:1434] T 896bd7870d87408e8adfdb7a78998111 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20251025 07:52:35.700409 1167 tablet_bootstrap.cc:492] T b97e39aaac3048ed93dbfe873b757254 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:35.702538 1048 raft_consensus.cc:3060] T 887e7c7a5d0e4aaf88f26246306df19c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.698125 1667 tablet_loader.cc:96] loaded metadata for tablet 6fb1a1d38fb94f07b79ac4f8334fa719 (table default.Table_d49fa3011bf146b5a0dcb1c2d4557138 [id=fc927d99526b494db7c23ccf3e94fa02])
I20251025 07:52:35.707795 1839 consensus_queue.cc:1048] T 6897405bbc1c47a4ab0aa3af80e60e3d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:35.708182 1670 consensus_queue.cc:1048] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:35.713469 1675 consensus_queue.cc:1048] T 2784bbfb97d5466f9ec856add7d0b29a P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:35.713608 1837 raft_consensus.cc:2749] T f0bdaa8cfd764f1b83f47881353d26b6 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:35.717345 1837 consensus_queue.cc:1048] T 887e7c7a5d0e4aaf88f26246306df19c P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:35.723634 1837 consensus_queue.cc:1048] T b772025852794a49b072f37cccbd3291 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:35.726855 1167 tablet_bootstrap.cc:654] T b97e39aaac3048ed93dbfe873b757254 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:35.727458 1441 raft_consensus.cc:493] T 896bd7870d87408e8adfdb7a78998111 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:35.727679 1441 raft_consensus.cc:515] T 896bd7870d87408e8adfdb7a78998111 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.728542 1441 leader_election.cc:290] T 896bd7870d87408e8adfdb7a78998111 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:35.730232 1667 tablet_loader.cc:96] loaded metadata for tablet 724d0b7e1b65445a9a1f056852983bdb (table default.Table_6fd078e1af7a41f8a24caebf5e12e2b2 [id=14ce70a07f554566bf85c2e37c80fa65])
I20251025 07:52:35.730639 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "896bd7870d87408e8adfdb7a78998111" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:35.731567 712 leader_election.cc:343] T 896bd7870d87408e8adfdb7a78998111 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:35.733793 1667 tablet_loader.cc:96] loaded metadata for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f])
I20251025 07:52:35.743525 1167 tablet_bootstrap.cc:492] T b97e39aaac3048ed93dbfe873b757254 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:35.745054 1561 leader_election.cc:290] T 87268e27540c4ec58ce1e62df819f19d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:35.746157 1667 tablet_loader.cc:96] loaded metadata for tablet 792f32cbd66449ca98813c669b2a261d (table default.Table_23cacc46bec647adbc62b457f73487cf [id=a8b081495f564c10aa6a05e5eed80488])
I20251025 07:52:35.746439 1167 ts_tablet_manager.cc:1403] T b97e39aaac3048ed93dbfe873b757254 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.046s user 0.000s sys 0.005s
I20251025 07:52:35.747359 1167 raft_consensus.cc:359] T b97e39aaac3048ed93dbfe873b757254 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.749112 1667 tablet_loader.cc:96] loaded metadata for tablet 7e18611cbc1348eca927162fec70e36f (table default.Table_d67fb372b12e40d082d119e53b9ebc3e [id=a4f956a8e567440cbb5ad5214a0aa10a])
I20251025 07:52:35.753973 1167 raft_consensus.cc:385] T b97e39aaac3048ed93dbfe873b757254 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:35.754447 1167 raft_consensus.cc:740] T b97e39aaac3048ed93dbfe873b757254 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:35.755025 1167 consensus_queue.cc:260] T b97e39aaac3048ed93dbfe873b757254 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.756132 1167 ts_tablet_manager.cc:1434] T b97e39aaac3048ed93dbfe873b757254 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20251025 07:52:35.756902 1167 tablet_bootstrap.cc:492] T abf94c32ff2945119b710905c075d63c P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:35.759582 1667 tablet_loader.cc:96] loaded metadata for tablet 7e869c65771f4339a546dfeebb227c62 (table default.Table_8f502389fe564ef3a4e7e23819a8a6e8 [id=56f4e62bc2354befa7d7374e07b103de])
I20251025 07:52:35.760941 1167 tablet_bootstrap.cc:654] T abf94c32ff2945119b710905c075d63c P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:35.762053 1667 tablet_loader.cc:96] loaded metadata for tablet 80124cae4fb64858a1e39f99c5dd0419 (table default.Table_68c19f91d7094d5ab0ab4f77f74a937e [id=867342b181604482af90da2976ef21a0])
I20251025 07:52:35.764501 1667 tablet_loader.cc:96] loaded metadata for tablet 830fdbc6812e416690d0db42a73af503 (table default.Table_ab4b1a5a7abb46fd978c91bb6d74840e [id=fb21a45b86ab446e8c3dae4512a9598f])
I20251025 07:52:35.766801 1667 tablet_loader.cc:96] loaded metadata for tablet 8707b93d491e4e4386d148087a2f0870 (table default.Table_fd5720d69b084df1b36485b7890ac6c2 [id=59840a567e5244e8a9024cbad42471e4])
I20251025 07:52:35.767272 1667 tablet_loader.cc:96] loaded metadata for tablet 87268e27540c4ec58ce1e62df819f19d (table default.Table_350383f0e98947eba77186e74ad88781 [id=96b0e47c666c4d8f982741aea7b17ae7])
I20251025 07:52:35.767398 1837 consensus_queue.cc:1048] T 2f26667f053a4f16b5da330c0b5e40f9 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:35.769935 1667 tablet_loader.cc:96] loaded metadata for tablet 872cb7512d104827b1f1f186d33a4a44 (table default.Table_d5c65dce8b1b4214ba519d964012bfc0 [id=d4be274b36a845a99a4b96b8846248fd])
I20251025 07:52:35.770951 1667 tablet_loader.cc:96] loaded metadata for tablet 88bfc23acd274041a9457079c1709d82 (table default.Table_45f85cb4514d4ec3b06cff95dd4a70f6 [id=b7073684e63d4875841df3dd5171bcc2])
I20251025 07:52:35.772902 1667 tablet_loader.cc:96] loaded metadata for tablet 8917a289bd9b4428a4603429edd94581 (table default.Table_45faec266a9b4b1b9f5ef3ac915e3bbc [id=bc377ac4bb164a80b9ebce289a790326])
I20251025 07:52:35.774091 1169 tablet_bootstrap.cc:492] T f3001314b56345f5a70314777cae0125 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:35.774325 1169 ts_tablet_manager.cc:1403] T f3001314b56345f5a70314777cae0125 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.294s user 0.004s sys 0.002s
I20251025 07:52:35.775885 1667 tablet_loader.cc:96] loaded metadata for tablet 8c6ff4c73e624af3bf2de36c4493b462 (table default.Table_9f6b2f95721f488c86fde259a7cf95ec [id=277005e0d596465293065585a7555f4b])
I20251025 07:52:35.781258 1167 tablet_bootstrap.cc:492] T abf94c32ff2945119b710905c075d63c P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:35.781510 1167 ts_tablet_manager.cc:1403] T abf94c32ff2945119b710905c075d63c P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.002s
I20251025 07:52:35.784744 1167 raft_consensus.cc:359] T abf94c32ff2945119b710905c075d63c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.785118 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8956ad28b35d45f28c683ac055bba441" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:35.785378 913 raft_consensus.cc:3060] T 8956ad28b35d45f28c683ac055bba441 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.785449 901 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "896bd7870d87408e8adfdb7a78998111" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:35.785712 906 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8956ad28b35d45f28c683ac055bba441" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:35.786196 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bef0ef23d90d480d9c99aef21d7271fd" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:35.786446 908 raft_consensus.cc:2393] T bef0ef23d90d480d9c99aef21d7271fd P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 23eeffd3de254ce18db5febfde971c6b in this term.
I20251025 07:52:35.787142 1167 raft_consensus.cc:385] T abf94c32ff2945119b710905c075d63c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:35.787261 1167 raft_consensus.cc:740] T abf94c32ff2945119b710905c075d63c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:35.787577 1167 consensus_queue.cc:260] T abf94c32ff2945119b710905c075d63c P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.788084 1167 ts_tablet_manager.cc:1434] T abf94c32ff2945119b710905c075d63c P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20251025 07:52:35.788427 1167 tablet_bootstrap.cc:492] T d5e5e5bbcf9f43118eea502b69880af5 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:35.788439 913 raft_consensus.cc:2468] T 8956ad28b35d45f28c683ac055bba441 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
W20251025 07:52:35.788784 710 leader_election.cc:343] T 896bd7870d87408e8adfdb7a78998111 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:35.788938 710 leader_election.cc:304] T 896bd7870d87408e8adfdb7a78998111 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:35.785112 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9979cae195e849908677d78c6c33be71" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:35.785176 907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db99a1e68daf4e948a3d66ad276cc5f9" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:35.789752 1829 raft_consensus.cc:2749] T 896bd7870d87408e8adfdb7a78998111 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:35.789947 1829 consensus_queue.cc:1048] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:35.791074 1667 tablet_loader.cc:96] loaded metadata for tablet 8e6d76b1a97945cfb6c3055e1b622f8a (table default.Table_10e57b42f97544208675708b1e028846 [id=f5c1be203e2b4831a025871cf8a2ef17])
W20251025 07:52:35.791704 710 leader_election.cc:343] T 9979cae195e849908677d78c6c33be71 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:35.791718 983 leader_election.cc:304] T bef0ef23d90d480d9c99aef21d7271fd P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:35.791829 710 leader_election.cc:304] T 9979cae195e849908677d78c6c33be71 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:35.791857 1667 tablet_loader.cc:96] loaded metadata for tablet 8eaf6c97ab9b41e9ae746a9c5abe1735 (table default.Table_bf5cb8a733934f4ca29ecc9d473e4f2c [id=4bb4741007834a24aca34e608365ea03])
I20251025 07:52:35.792001 1671 raft_consensus.cc:3060] T bef0ef23d90d480d9c99aef21d7271fd P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:35.792200 710 leader_election.cc:343] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:35.792222 1667 tablet_loader.cc:96] loaded metadata for tablet 8f2c8582fed34c848d9903b8ad8b4057 (table default.Table_438f67f3dc62465999a917986922b3ed [id=7c7fcd265e624d7fab74661db505d9c3])
I20251025 07:52:35.792311 710 leader_election.cc:304] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:35.792641 1667 tablet_loader.cc:96] loaded metadata for tablet 8ff73ffc42434f74a4949f156771d5a4 (table default.Table_e244b2d2db74449aabd10a8b28e4afd2 [id=8bb9f6cbc77241f0b64c7ab15d1fe266])
I20251025 07:52:35.792861 1441 raft_consensus.cc:2749] T 9979cae195e849908677d78c6c33be71 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:35.792945 1670 raft_consensus.cc:2749] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:35.793056 1441 consensus_queue.cc:1048] T fa77085e81c7440d9e255c44d1e80c26 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:35.793468 1167 tablet_bootstrap.cc:654] T d5e5e5bbcf9f43118eea502b69880af5 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:35.794840 1667 tablet_loader.cc:96] loaded metadata for tablet 92aa605fabbc4257a6a204c087e7cc7f (table default.Table_78724078b4574f35879fd2db388fc4f4 [id=187bde879b044237ada58158eb000123])
I20251025 07:52:35.795020 1671 raft_consensus.cc:2749] T bef0ef23d90d480d9c99aef21d7271fd P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:35.796178 1667 tablet_loader.cc:96] loaded metadata for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236])
I20251025 07:52:35.796644 1667 tablet_loader.cc:96] loaded metadata for tablet 9979cae195e849908677d78c6c33be71 (table default.Table_208cd3f3c07744ed92b1260ebb401310 [id=d6c5598979e84e36a3bf1996f37f9c28])
I20251025 07:52:35.796984 1667 tablet_loader.cc:96] loaded metadata for tablet 99ea11a2729442918a0a6485d0d83cac (table default.Table_a0665a4cb38b40109df3991cb2f71ed6 [id=dbd92b3210f24fc2b8146dcfcb18833a])
I20251025 07:52:35.797328 1667 tablet_loader.cc:96] loaded metadata for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3])
I20251025 07:52:35.798440 1667 tablet_loader.cc:96] loaded metadata for tablet 9f039d9107ae493486d799ab6a1d731d (table default.Table_3b10c71d2c1b4bd2849e0cad9ec767d7 [id=ee09c3f16a194bbf97dd2768b5231c06])
I20251025 07:52:35.799186 1667 tablet_loader.cc:96] loaded metadata for tablet a0ea1edeca7347fc844faa5e4be1ae94 (table default.Table_7f5a5345567a48d09dedce1df9d63a3b [id=7eaeb32957a946998e90658207639ed7])
I20251025 07:52:35.802493 1829 consensus_queue.cc:1048] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:35.809890 1667 tablet_loader.cc:96] loaded metadata for tablet a6587f4be6af44dd944e6b6f9599dfda (table default.Table_1431c97720de4d5a9a92bd2d574eb904 [id=484c28bccff0448bbca48cb7eaaa55df])
I20251025 07:52:35.810328 1667 tablet_loader.cc:96] loaded metadata for tablet a6ae1315b28940d49b667e3c80347021 (table default.Table_ac3211ef7b0941d2ad4db2fab62231b2 [id=f721af9327cd4e00a20fc1bfebd9e17b])
I20251025 07:52:35.817001 1671 consensus_queue.cc:1048] T a6ddefa1738f4ff98586900c344359d2 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:35.817497 1667 tablet_loader.cc:96] loaded metadata for tablet a758a518c37b461f85e7b98e87020fc1 (table default.Table_ab87db32c7024d709eee5476d53f17ce [id=b1458cb701e14086a5a7fd82db235571])
I20251025 07:52:35.820711 1166 tablet_bootstrap.cc:492] T db99a1e68daf4e948a3d66ad276cc5f9 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:35.821019 1166 ts_tablet_manager.cc:1403] T db99a1e68daf4e948a3d66ad276cc5f9 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.244s user 0.010s sys 0.002s
I20251025 07:52:35.822196 1166 raft_consensus.cc:359] T db99a1e68daf4e948a3d66ad276cc5f9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.822531 1166 raft_consensus.cc:385] T db99a1e68daf4e948a3d66ad276cc5f9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:35.822681 1166 raft_consensus.cc:740] T db99a1e68daf4e948a3d66ad276cc5f9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:35.823187 1667 tablet_loader.cc:96] loaded metadata for tablet a77eb82f26ae43b48de1e85693372d34 (table default.Table_bcd5afa1072f47898d67504fc72b84c4 [id=3d6f0426d32f481ba8152eaacb298784])
I20251025 07:52:35.823020 1166 consensus_queue.cc:260] T db99a1e68daf4e948a3d66ad276cc5f9 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.823900 1667 tablet_loader.cc:96] loaded metadata for tablet a7ab3150a35b4caca9119a1374d3f2f7 (table default.Table_8129e15bf5c549bd853c04b227a7ef26 [id=e9e8547061ba4ac987d0aa7616c7f3ea])
I20251025 07:52:35.829417 1667 tablet_loader.cc:96] loaded metadata for tablet a8fddd7cd7f543b190fb608a331bb3ac (table default.Table_58bd2fb570eb4cfebf3ce98477e23525 [id=f4668f4d18304a619724e52d76d6c915])
I20251025 07:52:35.830555 1667 tablet_loader.cc:96] loaded metadata for tablet aa9d9ab8073945d28d52fb02213f201a (table default.Table_4791672658f043f6b97b9320a531ebb5 [id=dfd20cf9c3e541c4abe23d59e0aa7f2a])
I20251025 07:52:35.831962 1667 tablet_loader.cc:96] loaded metadata for tablet ab42909afc524f2fa3c1041495d13a59 (table default.Table_7eb8424d5192485da1a2d67bfdb2bbbf [id=40c29df99590408cbd5c583d4fc741eb])
I20251025 07:52:35.833359 1667 tablet_loader.cc:96] loaded metadata for tablet ab68e5a89b234064b27aa7cae9467e14 (table default.Table_4b1446ea61a9442fb3e397dfe2c5fb03 [id=e9652929eaab46cab42bf8d6896577ed])
I20251025 07:52:35.843067 1659 raft_consensus.cc:493] T c59f6e6eb9594a8e83f109a09b034045 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:35.843225 1667 tablet_loader.cc:96] loaded metadata for tablet aea8fcf0154a4bd4a05b19e27ab78b79 (table default.Table_c26454d3c928434ab0dae167abcc2ff2 [id=4cd0c9a010af47939a90da1132b9e6be])
I20251025 07:52:35.776996 1169 raft_consensus.cc:359] T f3001314b56345f5a70314777cae0125 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.843430 1169 raft_consensus.cc:385] T f3001314b56345f5a70314777cae0125 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:35.843400 1659 raft_consensus.cc:515] T c59f6e6eb9594a8e83f109a09b034045 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.843513 1169 raft_consensus.cc:740] T f3001314b56345f5a70314777cae0125 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:35.843776 1169 consensus_queue.cc:260] T f3001314b56345f5a70314777cae0125 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.844688 1659 leader_election.cc:290] T c59f6e6eb9594a8e83f109a09b034045 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:35.845105 1169 ts_tablet_manager.cc:1434] T f3001314b56345f5a70314777cae0125 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.071s user 0.002s sys 0.000s
I20251025 07:52:35.845314 1667 tablet_loader.cc:96] loaded metadata for tablet b02f5c0ab0b24288bb50a3ba0e6bd78a (table default.Table_29aeaf089ca24270b3f21c2e08d61634 [id=3294d42240024d25ab2487e75640ddd0])
I20251025 07:52:35.846262 1169 tablet_bootstrap.cc:492] T c4610b0c62e741eca28c21f63cc3de81 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:35.846697 1667 tablet_loader.cc:96] loaded metadata for tablet b17d2a9ad6be49d29f2413666fba7e69 (table default.Table_fcddb31989ee4236a4981c12779b3f5d [id=f3794fa081364666b944c412415bdba0])
I20251025 07:52:35.847317 1612 consensus_queue.cc:1048] T a6ddefa1738f4ff98586900c344359d2 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:35.847865 1667 tablet_loader.cc:96] loaded metadata for tablet b2044ca206634eb285197af4ba2073e7 (table default.Table_3292d47465544af0970067fb567e7ede [id=41fc0b99786749259c0a0059dec97ea1])
I20251025 07:52:35.848840 1667 tablet_loader.cc:96] loaded metadata for tablet b71bdfd09a27449d98509bb220680147 (table default.Table_f24cd210ae3342e68c69aa6670e3e4b0 [id=22cab1e94aa544fa828350f9a70f8555])
I20251025 07:52:35.851162 1667 tablet_loader.cc:96] loaded metadata for tablet b928bc638aaa4700ba7a6478a5097c5b (table default.Table_d5ae5bc2cb804015ba50415cd3f62847 [id=951b49e5f0ca41488c2e92be79288158])
I20251025 07:52:35.851811 1659 raft_consensus.cc:493] T b97e39aaac3048ed93dbfe873b757254 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:35.851977 1659 raft_consensus.cc:515] T b97e39aaac3048ed93dbfe873b757254 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.853173 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b97e39aaac3048ed93dbfe873b757254" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:35.854629 1667 tablet_loader.cc:96] loaded metadata for tablet b97e39aaac3048ed93dbfe873b757254 (table default.Table_99fd36369f9a4e0fb14f234c98635203 [id=c8059ab6cec146efa631d06799d969a9])
W20251025 07:52:35.855435 712 leader_election.cc:343] T b97e39aaac3048ed93dbfe873b757254 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:35.858759 1659 leader_election.cc:290] T b97e39aaac3048ed93dbfe873b757254 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:35.859071 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c59f6e6eb9594a8e83f109a09b034045" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:35.859876 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c59f6e6eb9594a8e83f109a09b034045" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:35.860391 710 leader_election.cc:343] T c59f6e6eb9594a8e83f109a09b034045 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:35.860745 905 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b97e39aaac3048ed93dbfe873b757254" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:35.861240 710 leader_election.cc:343] T b97e39aaac3048ed93dbfe873b757254 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:35.861755 1667 tablet_loader.cc:96] loaded metadata for tablet bae843ed66fb4dcc8af9ab55693439ac (table default.Table_8d0e78b0bb8d433f9ed8e49d24d7e95d [id=8be005a6c0ba404da6ebed5da21f7498])
I20251025 07:52:35.862787 1667 tablet_loader.cc:96] loaded metadata for tablet bbe1d40d67594ad389ccc97bb9e2ac3b (table default.Table_3f291c48b0144f0a8ddb1b51ebbe31cd [id=98db26b2e7bf409dbfe8d201ae701df7])
I20251025 07:52:35.864396 1667 tablet_loader.cc:96] loaded metadata for tablet bc2d16b33ca8426abbd7bd38339c9f1d (table default.Table_4ea985df4c5b415c9570c64506a98cc4 [id=98d2b7797e33485e8872054aa2eb8ec9])
I20251025 07:52:35.865147 1166 ts_tablet_manager.cc:1434] T db99a1e68daf4e948a3d66ad276cc5f9 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.044s user 0.002s sys 0.000s
I20251025 07:52:35.865705 1667 tablet_loader.cc:96] loaded metadata for tablet bdaeedbcf35a4aad97f9ef192991978a (table default.Table_cd12897d1fc5454ab07cdc13fefb6978 [id=1655781ab7a046ad8e6dcfacaf2db1b1])
I20251025 07:52:35.867895 1667 tablet_loader.cc:96] loaded metadata for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049])
I20251025 07:52:35.868674 1166 tablet_bootstrap.cc:492] T c59f6e6eb9594a8e83f109a09b034045 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:35.869370 1667 tablet_loader.cc:96] loaded metadata for tablet c187a40f8ff14c6a8c184fbeb030ad13 (table default.Table_34b975902f884c658093a51baa54b095 [id=ec0cc315ecba4988a97d68f00ff22697])
I20251025 07:52:35.869946 1667 tablet_loader.cc:96] loaded metadata for tablet c2713942584e4734acbc0775ead772fa (table default.Table_6140b1f390084256848ae1b746059268 [id=adaaf2bcd09f4c779aee11bb7cd4ba15])
I20251025 07:52:35.870461 1166 tablet_bootstrap.cc:654] T c59f6e6eb9594a8e83f109a09b034045 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:35.871207 1667 tablet_loader.cc:96] loaded metadata for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_87d38832f85e4965bfab1a077990114f [id=ad812862698d421184ebf501ce79b7be])
W20251025 07:52:35.873241 1354 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.62:37421, user_credentials={real_user=slave}} blocked reactor thread for 119120us
I20251025 07:52:35.873572 1667 tablet_loader.cc:96] loaded metadata for tablet c7691c7bdd89437c8db060254c0db25d (table default.Table_1f11da1b07914daabda9c8e29a9230fe [id=27557bb332dd49bb8f386c294bb12a69])
I20251025 07:52:35.873832 710 leader_election.cc:304] T b97e39aaac3048ed93dbfe873b757254 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:35.874372 712 leader_election.cc:343] T c59f6e6eb9594a8e83f109a09b034045 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:35.874532 712 leader_election.cc:304] T c59f6e6eb9594a8e83f109a09b034045 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:35.874981 1829 raft_consensus.cc:2749] T c59f6e6eb9594a8e83f109a09b034045 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:35.874261 1659 raft_consensus.cc:2749] T b97e39aaac3048ed93dbfe873b757254 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:35.875233 1707 consensus_queue.cc:1048] T b772025852794a49b072f37cccbd3291 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:35.881069 1667 tablet_loader.cc:96] loaded metadata for tablet c87ab6a3d08f43e0b2fe58a6d7b11588 (table default.Table_dde55dddf5d14de084003c22da08f6cd [id=c3f37f5469ab4d60a7597994c11357cf])
I20251025 07:52:35.884032 1169 tablet_bootstrap.cc:654] T c4610b0c62e741eca28c21f63cc3de81 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:35.892089 1659 consensus_queue.cc:1048] T 7f29b64704e147aa9b0b5f6cda860f09 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:35.902283 1667 tablet_loader.cc:96] loaded metadata for tablet c9f7ba3709c745d4a625449d6e1c8ba1 (table default.Table_380ca630f4144f40a04084d2ef8600e3 [id=06b6a39af6764e44972d57ddd722f838])
I20251025 07:52:35.912487 1667 tablet_loader.cc:96] loaded metadata for tablet cb829649f73d4d3f8862d268c953bd86 (table default.Table_ba4b8cc7b930482690b7e192e0edff12 [id=7944225d3ce7439b881bfb9289d9f040])
I20251025 07:52:35.915107 1667 tablet_loader.cc:96] loaded metadata for tablet cd23b8094d7e472eba6bd170248858a9 (table default.Table_dfc7f1d4bf8b48bebbb00313419c3fd8 [id=d82b3e1317434cf08a0d51e8a9427224])
I20251025 07:52:35.916944 1667 tablet_loader.cc:96] loaded metadata for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7])
I20251025 07:52:35.917783 1829 consensus_queue.cc:1048] T fa77085e81c7440d9e255c44d1e80c26 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:35.949532 1667 tablet_loader.cc:96] loaded metadata for tablet d40519ed9ee746348b17be2786d60e95 (table default.Table_7d2010ece331457fa18a0f13ac5cbdfa [id=c0e60203ceb3445ca63dc5f8f81c37b3])
I20251025 07:52:35.950196 1667 tablet_loader.cc:96] loaded metadata for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_a6c869949b744bc98efb735b0e73e7aa [id=bf77b5ff6f1048dd996d948437f9ff5e])
I20251025 07:52:35.952805 1667 tablet_loader.cc:96] loaded metadata for tablet dbb254115a574bf4976aff0ea56f3bd9 (table default.Table_a55d3aa69b804240b042e7b94ca493d5 [id=83841f9dd7b34f27a36830af24022ab9])
I20251025 07:52:35.953295 1667 tablet_loader.cc:96] loaded metadata for tablet dc4a788dca9d4a8ba40f8fdf1a4b915e (table default.Table_d5ce3dfc91174e268c17c2d5b3f93b86 [id=607c24e5c581467db1eac04c5165450d])
I20251025 07:52:35.954433 1667 tablet_loader.cc:96] loaded metadata for tablet de8da6abeda74a2a96d94ce5ac8fb7e5 (table default.Table_17e5895bc86c441ba014d094c1bf3893 [id=c8c37f4cecb94242b3cea6e61a2c1cc4])
I20251025 07:52:35.954895 1667 tablet_loader.cc:96] loaded metadata for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_e90d7f5ff13b40878e5b625c97792c7d [id=fc8180c09d7b4e74b2e161f7acb8fe0e])
I20251025 07:52:35.958550 1667 tablet_loader.cc:96] loaded metadata for tablet df1248f6dbd94af68c587da7d149e279 (table default.Table_73279bdee234454e9def34441da9579a [id=6430d82b82c042469a77ea9f4dad15fc])
I20251025 07:52:35.962831 1667 tablet_loader.cc:96] loaded metadata for tablet df9db9dd31d840e0b1394db04d662a62 (table default.Table_da4ed35a8ed24427b6f4499fd1cc203c [id=cdbb76e674fc41cfb0f86d678a3f4f0b])
I20251025 07:52:35.963785 1670 raft_consensus.cc:493] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:35.963954 1670 raft_consensus.cc:515] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.964119 1667 tablet_loader.cc:96] loaded metadata for tablet dff7f86ae8924685b26011e504e2247b (table default.Table_dbface3af7f6413d9060b881447d34b3 [id=f74f5abb40284e08a990f9caaa828636])
I20251025 07:52:35.964637 1667 tablet_loader.cc:96] loaded metadata for tablet e429b94326cc4827a03968fd92de38e0 (table default.Table_0b0bd6b71b92454fb8cdf967ffd8349e [id=e9b21659de7d4276aa1f9effb7076eb4])
I20251025 07:52:35.964656 1670 leader_election.cc:290] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:35.965145 1667 tablet_loader.cc:96] loaded metadata for tablet e47c4f741f354a19b49777bd06e54764 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439])
I20251025 07:52:35.967901 1667 tablet_loader.cc:96] loaded metadata for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e])
I20251025 07:52:35.973517 1667 tablet_loader.cc:96] loaded metadata for tablet e9e8fffb8e984463b5fa2c7dd9e93227 (table default.Table_3353358c7aab4407880b823e1500b6d3 [id=70586b36140141d9b7579a32df907ae5])
I20251025 07:52:35.973664 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be5ab0db6ac34c2c9a370445e251eb46" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:35.974174 710 leader_election.cc:343] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:35.974807 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be5ab0db6ac34c2c9a370445e251eb46" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:35.975090 1044 raft_consensus.cc:2468] T be5ab0db6ac34c2c9a370445e251eb46 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:35.975524 1667 tablet_loader.cc:96] loaded metadata for tablet e9f7cd765d614d59a7c190b744b0844d (table default.Table_5ecc1936fc8447669d0602cfd6133c6b [id=551028aad5c246378c7acb8e443853a7])
I20251025 07:52:35.975543 712 leader_election.cc:304] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters: 3aea2869ff814cc7b142012faa274382
I20251025 07:52:35.975833 1670 raft_consensus.cc:2804] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:35.975946 1670 raft_consensus.cc:493] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:35.976025 1670 raft_consensus.cc:3060] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.976387 1667 tablet_loader.cc:96] loaded metadata for tablet ebf3a2a926754305b5bd95fcebb25d29 (table default.Table_ac4beca0c3bd4cf29565faf5323d9ff7 [id=857bcd1dbe2e477bbc0d77f79a58f7f7])
I20251025 07:52:35.977883 1667 tablet_loader.cc:96] loaded metadata for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c])
I20251025 07:52:35.978770 1667 tablet_loader.cc:96] loaded metadata for tablet ef1854d35a3047e3a9fbe0166ab1295a (table default.Table_088aad2a46b041bfa29ed8e6929760b1 [id=834669c54dae449790818684fd8659bc])
I20251025 07:52:35.979339 1670 raft_consensus.cc:515] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.979841 1167 tablet_bootstrap.cc:492] T d5e5e5bbcf9f43118eea502b69880af5 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:35.980010 1167 ts_tablet_manager.cc:1403] T d5e5e5bbcf9f43118eea502b69880af5 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.192s user 0.030s sys 0.011s
I20251025 07:52:35.980844 1167 raft_consensus.cc:359] T d5e5e5bbcf9f43118eea502b69880af5 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.981052 1167 raft_consensus.cc:385] T d5e5e5bbcf9f43118eea502b69880af5 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:35.981151 1167 raft_consensus.cc:740] T d5e5e5bbcf9f43118eea502b69880af5 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:35.981472 1167 consensus_queue.cc:260] T d5e5e5bbcf9f43118eea502b69880af5 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:35.981482 1670 leader_election.cc:290] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:35.981768 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be5ab0db6ac34c2c9a370445e251eb46" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:35.983350 1166 tablet_bootstrap.cc:492] T c59f6e6eb9594a8e83f109a09b034045 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
W20251025 07:52:35.983634 710 leader_election.cc:343] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:35.983708 1166 ts_tablet_manager.cc:1403] T c59f6e6eb9594a8e83f109a09b034045 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.115s user 0.004s sys 0.000s
I20251025 07:52:35.984090 1167 ts_tablet_manager.cc:1434] T d5e5e5bbcf9f43118eea502b69880af5 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251025 07:52:35.984401 1167 tablet_bootstrap.cc:492] T 53198283718d4b45b30df65b5bde96a6 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:35.986322 1167 tablet_bootstrap.cc:654] T 53198283718d4b45b30df65b5bde96a6 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:35.987404 1169 tablet_bootstrap.cc:492] T c4610b0c62e741eca28c21f63cc3de81 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:35.987798 1169 ts_tablet_manager.cc:1403] T c4610b0c62e741eca28c21f63cc3de81 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.142s user 0.003s sys 0.002s
I20251025 07:52:35.988101 1667 tablet_loader.cc:96] loaded metadata for tablet ef9a2c9f2ccf432ab835396020872208 (table default.Table_2b9b772d4c234ac286bef3658482e490 [id=409fab60f9d744798e6f92b868f0f7c2])
I20251025 07:52:35.988399 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be5ab0db6ac34c2c9a370445e251eb46" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:35.988801 1044 raft_consensus.cc:3060] T be5ab0db6ac34c2c9a370445e251eb46 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:35.999506 1667 tablet_loader.cc:96] loaded metadata for tablet f33ff0d79e8846c5b58453f097d2ea88 (table default.Table_6eb139482bc940989a606a0d8745d54f [id=616b3a99164f47fbb918c4e91d0c2e14])
I20251025 07:52:36.001111 1667 tablet_loader.cc:96] loaded metadata for tablet f65ae042d1da472f9e256c125ce7898a (table default.Table_74277c3abac14953b2eb73c850e3e727 [id=afdcd9fdb01f4cad85633186bed13682])
I20251025 07:52:36.001837 1667 tablet_loader.cc:96] loaded metadata for tablet f7011f1dcad74fe4ba0356c618890e06 (table default.Table_6f5fbeb82b024bfe95511811dcc51406 [id=9c2312d7095f407d9d30f224cdce66c6])
I20251025 07:52:36.003054 1667 tablet_loader.cc:96] loaded metadata for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84])
I20251025 07:52:36.004161 1167 tablet_bootstrap.cc:492] T 53198283718d4b45b30df65b5bde96a6 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:36.004447 1167 ts_tablet_manager.cc:1403] T 53198283718d4b45b30df65b5bde96a6 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.004s
I20251025 07:52:36.005270 1167 raft_consensus.cc:359] T 53198283718d4b45b30df65b5bde96a6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.005568 1167 raft_consensus.cc:385] T 53198283718d4b45b30df65b5bde96a6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.005708 1167 raft_consensus.cc:740] T 53198283718d4b45b30df65b5bde96a6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.006013 1167 consensus_queue.cc:260] T 53198283718d4b45b30df65b5bde96a6 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.006604 1167 ts_tablet_manager.cc:1434] T 53198283718d4b45b30df65b5bde96a6 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:36.006971 1167 tablet_bootstrap.cc:492] T 07898860aa594a05a211fc89768cb76d P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:36.008869 1167 tablet_bootstrap.cc:654] T 07898860aa594a05a211fc89768cb76d P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.009227 1667 tablet_loader.cc:96] loaded metadata for tablet fa998b9ca6724c4cb1eb68a3eecb0b11 (table default.Table_4976d4f5d05540258d144751d129bdb8 [id=8cde66661cf749919839da894ed4dc05])
I20251025 07:52:36.011005 1667 tablet_loader.cc:96] loaded metadata for tablet fce535a0c0bd45c6bf17f81add297f40 (table default.Table_1de0b151cf4c4c10aca35e45623caf2c [id=a947f21398254a53bc6668c4f8d39e53])
I20251025 07:52:36.011744 1667 tablet_loader.cc:96] loaded metadata for tablet fd9b4be139d347f4a6a60767029c342a (table default.Table_4145d1c9c33c4351a89976c5c4c18d75 [id=58190b46780d49eb9f429626ddce531c])
I20251025 07:52:36.013284 1667 tablet_loader.cc:96] loaded metadata for tablet fe5033e5ff404a5ebc6c8443d7548327 (table default.Table_054db0ca31bd42ebb5d61ec81db8146b [id=85f4e4639f224541a0dec677ff5e62d5])
W20251025 07:52:36.014258 1667 catalog_manager.cc:1486] Time spent T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Loading table and tablet metadata into memory: real 1.338s user 0.219s sys 0.048s
I20251025 07:52:36.014430 1667 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251025 07:52:36.015409 454 rpcz_store.cc:275] Call kudu.master.MasterService.ConnectToMaster from 127.0.0.1:47610 (request call id 8) took 1328 ms. Trace:
I20251025 07:52:36.018162 454 rpcz_store.cc:276] 1025 07:52:34.687120 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:34.687176 (+ 56us) service_pool.cc:225] Handling call
1025 07:52:36.015362 (+1328186us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 07:52:36.019390 464 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b4a999d754c941ec850c143e4e24c407" candidate_term: 3 candidate_status { last_received { term: 2 index: 668 } } ignore_live_leader: false dest_uuid: "b2dfbd0b44a441c9be9254f660974944" is_pre_election: true
I20251025 07:52:36.019877 463 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b4a999d754c941ec850c143e4e24c407" candidate_term: 3 candidate_status { last_received { term: 2 index: 668 } } ignore_live_leader: false dest_uuid: "b2dfbd0b44a441c9be9254f660974944" is_pre_election: true
W20251025 07:52:36.020170 463 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59782 (request call id 3) took 951 ms (client timeout 684 ms). Trace:
I20251025 07:52:36.020114 1354 leader_election.cc:304] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b4a999d754c941ec850c143e4e24c407; no voters: 8603bddd8a084ff7a73a25641768dfa9, b2dfbd0b44a441c9be9254f660974944
W20251025 07:52:36.020334 463 rpcz_store.cc:269] 1025 07:52:35.068725 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:35.068795 (+ 70us) service_pool.cc:225] Handling call
1025 07:52:36.020118 (+951323us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 07:52:36.022619 1166 raft_consensus.cc:359] T c59f6e6eb9594a8e83f109a09b034045 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.023263 1167 tablet_bootstrap.cc:492] T 07898860aa594a05a211fc89768cb76d P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:36.023440 1167 ts_tablet_manager.cc:1403] T 07898860aa594a05a211fc89768cb76d P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20251025 07:52:36.024351 1167 raft_consensus.cc:359] T 07898860aa594a05a211fc89768cb76d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:36.024678 1167 raft_consensus.cc:385] T 07898860aa594a05a211fc89768cb76d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.024771 1167 raft_consensus.cc:740] T 07898860aa594a05a211fc89768cb76d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.024989 1167 consensus_queue.cc:260] T 07898860aa594a05a211fc89768cb76d P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:36.025555 1167 ts_tablet_manager.cc:1434] T 07898860aa594a05a211fc89768cb76d P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:36.026031 1167 tablet_bootstrap.cc:492] T c88c46e022ff41a58b02538f2151e260 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:36.028077 1167 tablet_bootstrap.cc:654] T c88c46e022ff41a58b02538f2151e260 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.028954 1166 raft_consensus.cc:385] T c59f6e6eb9594a8e83f109a09b034045 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.029143 1166 raft_consensus.cc:740] T c59f6e6eb9594a8e83f109a09b034045 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.044744 1659 consensus_queue.cc:1048] T 2209089667dd4937abd96615f6929449 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:36.049474 1667 catalog_manager.cc:1269] Loaded cluster ID: aee99068aba246df9094aaec3350d6ff
I20251025 07:52:36.050760 1167 tablet_bootstrap.cc:492] T c88c46e022ff41a58b02538f2151e260 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:36.051017 1167 ts_tablet_manager.cc:1403] T c88c46e022ff41a58b02538f2151e260 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.002s
I20251025 07:52:36.051990 1167 raft_consensus.cc:359] T c88c46e022ff41a58b02538f2151e260 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.052297 1167 raft_consensus.cc:385] T c88c46e022ff41a58b02538f2151e260 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.052415 1167 raft_consensus.cc:740] T c88c46e022ff41a58b02538f2151e260 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.052759 1167 consensus_queue.cc:260] T c88c46e022ff41a58b02538f2151e260 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.053504 1167 ts_tablet_manager.cc:1434] T c88c46e022ff41a58b02538f2151e260 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251025 07:52:36.053968 1167 tablet_bootstrap.cc:492] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:36.056041 1167 tablet_bootstrap.cc:654] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.057199 1667 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251025 07:52:36.062625 1169 raft_consensus.cc:359] T c4610b0c62e741eca28c21f63cc3de81 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.062887 1169 raft_consensus.cc:385] T c4610b0c62e741eca28c21f63cc3de81 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.062834 1166 consensus_queue.cc:260] T c59f6e6eb9594a8e83f109a09b034045 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.062971 1169 raft_consensus.cc:740] T c4610b0c62e741eca28c21f63cc3de81 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.063387 1169 consensus_queue.cc:260] T c4610b0c62e741eca28c21f63cc3de81 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.065227 1166 ts_tablet_manager.cc:1434] T c59f6e6eb9594a8e83f109a09b034045 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.081s user 0.002s sys 0.000s
I20251025 07:52:36.066303 1659 consensus_queue.cc:1048] T cb824bd780524832b2b60f5618a39da5 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:36.076699 1166 tablet_bootstrap.cc:492] T c4610b0c62e741eca28c21f63cc3de81 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:36.079727 1659 raft_consensus.cc:493] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:36.079952 1659 raft_consensus.cc:515] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.080751 1659 leader_election.cc:290] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:36.081535 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3001314b56345f5a70314777cae0125" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:36.081758 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3001314b56345f5a70314777cae0125" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:36.082023 1045 raft_consensus.cc:2468] T f3001314b56345f5a70314777cae0125 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
W20251025 07:52:36.082391 710 leader_election.cc:343] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:36.082595 712 leader_election.cc:304] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters: 3aea2869ff814cc7b142012faa274382
I20251025 07:52:36.083096 1659 raft_consensus.cc:2804] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:36.083185 1659 raft_consensus.cc:493] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:36.083281 1659 raft_consensus.cc:3060] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:36.084074 1166 tablet_bootstrap.cc:654] T c4610b0c62e741eca28c21f63cc3de81 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.085201 1169 ts_tablet_manager.cc:1434] T c4610b0c62e741eca28c21f63cc3de81 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.097s user 0.001s sys 0.001s
I20251025 07:52:36.087183 1667 catalog_manager.cc:1514] Loading token signing keys...
I20251025 07:52:36.090018 1908 raft_consensus.cc:2749] T 00000000000000000000000000000000 P b4a999d754c941ec850c143e4e24c407 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251025 07:52:36.098924 1354 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.62:37421, user_credentials={real_user=slave}} blocked reactor thread for 78824.2us
I20251025 07:52:36.102572 1829 consensus_queue.cc:1048] T cb824bd780524832b2b60f5618a39da5 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:36.102746 1167 tablet_bootstrap.cc:492] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:36.102906 1167 ts_tablet_manager.cc:1403] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.049s user 0.004s sys 0.001s
I20251025 07:52:36.103744 1167 raft_consensus.cc:359] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:36.103930 1167 raft_consensus.cc:385] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.104012 1167 raft_consensus.cc:740] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.104255 1167 consensus_queue.cc:260] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:36.104813 1167 ts_tablet_manager.cc:1434] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:36.105094 1167 tablet_bootstrap.cc:492] T 509c94f2a422493ea4390ecee576db72 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:36.109409 1659 raft_consensus.cc:515] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.110805 1167 tablet_bootstrap.cc:654] T 509c94f2a422493ea4390ecee576db72 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.112803 1659 leader_election.cc:290] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:36.112814 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3001314b56345f5a70314777cae0125" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:36.113016 1048 raft_consensus.cc:3060] T f3001314b56345f5a70314777cae0125 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:36.113502 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3001314b56345f5a70314777cae0125" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
W20251025 07:52:36.114161 710 leader_election.cc:343] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:36.125273 1659 raft_consensus.cc:493] T d5e5e5bbcf9f43118eea502b69880af5 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:36.125586 1659 raft_consensus.cc:515] T d5e5e5bbcf9f43118eea502b69880af5 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.127151 1659 leader_election.cc:290] T d5e5e5bbcf9f43118eea502b69880af5 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:36.127513 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5e5e5bbcf9f43118eea502b69880af5" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:36.129627 1167 tablet_bootstrap.cc:492] T 509c94f2a422493ea4390ecee576db72 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:36.129874 1167 ts_tablet_manager.cc:1403] T 509c94f2a422493ea4390ecee576db72 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.002s
W20251025 07:52:36.130102 712 leader_election.cc:343] T d5e5e5bbcf9f43118eea502b69880af5 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:36.130820 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5e5e5bbcf9f43118eea502b69880af5" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:36.130931 1167 raft_consensus.cc:359] T 509c94f2a422493ea4390ecee576db72 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.131166 1167 raft_consensus.cc:385] T 509c94f2a422493ea4390ecee576db72 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.131259 1167 raft_consensus.cc:740] T 509c94f2a422493ea4390ecee576db72 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
W20251025 07:52:36.131542 710 leader_election.cc:343] T d5e5e5bbcf9f43118eea502b69880af5 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:36.131711 710 leader_election.cc:304] T d5e5e5bbcf9f43118eea502b69880af5 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:36.131970 1659 raft_consensus.cc:2749] T d5e5e5bbcf9f43118eea502b69880af5 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:36.131544 1167 consensus_queue.cc:260] T 509c94f2a422493ea4390ecee576db72 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.133112 1167 ts_tablet_manager.cc:1434] T 509c94f2a422493ea4390ecee576db72 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251025 07:52:36.133706 1169 tablet_bootstrap.cc:492] T c59f6e6eb9594a8e83f109a09b034045 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:36.138455 1675 consensus_queue.cc:1048] T 3775a2e08d4e40bd9f57e4cda759d535 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:36.141201 1667 catalog_manager.cc:6033] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Loaded TSK: 0
I20251025 07:52:36.141969 1441 consensus_queue.cc:1048] T 2209089667dd4937abd96615f6929449 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:36.143177 1667 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251025 07:52:36.143420 1044 raft_consensus.cc:2468] T be5ab0db6ac34c2c9a370445e251eb46 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:36.147708 712 leader_election.cc:304] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters: 3aea2869ff814cc7b142012faa274382
I20251025 07:52:36.148425 1441 raft_consensus.cc:2804] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:36.148576 1441 raft_consensus.cc:697] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:36.150658 1441 consensus_queue.cc:237] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.156020 1612 consensus_queue.cc:1048] T 3775a2e08d4e40bd9f57e4cda759d535 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:36.156669 1048 raft_consensus.cc:2468] T f3001314b56345f5a70314777cae0125 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:36.157219 712 leader_election.cc:304] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters: 3aea2869ff814cc7b142012faa274382
I20251025 07:52:36.157732 1829 raft_consensus.cc:2804] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:36.157912 1829 raft_consensus.cc:697] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:36.158324 1829 consensus_queue.cc:237] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.159960 1612 raft_consensus.cc:493] T 1d261e432cf5403ea162129706bc4d4d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:36.160138 1612 raft_consensus.cc:515] T 1d261e432cf5403ea162129706bc4d4d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.160811 1612 leader_election.cc:290] T 1d261e432cf5403ea162129706bc4d4d P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:36.161532 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d261e432cf5403ea162129706bc4d4d" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:36.165274 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d261e432cf5403ea162129706bc4d4d" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:36.165531 774 raft_consensus.cc:2393] T 1d261e432cf5403ea162129706bc4d4d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:36.166155 982 leader_election.cc:304] T 1d261e432cf5403ea162129706bc4d4d P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:36.166566 1569 raft_consensus.cc:3060] T 1d261e432cf5403ea162129706bc4d4d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:36.167418 1167 tablet_bootstrap.cc:492] T 2c50ec95e48e49b59a76fac4fc862d3d P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:36.169386 1569 raft_consensus.cc:2749] T 1d261e432cf5403ea162129706bc4d4d P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:36.169948 1167 tablet_bootstrap.cc:654] T 2c50ec95e48e49b59a76fac4fc862d3d P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.175343 1169 tablet_bootstrap.cc:654] T c59f6e6eb9594a8e83f109a09b034045 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.187937 1166 tablet_bootstrap.cc:492] T c4610b0c62e741eca28c21f63cc3de81 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:36.188196 1166 ts_tablet_manager.cc:1403] T c4610b0c62e741eca28c21f63cc3de81 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.112s user 0.006s sys 0.005s
I20251025 07:52:36.189235 1166 raft_consensus.cc:359] T c4610b0c62e741eca28c21f63cc3de81 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.189466 1166 raft_consensus.cc:385] T c4610b0c62e741eca28c21f63cc3de81 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.189560 1166 raft_consensus.cc:740] T c4610b0c62e741eca28c21f63cc3de81 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.189827 1166 consensus_queue.cc:260] T c4610b0c62e741eca28c21f63cc3de81 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.190675 1166 ts_tablet_manager.cc:1434] T c4610b0c62e741eca28c21f63cc3de81 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:36.191037 1166 tablet_bootstrap.cc:492] T 71432a6848944a0da06ef9b81f160dc4 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:36.191655 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 509c94f2a422493ea4390ecee576db72 (table default.Table_3f081b7bb4e1420c8121ea4115439f3d [id=696a1285fbfe4038a8b1b38708c8ea49]). Expected version 1 got 0
W20251025 07:52:36.193145 453 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 509c94f2a422493ea4390ecee576db72 (table default.Table_3f081b7bb4e1420c8121ea4115439f3d [id=696a1285fbfe4038a8b1b38708c8ea49], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 509c94f2a422493ea4390ecee576db72 consensus state has no leader
I20251025 07:52:36.194741 1169 tablet_bootstrap.cc:492] T c59f6e6eb9594a8e83f109a09b034045 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:36.194913 1169 ts_tablet_manager.cc:1403] T c59f6e6eb9594a8e83f109a09b034045 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.061s user 0.004s sys 0.000s
I20251025 07:52:36.195720 1169 raft_consensus.cc:359] T c59f6e6eb9594a8e83f109a09b034045 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.195950 1169 raft_consensus.cc:385] T c59f6e6eb9594a8e83f109a09b034045 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.196044 1169 raft_consensus.cc:740] T c59f6e6eb9594a8e83f109a09b034045 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.196357 1169 consensus_queue.cc:260] T c59f6e6eb9594a8e83f109a09b034045 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.196692 1095 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
I20251025 07:52:36.197126 1169 ts_tablet_manager.cc:1434] T c59f6e6eb9594a8e83f109a09b034045 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 07:52:36.197544 1169 tablet_bootstrap.cc:492] T db99a1e68daf4e948a3d66ad276cc5f9 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:36.201848 1011 tablet_service.cc:1552] Processing DeleteTablet for tablet c4610b0c62e741eca28c21f63cc3de81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:36.202241 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 2209089667dd4937abd96615f6929449 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:36.202473 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet cb824bd780524832b2b60f5618a39da5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:36.205323 1169 tablet_bootstrap.cc:654] T db99a1e68daf4e948a3d66ad276cc5f9 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.205950 1166 tablet_bootstrap.cc:654] T 71432a6848944a0da06ef9b81f160dc4 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.210824 1218 consensus_queue.cc:1048] T 0b21b7da1e2f47829c3f6cdaee689e8a P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251025 07:52:36.219033 442 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 509c94f2a422493ea4390ecee576db72 (table default.Table_3f081b7bb4e1420c8121ea4115439f3d [id=696a1285fbfe4038a8b1b38708c8ea49], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 509c94f2a422493ea4390ecee576db72 consensus state has no leader
I20251025 07:52:36.219697 822 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
I20251025 07:52:36.237946 1166 tablet_bootstrap.cc:492] T 71432a6848944a0da06ef9b81f160dc4 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:36.238286 1166 ts_tablet_manager.cc:1403] T 71432a6848944a0da06ef9b81f160dc4 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.001s
I20251025 07:52:36.239341 1166 raft_consensus.cc:359] T 71432a6848944a0da06ef9b81f160dc4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:36.239661 1166 raft_consensus.cc:385] T 71432a6848944a0da06ef9b81f160dc4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.239809 1166 raft_consensus.cc:740] T 71432a6848944a0da06ef9b81f160dc4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.240178 1166 consensus_queue.cc:260] T 71432a6848944a0da06ef9b81f160dc4 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:36.240935 1659 raft_consensus.cc:493] T c4610b0c62e741eca28c21f63cc3de81 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:36.241125 1659 raft_consensus.cc:515] T c4610b0c62e741eca28c21f63cc3de81 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.241715 1169 tablet_bootstrap.cc:492] T db99a1e68daf4e948a3d66ad276cc5f9 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:36.241895 1169 ts_tablet_manager.cc:1403] T db99a1e68daf4e948a3d66ad276cc5f9 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.044s user 0.017s sys 0.002s
I20251025 07:52:36.242568 1659 leader_election.cc:290] T c4610b0c62e741eca28c21f63cc3de81 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:36.242772 1167 tablet_bootstrap.cc:492] T 2c50ec95e48e49b59a76fac4fc862d3d P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:36.242949 1167 ts_tablet_manager.cc:1403] T 2c50ec95e48e49b59a76fac4fc862d3d P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.076s user 0.008s sys 0.000s
I20251025 07:52:36.243316 1169 raft_consensus.cc:359] T db99a1e68daf4e948a3d66ad276cc5f9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.243527 1169 raft_consensus.cc:385] T db99a1e68daf4e948a3d66ad276cc5f9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.243621 1169 raft_consensus.cc:740] T db99a1e68daf4e948a3d66ad276cc5f9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.244274 1167 raft_consensus.cc:359] T 2c50ec95e48e49b59a76fac4fc862d3d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.244578 1167 raft_consensus.cc:385] T 2c50ec95e48e49b59a76fac4fc862d3d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.244730 1167 raft_consensus.cc:740] T 2c50ec95e48e49b59a76fac4fc862d3d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.245044 1167 consensus_queue.cc:260] T 2c50ec95e48e49b59a76fac4fc862d3d P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.242743 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4610b0c62e741eca28c21f63cc3de81" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:36.246366 1169 consensus_queue.cc:260] T db99a1e68daf4e948a3d66ad276cc5f9 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.246469 903 raft_consensus.cc:2468] T c4610b0c62e741eca28c21f63cc3de81 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:36.246976 1169 ts_tablet_manager.cc:1434] T db99a1e68daf4e948a3d66ad276cc5f9 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20251025 07:52:36.246995 710 leader_election.cc:304] T c4610b0c62e741eca28c21f63cc3de81 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:36.247303 1169 tablet_bootstrap.cc:492] T c88c46e022ff41a58b02538f2151e260 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:36.249127 1169 tablet_bootstrap.cc:654] T c88c46e022ff41a58b02538f2151e260 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.251732 1166 ts_tablet_manager.cc:1434] T 71432a6848944a0da06ef9b81f160dc4 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20251025 07:52:36.251955 1659 raft_consensus.cc:2804] T c4610b0c62e741eca28c21f63cc3de81 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:36.252136 1659 raft_consensus.cc:493] T c4610b0c62e741eca28c21f63cc3de81 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:36.252530 1659 raft_consensus.cc:3060] T c4610b0c62e741eca28c21f63cc3de81 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:36.252823 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4610b0c62e741eca28c21f63cc3de81" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:36.253021 1045 raft_consensus.cc:2468] T c4610b0c62e741eca28c21f63cc3de81 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:36.255755 452 master_service.cc:954] ReplaceTablet: received request to replace tablet e47c4f741f354a19b49777bd06e54764 from {username='slave'} at 127.0.0.1:47610
I20251025 07:52:36.255810 451 master_service.cc:954] ReplaceTablet: received request to replace tablet 570201e7bf05406a97c81dde071d894b from {username='slave'} at 127.0.0.1:47610
I20251025 07:52:36.258283 450 master_service.cc:954] ReplaceTablet: received request to replace tablet 85fba8b37f2c4540aa81701052bf222a from {username='slave'} at 127.0.0.1:47610
I20251025 07:52:36.258898 1659 raft_consensus.cc:515] T c4610b0c62e741eca28c21f63cc3de81 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.259435 450 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_f32fe407bbe34695884f8ee432f0c947"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:36.260504 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4610b0c62e741eca28c21f63cc3de81" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:36.260703 1045 raft_consensus.cc:3060] T c4610b0c62e741eca28c21f63cc3de81 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:36.261443 450 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f32fe407bbe34695884f8ee432f0c947 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:36.261845 449 master_service.cc:954] ReplaceTablet: received request to replace tablet a7ab3150a35b4caca9119a1374d3f2f7 from {username='slave'} at 127.0.0.1:47610
I20251025 07:52:36.263312 448 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_aa4cf07c18a3424f91688e2a2ddc97b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:36.263957 448 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa4cf07c18a3424f91688e2a2ddc97b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:36.264662 1045 raft_consensus.cc:2468] T c4610b0c62e741eca28c21f63cc3de81 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:36.264606 447 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_ab4b1a5a7abb46fd978c91bb6d74840e" } new_table_name: "default.Table_92824b3f2a774564b3aea04aaa25d204" modify_external_catalogs: true
I20251025 07:52:36.265372 712 leader_election.cc:304] T c4610b0c62e741eca28c21f63cc3de81 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:36.265789 1829 raft_consensus.cc:2804] T c4610b0c62e741eca28c21f63cc3de81 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:36.265978 1829 raft_consensus.cc:697] T c4610b0c62e741eca28c21f63cc3de81 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:36.266505 1829 consensus_queue.cc:237] T c4610b0c62e741eca28c21f63cc3de81 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.266691 446 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_58b693b694cf4930bedf9f4cc6c52c35"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:36.267267 446 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58b693b694cf4930bedf9f4cc6c52c35 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:36.259924 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4610b0c62e741eca28c21f63cc3de81" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:36.273119 903 raft_consensus.cc:3060] T c4610b0c62e741eca28c21f63cc3de81 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:36.273136 455 master_service.cc:954] ReplaceTablet: received request to replace tablet 04449cc97f404a19b6a1b20dea074f95 from {username='slave'} at 127.0.0.1:47610
I20251025 07:52:36.276618 903 raft_consensus.cc:2468] T c4610b0c62e741eca28c21f63cc3de81 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:36.272617 1218 consensus_queue.cc:1048] T 1d261e432cf5403ea162129706bc4d4d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:36.278384 1659 leader_election.cc:290] T c4610b0c62e741eca28c21f63cc3de81 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:36.280673 1659 consensus_queue.cc:1048] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:36.289395 1169 tablet_bootstrap.cc:492] T c88c46e022ff41a58b02538f2151e260 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:36.289623 1169 ts_tablet_manager.cc:1403] T c88c46e022ff41a58b02538f2151e260 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.003s
I20251025 07:52:36.290621 1169 raft_consensus.cc:359] T c88c46e022ff41a58b02538f2151e260 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.291844 1169 raft_consensus.cc:385] T c88c46e022ff41a58b02538f2151e260 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.291934 1169 raft_consensus.cc:740] T c88c46e022ff41a58b02538f2151e260 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.292232 1169 consensus_queue.cc:260] T c88c46e022ff41a58b02538f2151e260 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.294618 1860 consensus_queue.cc:1048] T 1d261e432cf5403ea162129706bc4d4d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:36.299284 960 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
I20251025 07:52:36.301072 1169 ts_tablet_manager.cc:1434] T c88c46e022ff41a58b02538f2151e260 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251025 07:52:36.301481 1169 tablet_bootstrap.cc:492] T 896bd7870d87408e8adfdb7a78998111 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:36.304517 1169 tablet_bootstrap.cc:654] T 896bd7870d87408e8adfdb7a78998111 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.305794 1929 tablet_replica.cc:333] T c4610b0c62e741eca28c21f63cc3de81 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:36.306054 1929 raft_consensus.cc:2243] T c4610b0c62e741eca28c21f63cc3de81 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:36.306325 1929 raft_consensus.cc:2272] T c4610b0c62e741eca28c21f63cc3de81 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:36.308784 1829 consensus_queue.cc:1048] T 4f25c9caa05b4da4bd56a8a21fde423c P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20251025 07:52:36.311236 454 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_ab6798879a554075b3deaa059ad077c7" } new_table_name: "default.Table_bd82d871f84943f4baa967f5c0ec74a3" modify_external_catalogs: true
W20251025 07:52:36.315827 1830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251025 07:52:36.316354 1830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251025 07:52:36.316567 1830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20251025 07:52:36.319901 1612 consensus_queue.cc:1048] T f0bdaa8cfd764f1b83f47881353d26b6 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251025 07:52:36.323040 1829 consensus_queue.cc:1048] T 8ff73ffc42434f74a4949f156771d5a4 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:36.326383 1829 consensus_queue.cc:1048] T 8ff73ffc42434f74a4949f156771d5a4 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:36.335282 1829 consensus_queue.cc:1048] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251025 07:52:36.337956 1830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251025 07:52:36.338162 1830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251025 07:52:36.338308 1830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251025 07:52:36.338436 1830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251025 07:52:36.344259 1169 tablet_bootstrap.cc:492] T 896bd7870d87408e8adfdb7a78998111 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:36.344489 1169 ts_tablet_manager.cc:1403] T 896bd7870d87408e8adfdb7a78998111 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.043s user 0.006s sys 0.001s
I20251025 07:52:36.345841 1169 raft_consensus.cc:359] T 896bd7870d87408e8adfdb7a78998111 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.346045 1169 raft_consensus.cc:385] T 896bd7870d87408e8adfdb7a78998111 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.346169 1169 raft_consensus.cc:740] T 896bd7870d87408e8adfdb7a78998111 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.346472 1169 consensus_queue.cc:260] T 896bd7870d87408e8adfdb7a78998111 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.347155 1169 ts_tablet_manager.cc:1434] T 896bd7870d87408e8adfdb7a78998111 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20251025 07:52:36.347494 1169 tablet_bootstrap.cc:492] T 9d6a9937d10a4cc2813671dd1bfa1664 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:36.349612 1169 tablet_bootstrap.cc:654] T 9d6a9937d10a4cc2813671dd1bfa1664 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.352109 1167 ts_tablet_manager.cc:1434] T 2c50ec95e48e49b59a76fac4fc862d3d P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.048s user 0.002s sys 0.000s
I20251025 07:52:36.352847 1167 tablet_bootstrap.cc:492] T 70fe9aa2f58d4abea3b3b71066a29653 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:36.356353 1166 tablet_bootstrap.cc:492] T f3001314b56345f5a70314777cae0125 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:36.360265 1830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:40091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:37421,127.0.104.61:42203,127.0.104.60:37611
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:42203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=45311
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 25 Oct 2025 07:43:22 UTC on 24a791456cd2
build id 8708
ASAN enabled
I20251025 07:52:36.363369 1830 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251025 07:52:36.364760 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 2209089667dd4937abd96615f6929449 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:36.366036 1830 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251025 07:52:36.367084 1929 ts_tablet_manager.cc:1916] T c4610b0c62e741eca28c21f63cc3de81 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:36.371783 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 3775a2e08d4e40bd9f57e4cda759d535 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:36.375448 1569 consensus_queue.cc:1048] T f0bdaa8cfd764f1b83f47881353d26b6 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:36.382251 1167 tablet_bootstrap.cc:654] T 70fe9aa2f58d4abea3b3b71066a29653 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:36.391870 1954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 07:52:36.395042 1952 tablet_replica.cc:333] T 2209089667dd4937abd96615f6929449 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:36.395515 1952 raft_consensus.cc:2243] T 2209089667dd4937abd96615f6929449 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:36.396147 1952 raft_consensus.cc:2272] T 2209089667dd4937abd96615f6929449 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:36.400950 1956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251025 07:52:36.408432 1950 log.cc:927] Time spent T 0b21b7da1e2f47829c3f6cdaee689e8a P 3aea2869ff814cc7b142012faa274382: Append to log took a long time: real 0.050s user 0.001s sys 0.000s
W20251025 07:52:36.415501 1961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251025 07:52:36.417054 1860 consensus_queue.cc:1048] T 0b21b7da1e2f47829c3f6cdaee689e8a P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20251025 07:52:36.417809 1830 server_base.cc:1047] running on GCE node
I20251025 07:52:36.419170 1830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251025 07:52:36.423172 1166 tablet_bootstrap.cc:654] T f3001314b56345f5a70314777cae0125 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.425730 1830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251025 07:52:36.426023 1218 consensus_queue.cc:1048] T 87268e27540c4ec58ce1e62df819f19d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:36.428558 1167 tablet_bootstrap.cc:492] T 70fe9aa2f58d4abea3b3b71066a29653 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:36.428728 1167 ts_tablet_manager.cc:1403] T 70fe9aa2f58d4abea3b3b71066a29653 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.076s user 0.004s sys 0.000s
I20251025 07:52:36.429261 1830 hybrid_clock.cc:648] HybridClock initialized: now 1761378756429158 us; error 91 us; skew 500 ppm
I20251025 07:52:36.429749 1830 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251025 07:52:36.404918 1952 ts_tablet_manager.cc:1916] T 2209089667dd4937abd96615f6929449 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:36.435791 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 2209089667dd4937abd96615f6929449 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:36.435842 875 tablet_service.cc:1552] Processing DeleteTablet for tablet 3775a2e08d4e40bd9f57e4cda759d535 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:36.436488 1966 tablet_replica.cc:333] T 2209089667dd4937abd96615f6929449 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:36.437289 1966 raft_consensus.cc:2243] T 2209089667dd4937abd96615f6929449 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:36.437887 1966 raft_consensus.cc:2272] T 2209089667dd4937abd96615f6929449 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:36.449271 1561 consensus_queue.cc:1048] T 87268e27540c4ec58ce1e62df819f19d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:36.459113 1929 ts_tablet_manager.cc:1929] T c4610b0c62e741eca28c21f63cc3de81 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:36.459291 1929 log.cc:1199] T c4610b0c62e741eca28c21f63cc3de81 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/c4610b0c62e741eca28c21f63cc3de81
I20251025 07:52:36.459731 1929 ts_tablet_manager.cc:1950] T c4610b0c62e741eca28c21f63cc3de81 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:36.463243 1169 tablet_bootstrap.cc:492] T 9d6a9937d10a4cc2813671dd1bfa1664 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:36.463431 1169 ts_tablet_manager.cc:1403] T 9d6a9937d10a4cc2813671dd1bfa1664 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.116s user 0.017s sys 0.019s
I20251025 07:52:36.464584 1169 raft_consensus.cc:359] T 9d6a9937d10a4cc2813671dd1bfa1664 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:36.464869 1169 raft_consensus.cc:385] T 9d6a9937d10a4cc2813671dd1bfa1664 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.464967 1169 raft_consensus.cc:740] T 9d6a9937d10a4cc2813671dd1bfa1664 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.465317 1169 consensus_queue.cc:260] T 9d6a9937d10a4cc2813671dd1bfa1664 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:36.466447 1169 ts_tablet_manager.cc:1434] T 9d6a9937d10a4cc2813671dd1bfa1664 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20251025 07:52:36.467934 1929 tablet_replica.cc:333] T 2209089667dd4937abd96615f6929449 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:36.468657 1929 raft_consensus.cc:2243] T 2209089667dd4937abd96615f6929449 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:36.468876 1929 raft_consensus.cc:2272] T 2209089667dd4937abd96615f6929449 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:36.469108 1169 tablet_bootstrap.cc:492] T b97e39aaac3048ed93dbfe873b757254 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:36.470093 1966 ts_tablet_manager.cc:1916] T 2209089667dd4937abd96615f6929449 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:36.470975 1169 tablet_bootstrap.cc:654] T b97e39aaac3048ed93dbfe873b757254 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.468643 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet c4610b0c62e741eca28c21f63cc3de81 (table default.Table_c9bc68f9b5c54a33b854f9ee8a1ad311 [id=d8f616c04cc54b86af0af3cc073c5e2d]) successfully deleted
I20251025 07:52:36.473361 1659 raft_consensus.cc:493] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:36.473675 1659 raft_consensus.cc:515] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:36.429396 1167 raft_consensus.cc:359] T 70fe9aa2f58d4abea3b3b71066a29653 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.477770 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d6a9937d10a4cc2813671dd1bfa1664" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:36.478912 1166 tablet_bootstrap.cc:492] T f3001314b56345f5a70314777cae0125 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:36.479162 1830 webserver.cc:492] Webserver started at http://127.0.104.61:45311/ using document root <none> and password file <none>
I20251025 07:52:36.480057 1166 ts_tablet_manager.cc:1403] T f3001314b56345f5a70314777cae0125 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.123s user 0.005s sys 0.000s
I20251025 07:52:36.481096 1166 raft_consensus.cc:359] T f3001314b56345f5a70314777cae0125 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.481312 1166 raft_consensus.cc:385] T f3001314b56345f5a70314777cae0125 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.481397 1166 raft_consensus.cc:740] T f3001314b56345f5a70314777cae0125 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.481743 1166 consensus_queue.cc:260] T f3001314b56345f5a70314777cae0125 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.482266 1830 fs_manager.cc:362] Metadata directory not provided
I20251025 07:52:36.482445 1166 ts_tablet_manager.cc:1434] T f3001314b56345f5a70314777cae0125 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:36.482708 1830 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251025 07:52:36.484233 1929 ts_tablet_manager.cc:1916] T 2209089667dd4937abd96615f6929449 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:36.497252 1966 ts_tablet_manager.cc:1929] T 2209089667dd4937abd96615f6929449 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:36.497409 1966 log.cc:1199] T 2209089667dd4937abd96615f6929449 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/2209089667dd4937abd96615f6929449
I20251025 07:52:36.498095 1966 ts_tablet_manager.cc:1950] T 2209089667dd4937abd96615f6929449 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:36.498124 1929 ts_tablet_manager.cc:1929] T 2209089667dd4937abd96615f6929449 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:36.498338 1929 log.cc:1199] T 2209089667dd4937abd96615f6929449 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/2209089667dd4937abd96615f6929449
I20251025 07:52:36.498728 1929 ts_tablet_manager.cc:1950] T 2209089667dd4937abd96615f6929449 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:36.497340 1830 fs_manager.cc:714] Time spent opening directory manager: real 0.009s user 0.004s sys 0.000s
I20251025 07:52:36.499660 1929 tablet_replica.cc:333] T cb824bd780524832b2b60f5618a39da5 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:36.499856 1929 raft_consensus.cc:2243] T cb824bd780524832b2b60f5618a39da5 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:36.500027 1929 raft_consensus.cc:2272] T cb824bd780524832b2b60f5618a39da5 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:36.502485 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 2209089667dd4937abd96615f6929449 (table default.Table_692ab786006945128bee1c8578c2a0cd [id=1780f0b073574d5dad0e1489d1f777e2]) successfully deleted
I20251025 07:52:36.504091 1167 raft_consensus.cc:385] T 70fe9aa2f58d4abea3b3b71066a29653 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.504115 1659 leader_election.cc:290] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:36.504274 1167 raft_consensus.cc:740] T 70fe9aa2f58d4abea3b3b71066a29653 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.504616 1167 consensus_queue.cc:260] T 70fe9aa2f58d4abea3b3b71066a29653 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.509368 1952 ts_tablet_manager.cc:1929] T 2209089667dd4937abd96615f6929449 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:36.511907 1976 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251025 07:52:36.512605 1166 tablet_bootstrap.cc:492] T be5ab0db6ac34c2c9a370445e251eb46 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:36.512993 1929 ts_tablet_manager.cc:1916] T cb824bd780524832b2b60f5618a39da5 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:36.514410 1166 tablet_bootstrap.cc:654] T be5ab0db6ac34c2c9a370445e251eb46 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.515467 1966 tablet_replica.cc:333] T 3775a2e08d4e40bd9f57e4cda759d535 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:36.515667 1966 raft_consensus.cc:2243] T 3775a2e08d4e40bd9f57e4cda759d535 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:36.515842 1966 raft_consensus.cc:2272] T 3775a2e08d4e40bd9f57e4cda759d535 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:36.516897 1966 ts_tablet_manager.cc:1916] T 3775a2e08d4e40bd9f57e4cda759d535 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:36.517511 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d6a9937d10a4cc2813671dd1bfa1664" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:36.517771 1044 raft_consensus.cc:2468] T 9d6a9937d10a4cc2813671dd1bfa1664 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:36.519079 1830 fs_manager.cc:730] Time spent opening block manager: real 0.017s user 0.002s sys 0.004s
I20251025 07:52:36.519675 1830 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/data,/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/wal
uuid: "8603bddd8a084ff7a73a25641768dfa9"
format_stamp: "Formatted at 2025-10-25 07:52:21 on dist-test-slave-tm48"
I20251025 07:52:36.522071 1830 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251025 07:52:36.515937 1829 raft_consensus.cc:493] T 07898860aa594a05a211fc89768cb76d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:36.526240 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 2209089667dd4937abd96615f6929449 (table default.Table_692ab786006945128bee1c8578c2a0cd [id=1780f0b073574d5dad0e1489d1f777e2]) successfully deleted
I20251025 07:52:36.526465 1829 raft_consensus.cc:515] T 07898860aa594a05a211fc89768cb76d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:36.527285 1829 leader_election.cc:290] T 07898860aa594a05a211fc89768cb76d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:36.527320 1166 tablet_bootstrap.cc:492] T be5ab0db6ac34c2c9a370445e251eb46 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:36.527446 1829 raft_consensus.cc:493] T abf94c32ff2945119b710905c075d63c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:36.527493 1166 ts_tablet_manager.cc:1403] T be5ab0db6ac34c2c9a370445e251eb46 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.002s
I20251025 07:52:36.527657 1966 ts_tablet_manager.cc:1929] T 3775a2e08d4e40bd9f57e4cda759d535 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:36.527642 1829 raft_consensus.cc:515] T abf94c32ff2945119b710905c075d63c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.528297 1966 log.cc:1199] T 3775a2e08d4e40bd9f57e4cda759d535 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/3775a2e08d4e40bd9f57e4cda759d535
I20251025 07:52:36.528601 1829 leader_election.cc:290] T abf94c32ff2945119b710905c075d63c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:36.528861 1952 log.cc:1199] T 2209089667dd4937abd96615f6929449 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/2209089667dd4937abd96615f6929449
I20251025 07:52:36.528977 1166 raft_consensus.cc:359] T be5ab0db6ac34c2c9a370445e251eb46 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.529150 1166 raft_consensus.cc:385] T be5ab0db6ac34c2c9a370445e251eb46 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.529263 1166 raft_consensus.cc:740] T be5ab0db6ac34c2c9a370445e251eb46 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.529448 1569 consensus_queue.cc:1048] T 8956ad28b35d45f28c683ac055bba441 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:36.529490 1166 consensus_queue.cc:260] T be5ab0db6ac34c2c9a370445e251eb46 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.529680 1952 ts_tablet_manager.cc:1950] T 2209089667dd4937abd96615f6929449 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:36.530104 1166 ts_tablet_manager.cc:1434] T be5ab0db6ac34c2c9a370445e251eb46 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:36.530457 1166 tablet_bootstrap.cc:492] T 9979cae195e849908677d78c6c33be71 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:36.516906 1167 ts_tablet_manager.cc:1434] T 70fe9aa2f58d4abea3b3b71066a29653 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.088s user 0.003s sys 0.000s
I20251025 07:52:36.518299 712 leader_election.cc:304] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:36.519917 1441 raft_consensus.cc:493] T 2c50ec95e48e49b59a76fac4fc862d3d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:36.531450 1441 raft_consensus.cc:515] T 2c50ec95e48e49b59a76fac4fc862d3d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.532109 1441 leader_election.cc:290] T 2c50ec95e48e49b59a76fac4fc862d3d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:36.532308 1441 raft_consensus.cc:2804] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:36.532416 1441 raft_consensus.cc:493] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:36.532559 1441 raft_consensus.cc:3060] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:36.532615 1166 tablet_bootstrap.cc:654] T 9979cae195e849908677d78c6c33be71 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.534586 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07898860aa594a05a211fc89768cb76d" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:36.534726 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "abf94c32ff2945119b710905c075d63c" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:36.539089 1441 raft_consensus.cc:515] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:36.541045 1441 leader_election.cc:290] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:36.543821 1966 ts_tablet_manager.cc:1950] T 3775a2e08d4e40bd9f57e4cda759d535 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:36.544883 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 3775a2e08d4e40bd9f57e4cda759d535 (table default.Table_37ad28b796a342f3a6e962ead289ec5b [id=a4d25738410b4a168aa0c76094daf69b]) successfully deleted
W20251025 07:52:36.549667 712 leader_election.cc:343] T 07898860aa594a05a211fc89768cb76d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:36.550022 712 leader_election.cc:343] T abf94c32ff2945119b710905c075d63c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:36.550977 1659 raft_consensus.cc:493] T 53198283718d4b45b30df65b5bde96a6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:36.551311 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c50ec95e48e49b59a76fac4fc862d3d" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:36.551586 1659 raft_consensus.cc:515] T 53198283718d4b45b30df65b5bde96a6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.552271 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d6a9937d10a4cc2813671dd1bfa1664" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:36.552461 1049 raft_consensus.cc:3060] T 9d6a9937d10a4cc2813671dd1bfa1664 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:36.552497 1659 leader_election.cc:290] T 53198283718d4b45b30df65b5bde96a6 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:36.555714 1049 raft_consensus.cc:2468] T 9d6a9937d10a4cc2813671dd1bfa1664 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:36.557766 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53198283718d4b45b30df65b5bde96a6" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:36.558552 712 leader_election.cc:343] T 2c50ec95e48e49b59a76fac4fc862d3d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:36.558966 712 leader_election.cc:304] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:36.559387 910 raft_consensus.cc:3060] T 71432a6848944a0da06ef9b81f160dc4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:36.560961 712 leader_election.cc:343] T 53198283718d4b45b30df65b5bde96a6 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:36.561072 1612 consensus_queue.cc:1048] T 8956ad28b35d45f28c683ac055bba441 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:36.562647 1659 raft_consensus.cc:2804] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:36.562795 1659 raft_consensus.cc:697] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:36.563150 1659 consensus_queue.cc:237] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:36.565071 1612 consensus_queue.cc:1048] T 71432a6848944a0da06ef9b81f160dc4 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:36.566236 1166 tablet_bootstrap.cc:492] T 9979cae195e849908677d78c6c33be71 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:36.566411 1166 ts_tablet_manager.cc:1403] T 9979cae195e849908677d78c6c33be71 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.036s user 0.007s sys 0.007s
I20251025 07:52:36.567170 1166 raft_consensus.cc:359] T 9979cae195e849908677d78c6c33be71 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.567371 1166 raft_consensus.cc:385] T 9979cae195e849908677d78c6c33be71 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.567445 1166 raft_consensus.cc:740] T 9979cae195e849908677d78c6c33be71 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.567688 1166 consensus_queue.cc:260] T 9979cae195e849908677d78c6c33be71 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.568203 1166 ts_tablet_manager.cc:1434] T 9979cae195e849908677d78c6c33be71 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 07:52:36.568519 1166 tablet_bootstrap.cc:492] T c88c46e022ff41a58b02538f2151e260 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:36.576493 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 2209089667dd4937abd96615f6929449 (table default.Table_692ab786006945128bee1c8578c2a0cd [id=1780f0b073574d5dad0e1489d1f777e2]) successfully deleted
I20251025 07:52:36.576858 1167 tablet_bootstrap.cc:492] T 6ec421eb820b4f40afcff5a5bf80c508 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:36.579473 1167 tablet_bootstrap.cc:654] T 6ec421eb820b4f40afcff5a5bf80c508 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.579613 1166 tablet_bootstrap.cc:654] T c88c46e022ff41a58b02538f2151e260 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.590021 1929 ts_tablet_manager.cc:1929] T cb824bd780524832b2b60f5618a39da5 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:36.590085 1561 raft_consensus.cc:493] T 9979cae195e849908677d78c6c33be71 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:36.590255 1561 raft_consensus.cc:515] T 9979cae195e849908677d78c6c33be71 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.591545 1561 leader_election.cc:290] T 9979cae195e849908677d78c6c33be71 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:36.591979 1166 tablet_bootstrap.cc:492] T c88c46e022ff41a58b02538f2151e260 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:36.592152 1166 ts_tablet_manager.cc:1403] T c88c46e022ff41a58b02538f2151e260 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.000s
I20251025 07:52:36.592914 1166 raft_consensus.cc:359] T c88c46e022ff41a58b02538f2151e260 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.593094 1166 raft_consensus.cc:385] T c88c46e022ff41a58b02538f2151e260 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.593207 1166 raft_consensus.cc:740] T c88c46e022ff41a58b02538f2151e260 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.593757 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9979cae195e849908677d78c6c33be71" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:36.593997 776 raft_consensus.cc:2468] T 9979cae195e849908677d78c6c33be71 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:36.594821 1929 log.cc:1199] T cb824bd780524832b2b60f5618a39da5 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/cb824bd780524832b2b60f5618a39da5
I20251025 07:52:36.592224 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9979cae195e849908677d78c6c33be71" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:36.596447 850 leader_election.cc:343] T 9979cae195e849908677d78c6c33be71 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:36.597419 1612 consensus_queue.cc:1048] T 71432a6848944a0da06ef9b81f160dc4 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:36.598657 1166 consensus_queue.cc:260] T c88c46e022ff41a58b02538f2151e260 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.599032 1929 ts_tablet_manager.cc:1950] T cb824bd780524832b2b60f5618a39da5 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:36.600095 1166 ts_tablet_manager.cc:1434] T c88c46e022ff41a58b02538f2151e260 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.008s user 0.002s sys 0.001s
I20251025 07:52:36.600400 1166 tablet_bootstrap.cc:492] T f170424be2f643e7af499834955472b0 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:36.602010 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet cb824bd780524832b2b60f5618a39da5 (table default.Table_c1743cb60e4d45939f387235c1726ef3 [id=18e29b94f8a6443fb8e6a876119fd9a3]) successfully deleted
I20251025 07:52:36.602154 1166 tablet_bootstrap.cc:654] T f170424be2f643e7af499834955472b0 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.603463 847 leader_election.cc:304] T 9979cae195e849908677d78c6c33be71 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:36.603871 1561 raft_consensus.cc:2804] T 9979cae195e849908677d78c6c33be71 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:36.603981 1561 raft_consensus.cc:493] T 9979cae195e849908677d78c6c33be71 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:36.604063 1561 raft_consensus.cc:3060] T 9979cae195e849908677d78c6c33be71 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:36.607975 1561 raft_consensus.cc:515] T 9979cae195e849908677d78c6c33be71 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.609436 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9979cae195e849908677d78c6c33be71" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:36.610124 850 leader_election.cc:343] T 9979cae195e849908677d78c6c33be71 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:36.610426 1561 leader_election.cc:290] T 9979cae195e849908677d78c6c33be71 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:36.611878 1952 tablet_replica.cc:333] T 3775a2e08d4e40bd9f57e4cda759d535 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:36.612267 1952 raft_consensus.cc:2243] T 3775a2e08d4e40bd9f57e4cda759d535 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:36.612501 1952 raft_consensus.cc:2272] T 3775a2e08d4e40bd9f57e4cda759d535 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:36.615192 1166 tablet_bootstrap.cc:492] T f170424be2f643e7af499834955472b0 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:36.615363 1166 ts_tablet_manager.cc:1403] T f170424be2f643e7af499834955472b0 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.002s
I20251025 07:52:36.616150 1166 raft_consensus.cc:359] T f170424be2f643e7af499834955472b0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.616361 1166 raft_consensus.cc:385] T f170424be2f643e7af499834955472b0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.616569 1166 raft_consensus.cc:740] T f170424be2f643e7af499834955472b0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.617322 1167 tablet_bootstrap.cc:492] T 6ec421eb820b4f40afcff5a5bf80c508 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:36.617460 1167 ts_tablet_manager.cc:1403] T 6ec421eb820b4f40afcff5a5bf80c508 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.000s
W20251025 07:52:36.616225 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:43563, user_credentials={real_user=slave}} blocked reactor thread for 118108us
I20251025 07:52:36.616829 1166 consensus_queue.cc:260] T f170424be2f643e7af499834955472b0 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.618374 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9979cae195e849908677d78c6c33be71" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:36.618667 776 raft_consensus.cc:3060] T 9979cae195e849908677d78c6c33be71 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:36.618795 1166 ts_tablet_manager.cc:1434] T f170424be2f643e7af499834955472b0 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251025 07:52:36.619108 1166 tablet_bootstrap.cc:492] T 9d6a9937d10a4cc2813671dd1bfa1664 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
W20251025 07:52:36.618454 710 leader_election.cc:343] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:36.621340 1166 tablet_bootstrap.cc:654] T 9d6a9937d10a4cc2813671dd1bfa1664 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.627624 1169 tablet_bootstrap.cc:492] T b97e39aaac3048ed93dbfe873b757254 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:36.627815 1169 ts_tablet_manager.cc:1403] T b97e39aaac3048ed93dbfe873b757254 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.161s user 0.013s sys 0.013s
I20251025 07:52:36.628682 1169 raft_consensus.cc:359] T b97e39aaac3048ed93dbfe873b757254 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.628877 1169 raft_consensus.cc:385] T b97e39aaac3048ed93dbfe873b757254 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.628975 1169 raft_consensus.cc:740] T b97e39aaac3048ed93dbfe873b757254 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.629297 1952 ts_tablet_manager.cc:1916] T 3775a2e08d4e40bd9f57e4cda759d535 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:36.629248 1169 consensus_queue.cc:260] T b97e39aaac3048ed93dbfe873b757254 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.631783 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d6a9937d10a4cc2813671dd1bfa1664" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:36.631927 1973 raft_consensus.cc:493] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:36.632114 1973 raft_consensus.cc:515] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.632841 1973 leader_election.cc:290] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:36.633040 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53198283718d4b45b30df65b5bde96a6" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:36.634011 710 leader_election.cc:343] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:36.633996 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f170424be2f643e7af499834955472b0" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:36.634306 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07898860aa594a05a211fc89768cb76d" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:36.634518 712 leader_election.cc:343] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:36.634600 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "abf94c32ff2945119b710905c075d63c" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:36.634577 1973 consensus_queue.cc:1048] T 4f25c9caa05b4da4bd56a8a21fde423c P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:36.634832 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c50ec95e48e49b59a76fac4fc862d3d" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:36.635154 710 leader_election.cc:343] T 53198283718d4b45b30df65b5bde96a6 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:36.635334 710 leader_election.cc:304] T 53198283718d4b45b30df65b5bde96a6 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:36.635586 1973 raft_consensus.cc:2749] T 53198283718d4b45b30df65b5bde96a6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:36.638329 776 raft_consensus.cc:2468] T 9979cae195e849908677d78c6c33be71 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:36.641404 1169 ts_tablet_manager.cc:1434] T b97e39aaac3048ed93dbfe873b757254 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251025 07:52:36.642899 1169 tablet_bootstrap.cc:492] T f170424be2f643e7af499834955472b0 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:36.618206 1167 raft_consensus.cc:359] T 6ec421eb820b4f40afcff5a5bf80c508 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.643781 1167 raft_consensus.cc:385] T 6ec421eb820b4f40afcff5a5bf80c508 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.643920 1167 raft_consensus.cc:740] T 6ec421eb820b4f40afcff5a5bf80c508 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.644223 1167 consensus_queue.cc:260] T 6ec421eb820b4f40afcff5a5bf80c508 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.644810 1167 ts_tablet_manager.cc:1434] T 6ec421eb820b4f40afcff5a5bf80c508 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.027s user 0.000s sys 0.002s
I20251025 07:52:36.645627 1167 tablet_bootstrap.cc:492] T 551d07a9242740dcb9366bb2d707a4a8 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
W20251025 07:52:36.646574 710 leader_election.cc:343] T 07898860aa594a05a211fc89768cb76d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:36.646816 710 leader_election.cc:304] T 07898860aa594a05a211fc89768cb76d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:36.647611 1659 raft_consensus.cc:2749] T 07898860aa594a05a211fc89768cb76d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:36.647634 710 leader_election.cc:343] T abf94c32ff2945119b710905c075d63c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:36.648049 710 leader_election.cc:304] T abf94c32ff2945119b710905c075d63c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:36.648437 1167 tablet_bootstrap.cc:654] T 551d07a9242740dcb9366bb2d707a4a8 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:36.648777 710 leader_election.cc:343] T 2c50ec95e48e49b59a76fac4fc862d3d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:36.649144 710 leader_election.cc:304] T 2c50ec95e48e49b59a76fac4fc862d3d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:36.649583 1980 raft_consensus.cc:2749] T 2c50ec95e48e49b59a76fac4fc862d3d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:36.649003 1659 raft_consensus.cc:2749] T abf94c32ff2945119b710905c075d63c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:36.650028 1169 tablet_bootstrap.cc:654] T f170424be2f643e7af499834955472b0 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.650259 847 leader_election.cc:304] T 9979cae195e849908677d78c6c33be71 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:36.651633 1561 raft_consensus.cc:2804] T 9979cae195e849908677d78c6c33be71 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:36.651779 1561 raft_consensus.cc:697] T 9979cae195e849908677d78c6c33be71 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:36.652073 1561 consensus_queue.cc:237] T 9979cae195e849908677d78c6c33be71 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.668287 1166 tablet_bootstrap.cc:492] T 9d6a9937d10a4cc2813671dd1bfa1664 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:36.668488 1166 ts_tablet_manager.cc:1403] T 9d6a9937d10a4cc2813671dd1bfa1664 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.049s user 0.000s sys 0.009s
I20251025 07:52:36.669130 1659 raft_consensus.cc:493] T c88c46e022ff41a58b02538f2151e260 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:36.669252 1166 raft_consensus.cc:359] T 9d6a9937d10a4cc2813671dd1bfa1664 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:36.669324 1659 raft_consensus.cc:515] T c88c46e022ff41a58b02538f2151e260 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.669413 1166 raft_consensus.cc:385] T 9d6a9937d10a4cc2813671dd1bfa1664 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.669500 1166 raft_consensus.cc:740] T 9d6a9937d10a4cc2813671dd1bfa1664 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.669751 1166 consensus_queue.cc:260] T 9d6a9937d10a4cc2813671dd1bfa1664 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:36.669966 1659 leader_election.cc:290] T c88c46e022ff41a58b02538f2151e260 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:36.670846 1166 ts_tablet_manager.cc:1434] T 9d6a9937d10a4cc2813671dd1bfa1664 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251025 07:52:36.671216 1166 tablet_bootstrap.cc:492] T 07898860aa594a05a211fc89768cb76d P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:36.673314 1166 tablet_bootstrap.cc:654] T 07898860aa594a05a211fc89768cb76d P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.673296 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c88c46e022ff41a58b02538f2151e260" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:36.673552 1045 raft_consensus.cc:2468] T c88c46e022ff41a58b02538f2151e260 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:36.673960 712 leader_election.cc:304] T c88c46e022ff41a58b02538f2151e260 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:36.674341 1670 raft_consensus.cc:2804] T c88c46e022ff41a58b02538f2151e260 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:36.674446 1670 raft_consensus.cc:493] T c88c46e022ff41a58b02538f2151e260 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:36.674557 1670 raft_consensus.cc:3060] T c88c46e022ff41a58b02538f2151e260 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:36.675418 1167 tablet_bootstrap.cc:492] T 551d07a9242740dcb9366bb2d707a4a8 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:36.675590 1167 ts_tablet_manager.cc:1403] T 551d07a9242740dcb9366bb2d707a4a8 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.030s user 0.015s sys 0.000s
I20251025 07:52:36.676445 1167 raft_consensus.cc:359] T 551d07a9242740dcb9366bb2d707a4a8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.676651 1167 raft_consensus.cc:385] T 551d07a9242740dcb9366bb2d707a4a8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.676728 1167 raft_consensus.cc:740] T 551d07a9242740dcb9366bb2d707a4a8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.676999 1167 consensus_queue.cc:260] T 551d07a9242740dcb9366bb2d707a4a8 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.677598 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f170424be2f643e7af499834955472b0" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:36.677691 1670 raft_consensus.cc:515] T c88c46e022ff41a58b02538f2151e260 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.677816 902 raft_consensus.cc:2468] T f170424be2f643e7af499834955472b0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:36.678748 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c88c46e022ff41a58b02538f2151e260" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:36.678968 910 raft_consensus.cc:2468] T c88c46e022ff41a58b02538f2151e260 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:36.679265 1830 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251025 07:52:36.679724 710 leader_election.cc:304] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:36.680166 1980 raft_consensus.cc:2804] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:36.680316 1980 raft_consensus.cc:493] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:36.680409 1980 raft_consensus.cc:3060] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:36.681231 1830 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251025 07:52:36.681617 1830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251025 07:52:36.684325 1166 tablet_bootstrap.cc:492] T 07898860aa594a05a211fc89768cb76d P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:36.684502 1166 ts_tablet_manager.cc:1403] T 07898860aa594a05a211fc89768cb76d P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20251025 07:52:36.684549 1167 ts_tablet_manager.cc:1434] T 551d07a9242740dcb9366bb2d707a4a8 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20251025 07:52:36.684765 1980 raft_consensus.cc:515] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.685452 451 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet 570201e7bf05406a97c81dde071d894b
I20251025 07:52:36.684927 1167 tablet_bootstrap.cc:492] T d66ad049d5484228a268b0facb00ecf1 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:36.685392 1670 leader_election.cc:290] T c88c46e022ff41a58b02538f2151e260 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:36.686223 451 catalog_manager.cc:6719] ReplaceTablet: tablet 570201e7bf05406a97c81dde071d894b deleted and replaced by tablet 3a0f50c8000144a4acb5219484b01800
I20251025 07:52:36.686345 1011 tablet_service.cc:1552] Processing DeleteTablet for tablet 570201e7bf05406a97c81dde071d894b with delete_type TABLET_DATA_DELETED (replaced by tablet 3a0f50c8000144a4acb5219484b01800) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:36.686682 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 570201e7bf05406a97c81dde071d894b with delete_type TABLET_DATA_DELETED (replaced by tablet 3a0f50c8000144a4acb5219484b01800) from {username='slave'} at 127.0.0.1:58412
W20251025 07:52:36.687286 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 570201e7bf05406a97c81dde071d894b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 570201e7bf05406a97c81dde071d894b already in progress: creating tablet
I20251025 07:52:36.687757 1980 leader_election.cc:290] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:36.687696 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f170424be2f643e7af499834955472b0" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:36.688220 712 leader_election.cc:343] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:36.690353 452 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet e47c4f741f354a19b49777bd06e54764
I20251025 07:52:36.690289 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c88c46e022ff41a58b02538f2151e260" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:36.690829 452 catalog_manager.cc:6719] ReplaceTablet: tablet e47c4f741f354a19b49777bd06e54764 deleted and replaced by tablet 755f0d6c87044e20a8020f8410940123
I20251025 07:52:36.690577 1045 raft_consensus.cc:3060] T c88c46e022ff41a58b02538f2151e260 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:36.691092 748 tablet_service.cc:1552] Processing DeleteTablet for tablet e47c4f741f354a19b49777bd06e54764 with delete_type TABLET_DATA_DELETED (replaced by tablet 755f0d6c87044e20a8020f8410940123) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:36.691186 1011 tablet_service.cc:1552] Processing DeleteTablet for tablet e47c4f741f354a19b49777bd06e54764 with delete_type TABLET_DATA_DELETED (replaced by tablet 755f0d6c87044e20a8020f8410940123) from {username='slave'} at 127.0.0.1:44572
W20251025 07:52:36.691839 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet e47c4f741f354a19b49777bd06e54764 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e47c4f741f354a19b49777bd06e54764 already in progress: creating tablet
I20251025 07:52:36.689021 1167 tablet_bootstrap.cc:654] T d66ad049d5484228a268b0facb00ecf1 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.694106 1045 raft_consensus.cc:2468] T c88c46e022ff41a58b02538f2151e260 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:36.695310 1952 ts_tablet_manager.cc:1929] T 3775a2e08d4e40bd9f57e4cda759d535 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:36.695834 875 tablet_service.cc:1552] Processing DeleteTablet for tablet e47c4f741f354a19b49777bd06e54764 with delete_type TABLET_DATA_DELETED (replaced by tablet 755f0d6c87044e20a8020f8410940123) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:36.695834 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 570201e7bf05406a97c81dde071d894b with delete_type TABLET_DATA_DELETED (replaced by tablet 3a0f50c8000144a4acb5219484b01800) from {username='slave'} at 127.0.0.1:59616
W20251025 07:52:36.697013 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet e47c4f741f354a19b49777bd06e54764 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e47c4f741f354a19b49777bd06e54764 already in progress: creating tablet
W20251025 07:52:36.697266 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 570201e7bf05406a97c81dde071d894b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 570201e7bf05406a97c81dde071d894b already in progress: creating tablet
I20251025 07:52:36.699020 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c88c46e022ff41a58b02538f2151e260" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:36.699240 910 raft_consensus.cc:3060] T c88c46e022ff41a58b02538f2151e260 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:36.699383 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f170424be2f643e7af499834955472b0" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:36.699585 903 raft_consensus.cc:3060] T f170424be2f643e7af499834955472b0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:36.702287 1860 raft_consensus.cc:493] T db99a1e68daf4e948a3d66ad276cc5f9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:36.703001 910 raft_consensus.cc:2468] T c88c46e022ff41a58b02538f2151e260 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:36.703011 903 raft_consensus.cc:2468] T f170424be2f643e7af499834955472b0 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:36.703001 1860 raft_consensus.cc:515] T db99a1e68daf4e948a3d66ad276cc5f9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.707655 1860 leader_election.cc:290] T db99a1e68daf4e948a3d66ad276cc5f9 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:36.714749 449 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet a7ab3150a35b4caca9119a1374d3f2f7
I20251025 07:52:36.715456 449 catalog_manager.cc:6719] ReplaceTablet: tablet a7ab3150a35b4caca9119a1374d3f2f7 deleted and replaced by tablet c19c97f881cb45218d4a6f6f231aa96e
I20251025 07:52:36.715798 893 tablet_service.cc:1552] Processing DeleteTablet for tablet a7ab3150a35b4caca9119a1374d3f2f7 with delete_type TABLET_DATA_DELETED (replaced by tablet c19c97f881cb45218d4a6f6f231aa96e) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:36.716743 1011 tablet_service.cc:1552] Processing DeleteTablet for tablet a7ab3150a35b4caca9119a1374d3f2f7 with delete_type TABLET_DATA_DELETED (replaced by tablet c19c97f881cb45218d4a6f6f231aa96e) from {username='slave'} at 127.0.0.1:44572
W20251025 07:52:36.717667 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet a7ab3150a35b4caca9119a1374d3f2f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7ab3150a35b4caca9119a1374d3f2f7 already in progress: creating tablet
I20251025 07:52:36.695465 712 leader_election.cc:304] T c88c46e022ff41a58b02538f2151e260 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:36.709723 1167 tablet_bootstrap.cc:492] T d66ad049d5484228a268b0facb00ecf1 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:36.724992 1167 ts_tablet_manager.cc:1403] T d66ad049d5484228a268b0facb00ecf1 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.002s
I20251025 07:52:36.725056 1980 raft_consensus.cc:2804] T c88c46e022ff41a58b02538f2151e260 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:36.725234 1980 raft_consensus.cc:697] T c88c46e022ff41a58b02538f2151e260 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:36.725353 748 tablet_service.cc:1552] Processing DeleteTablet for tablet a7ab3150a35b4caca9119a1374d3f2f7 with delete_type TABLET_DATA_DELETED (replaced by tablet c19c97f881cb45218d4a6f6f231aa96e) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:36.725538 1980 consensus_queue.cc:237] T c88c46e022ff41a58b02538f2151e260 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.725931 1167 raft_consensus.cc:359] T d66ad049d5484228a268b0facb00ecf1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.726122 710 leader_election.cc:304] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:36.726367 1952 log.cc:1199] T 3775a2e08d4e40bd9f57e4cda759d535 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/3775a2e08d4e40bd9f57e4cda759d535
I20251025 07:52:36.722934 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db99a1e68daf4e948a3d66ad276cc5f9" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:36.726611 776 raft_consensus.cc:2468] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:36.726800 1670 raft_consensus.cc:2804] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:36.726114 1167 raft_consensus.cc:385] T d66ad049d5484228a268b0facb00ecf1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.727011 1167 raft_consensus.cc:740] T d66ad049d5484228a268b0facb00ecf1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.727115 847 leader_election.cc:304] T db99a1e68daf4e948a3d66ad276cc5f9 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:36.727279 1167 consensus_queue.cc:260] T d66ad049d5484228a268b0facb00ecf1 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.727928 1561 raft_consensus.cc:2804] T db99a1e68daf4e948a3d66ad276cc5f9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:36.728055 1561 raft_consensus.cc:493] T db99a1e68daf4e948a3d66ad276cc5f9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:36.728142 1561 raft_consensus.cc:3060] T db99a1e68daf4e948a3d66ad276cc5f9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:36.729468 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db99a1e68daf4e948a3d66ad276cc5f9" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:36.729715 1045 raft_consensus.cc:2468] T db99a1e68daf4e948a3d66ad276cc5f9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:36.731056 1670 raft_consensus.cc:697] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
W20251025 07:52:36.732124 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet a7ab3150a35b4caca9119a1374d3f2f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7ab3150a35b4caca9119a1374d3f2f7 already in progress: creating tablet
I20251025 07:52:36.733068 1561 raft_consensus.cc:515] T db99a1e68daf4e948a3d66ad276cc5f9 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.733206 1670 consensus_queue.cc:237] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.733916 1561 leader_election.cc:290] T db99a1e68daf4e948a3d66ad276cc5f9 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:36.734815 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db99a1e68daf4e948a3d66ad276cc5f9" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:36.735103 1049 raft_consensus.cc:3060] T db99a1e68daf4e948a3d66ad276cc5f9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:36.737834 1167 ts_tablet_manager.cc:1434] T d66ad049d5484228a268b0facb00ecf1 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20251025 07:52:36.738658 1049 raft_consensus.cc:2468] T db99a1e68daf4e948a3d66ad276cc5f9 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:36.739307 850 leader_election.cc:304] T db99a1e68daf4e948a3d66ad276cc5f9 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:36.739612 1167 tablet_bootstrap.cc:492] T 5361f08f50b745eb98bf02e979d7c71e P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:36.740604 1166 raft_consensus.cc:359] T 07898860aa594a05a211fc89768cb76d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:36.740876 1166 raft_consensus.cc:385] T 07898860aa594a05a211fc89768cb76d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.740958 1166 raft_consensus.cc:740] T 07898860aa594a05a211fc89768cb76d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.741175 1166 consensus_queue.cc:260] T 07898860aa594a05a211fc89768cb76d P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:36.742080 1838 raft_consensus.cc:2804] T db99a1e68daf4e948a3d66ad276cc5f9 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:36.742108 1166 ts_tablet_manager.cc:1434] T 07898860aa594a05a211fc89768cb76d P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.057s user 0.003s sys 0.000s
I20251025 07:52:36.742251 1838 raft_consensus.cc:697] T db99a1e68daf4e948a3d66ad276cc5f9 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:36.742524 1166 tablet_bootstrap.cc:492] T d5e5e5bbcf9f43118eea502b69880af5 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:36.742580 1838 consensus_queue.cc:237] T db99a1e68daf4e948a3d66ad276cc5f9 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:36.743496 439 consensus_peers.cc:597] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 -> Peer 8603bddd8a084ff7a73a25641768dfa9 (127.0.104.61:42203): Couldn't send request to peer 8603bddd8a084ff7a73a25641768dfa9. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.61:42203: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20251025 07:52:36.745010 1166 tablet_bootstrap.cc:654] T d5e5e5bbcf9f43118eea502b69880af5 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.745010 1167 tablet_bootstrap.cc:654] T 5361f08f50b745eb98bf02e979d7c71e P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.746529 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db99a1e68daf4e948a3d66ad276cc5f9" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:36.746793 776 raft_consensus.cc:3060] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:36.751289 776 raft_consensus.cc:2468] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:36.751672 455 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet 04449cc97f404a19b6a1b20dea074f95
I20251025 07:52:36.752312 455 catalog_manager.cc:6719] ReplaceTablet: tablet 04449cc97f404a19b6a1b20dea074f95 deleted and replaced by tablet f70091024c034c118fb86578db19a31c
I20251025 07:52:36.756031 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 04449cc97f404a19b6a1b20dea074f95 with delete_type TABLET_DATA_DELETED (replaced by tablet f70091024c034c118fb86578db19a31c) from {username='slave'} at 127.0.0.1:59616
W20251025 07:52:36.757371 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 04449cc97f404a19b6a1b20dea074f95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04449cc97f404a19b6a1b20dea074f95 already in progress: creating tablet
I20251025 07:52:36.758575 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 04449cc97f404a19b6a1b20dea074f95 with delete_type TABLET_DATA_DELETED (replaced by tablet f70091024c034c118fb86578db19a31c) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:36.759116 1011 tablet_service.cc:1552] Processing DeleteTablet for tablet 04449cc97f404a19b6a1b20dea074f95 with delete_type TABLET_DATA_DELETED (replaced by tablet f70091024c034c118fb86578db19a31c) from {username='slave'} at 127.0.0.1:44572
W20251025 07:52:36.759865 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 04449cc97f404a19b6a1b20dea074f95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04449cc97f404a19b6a1b20dea074f95 already in progress: creating tablet
I20251025 07:52:36.774215 1952 ts_tablet_manager.cc:1950] T 3775a2e08d4e40bd9f57e4cda759d535 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:36.775816 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 3775a2e08d4e40bd9f57e4cda759d535 (table default.Table_37ad28b796a342f3a6e962ead289ec5b [id=a4d25738410b4a168aa0c76094daf69b]) successfully deleted
W20251025 07:52:36.776700 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 570201e7bf05406a97c81dde071d894b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 570201e7bf05406a97c81dde071d894b already in progress: creating tablet
W20251025 07:52:36.776952 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet e47c4f741f354a19b49777bd06e54764 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e47c4f741f354a19b49777bd06e54764 already in progress: creating tablet
W20251025 07:52:36.777225 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet a7ab3150a35b4caca9119a1374d3f2f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7ab3150a35b4caca9119a1374d3f2f7 already in progress: creating tablet
W20251025 07:52:36.777417 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 04449cc97f404a19b6a1b20dea074f95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04449cc97f404a19b6a1b20dea074f95 already in progress: creating tablet
I20251025 07:52:36.780892 1169 tablet_bootstrap.cc:492] T f170424be2f643e7af499834955472b0 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:36.781090 1169 ts_tablet_manager.cc:1403] T f170424be2f643e7af499834955472b0 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.138s user 0.008s sys 0.024s
I20251025 07:52:36.781932 1169 raft_consensus.cc:359] T f170424be2f643e7af499834955472b0 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.782181 1169 raft_consensus.cc:385] T f170424be2f643e7af499834955472b0 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.782277 1169 raft_consensus.cc:740] T f170424be2f643e7af499834955472b0 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.782508 1169 consensus_queue.cc:260] T f170424be2f643e7af499834955472b0 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.782918 1166 tablet_bootstrap.cc:492] T d5e5e5bbcf9f43118eea502b69880af5 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:36.783143 1166 ts_tablet_manager.cc:1403] T d5e5e5bbcf9f43118eea502b69880af5 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.041s user 0.001s sys 0.003s
I20251025 07:52:36.783317 1169 ts_tablet_manager.cc:1434] T f170424be2f643e7af499834955472b0 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20251025 07:52:36.784400 1166 raft_consensus.cc:359] T d5e5e5bbcf9f43118eea502b69880af5 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.784606 1166 raft_consensus.cc:385] T d5e5e5bbcf9f43118eea502b69880af5 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.784687 1166 raft_consensus.cc:740] T d5e5e5bbcf9f43118eea502b69880af5 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.784942 1166 consensus_queue.cc:260] T d5e5e5bbcf9f43118eea502b69880af5 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.787149 1169 tablet_bootstrap.cc:492] T 07898860aa594a05a211fc89768cb76d P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:36.789929 1169 tablet_bootstrap.cc:654] T 07898860aa594a05a211fc89768cb76d P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.790267 1166 ts_tablet_manager.cc:1434] T d5e5e5bbcf9f43118eea502b69880af5 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251025 07:52:36.790647 1166 tablet_bootstrap.cc:492] T b97e39aaac3048ed93dbfe873b757254 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:36.792667 1166 tablet_bootstrap.cc:654] T b97e39aaac3048ed93dbfe873b757254 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.801903 450 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_9df11cfcda61482c85092272ca1f3093"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:36.802631 450 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9df11cfcda61482c85092272ca1f3093 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:36.801983 449 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_6caa55a17907446d81c7f0656b329cdd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:36.808734 1167 tablet_bootstrap.cc:492] T 5361f08f50b745eb98bf02e979d7c71e P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:36.808917 1167 ts_tablet_manager.cc:1403] T 5361f08f50b745eb98bf02e979d7c71e P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.071s user 0.008s sys 0.014s
I20251025 07:52:36.809257 446 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_e72e2534c5c44fd299ed3ec3a367a34c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:36.809854 446 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e72e2534c5c44fd299ed3ec3a367a34c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:36.809834 1167 raft_consensus.cc:359] T 5361f08f50b745eb98bf02e979d7c71e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.810061 1167 raft_consensus.cc:385] T 5361f08f50b745eb98bf02e979d7c71e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.810098 1169 tablet_bootstrap.cc:492] T 07898860aa594a05a211fc89768cb76d P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:36.810173 1167 raft_consensus.cc:740] T 5361f08f50b745eb98bf02e979d7c71e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.810289 1169 ts_tablet_manager.cc:1403] T 07898860aa594a05a211fc89768cb76d P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.004s
I20251025 07:52:36.810461 1167 consensus_queue.cc:260] T 5361f08f50b745eb98bf02e979d7c71e P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.810985 448 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_b3ca6007437c4040be247798121808d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:36.811322 1169 raft_consensus.cc:359] T 07898860aa594a05a211fc89768cb76d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
W20251025 07:52:36.811504 448 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3ca6007437c4040be247798121808d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:36.811535 1169 raft_consensus.cc:385] T 07898860aa594a05a211fc89768cb76d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.811628 1169 raft_consensus.cc:740] T 07898860aa594a05a211fc89768cb76d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.812265 1169 consensus_queue.cc:260] T 07898860aa594a05a211fc89768cb76d P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:36.813071 1169 ts_tablet_manager.cc:1434] T 07898860aa594a05a211fc89768cb76d P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251025 07:52:36.813565 1169 tablet_bootstrap.cc:492] T d5e5e5bbcf9f43118eea502b69880af5 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:36.816996 455 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_6eff76f8460c40668372f850e2fc299a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:36.818928 1167 ts_tablet_manager.cc:1434] T 5361f08f50b745eb98bf02e979d7c71e P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20251025 07:52:36.819373 1167 tablet_bootstrap.cc:492] T ef1854d35a3047e3a9fbe0166ab1295a P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:36.821403 1167 tablet_bootstrap.cc:654] T ef1854d35a3047e3a9fbe0166ab1295a P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.821864 1169 tablet_bootstrap.cc:654] T d5e5e5bbcf9f43118eea502b69880af5 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:36.834678 455 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6eff76f8460c40668372f850e2fc299a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:36.835208 1167 tablet_bootstrap.cc:492] T ef1854d35a3047e3a9fbe0166ab1295a P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:36.835378 1167 ts_tablet_manager.cc:1403] T ef1854d35a3047e3a9fbe0166ab1295a P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.003s
I20251025 07:52:36.836149 1167 raft_consensus.cc:359] T ef1854d35a3047e3a9fbe0166ab1295a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.836335 1167 raft_consensus.cc:385] T ef1854d35a3047e3a9fbe0166ab1295a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.836438 1167 raft_consensus.cc:740] T ef1854d35a3047e3a9fbe0166ab1295a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.836781 1167 consensus_queue.cc:260] T ef1854d35a3047e3a9fbe0166ab1295a P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.837360 1167 ts_tablet_manager.cc:1434] T ef1854d35a3047e3a9fbe0166ab1295a P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:36.837643 1167 tablet_bootstrap.cc:492] T 5fca357d7e24494fb6af4bd73d98833c P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
W20251025 07:52:36.838376 449 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6caa55a17907446d81c7f0656b329cdd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:36.838930 1838 raft_consensus.cc:493] T c59f6e6eb9594a8e83f109a09b034045 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:36.839331 1169 tablet_bootstrap.cc:492] T d5e5e5bbcf9f43118eea502b69880af5 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:36.839314 1838 raft_consensus.cc:515] T c59f6e6eb9594a8e83f109a09b034045 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.839511 1169 ts_tablet_manager.cc:1403] T d5e5e5bbcf9f43118eea502b69880af5 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.003s
I20251025 07:52:36.839996 1838 leader_election.cc:290] T c59f6e6eb9594a8e83f109a09b034045 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:36.840268 1169 raft_consensus.cc:359] T d5e5e5bbcf9f43118eea502b69880af5 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.840472 1169 raft_consensus.cc:385] T d5e5e5bbcf9f43118eea502b69880af5 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.840564 1169 raft_consensus.cc:740] T d5e5e5bbcf9f43118eea502b69880af5 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.840705 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c59f6e6eb9594a8e83f109a09b034045" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:36.840898 1045 raft_consensus.cc:2468] T c59f6e6eb9594a8e83f109a09b034045 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:36.840848 1169 consensus_queue.cc:260] T d5e5e5bbcf9f43118eea502b69880af5 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.842015 1169 ts_tablet_manager.cc:1434] T d5e5e5bbcf9f43118eea502b69880af5 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:36.842981 1169 tablet_bootstrap.cc:492] T 53198283718d4b45b30df65b5bde96a6 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:36.843557 850 leader_election.cc:304] T c59f6e6eb9594a8e83f109a09b034045 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:36.843953 1838 raft_consensus.cc:2804] T c59f6e6eb9594a8e83f109a09b034045 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:36.844045 1838 raft_consensus.cc:493] T c59f6e6eb9594a8e83f109a09b034045 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:36.844144 1838 raft_consensus.cc:3060] T c59f6e6eb9594a8e83f109a09b034045 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:36.844609 1169 tablet_bootstrap.cc:654] T 53198283718d4b45b30df65b5bde96a6 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.847304 1838 raft_consensus.cc:515] T c59f6e6eb9594a8e83f109a09b034045 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.848628 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c59f6e6eb9594a8e83f109a09b034045" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:36.848825 1045 raft_consensus.cc:3060] T c59f6e6eb9594a8e83f109a09b034045 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:36.848990 450 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_be8af62c5969471b9e7b869c6820db0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:36.849568 450 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be8af62c5969471b9e7b869c6820db0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:36.852116 1838 leader_election.cc:290] T c59f6e6eb9594a8e83f109a09b034045 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:36.852610 1045 raft_consensus.cc:2468] T c59f6e6eb9594a8e83f109a09b034045 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:36.853073 850 leader_election.cc:304] T c59f6e6eb9594a8e83f109a09b034045 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:36.853412 1838 raft_consensus.cc:2804] T c59f6e6eb9594a8e83f109a09b034045 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:36.853736 1838 raft_consensus.cc:697] T c59f6e6eb9594a8e83f109a09b034045 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:36.854594 1838 consensus_queue.cc:237] T c59f6e6eb9594a8e83f109a09b034045 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.856470 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c59f6e6eb9594a8e83f109a09b034045" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:36.856680 776 raft_consensus.cc:2468] T c59f6e6eb9594a8e83f109a09b034045 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:36.857692 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c59f6e6eb9594a8e83f109a09b034045" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:36.857861 776 raft_consensus.cc:3060] T c59f6e6eb9594a8e83f109a09b034045 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:36.858476 1167 tablet_bootstrap.cc:654] T 5fca357d7e24494fb6af4bd73d98833c P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.861047 776 raft_consensus.cc:2468] T c59f6e6eb9594a8e83f109a09b034045 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:36.864578 446 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_dfc7f1d4bf8b48bebbb00313419c3fd8" } new_table_name: "default.Table_ca479cb5c1184c5391b7e1c9d8a08937" modify_external_catalogs: true
I20251025 07:52:36.866364 1670 raft_consensus.cc:493] T 5361f08f50b745eb98bf02e979d7c71e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:36.866523 1670 raft_consensus.cc:515] T 5361f08f50b745eb98bf02e979d7c71e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.867105 1670 leader_election.cc:290] T 5361f08f50b745eb98bf02e979d7c71e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:36.867830 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5361f08f50b745eb98bf02e979d7c71e" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:36.868130 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5361f08f50b745eb98bf02e979d7c71e" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:36.868717 712 leader_election.cc:343] T 5361f08f50b745eb98bf02e979d7c71e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:36.869815 710 leader_election.cc:343] T 5361f08f50b745eb98bf02e979d7c71e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:36.870538 1569 raft_consensus.cc:493] T d5e5e5bbcf9f43118eea502b69880af5 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:36.870549 710 leader_election.cc:304] T 5361f08f50b745eb98bf02e979d7c71e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:36.870676 1569 raft_consensus.cc:515] T d5e5e5bbcf9f43118eea502b69880af5 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.871052 1670 raft_consensus.cc:2749] T 5361f08f50b745eb98bf02e979d7c71e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:36.871297 1569 leader_election.cc:290] T d5e5e5bbcf9f43118eea502b69880af5 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:36.871999 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5e5e5bbcf9f43118eea502b69880af5" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:36.872227 776 raft_consensus.cc:2468] T d5e5e5bbcf9f43118eea502b69880af5 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:36.872279 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5e5e5bbcf9f43118eea502b69880af5" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:36.872488 913 raft_consensus.cc:2468] T d5e5e5bbcf9f43118eea502b69880af5 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:36.872692 982 leader_election.cc:304] T d5e5e5bbcf9f43118eea502b69880af5 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:36.872951 1569 raft_consensus.cc:2804] T d5e5e5bbcf9f43118eea502b69880af5 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:36.873061 1569 raft_consensus.cc:493] T d5e5e5bbcf9f43118eea502b69880af5 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:36.873150 1569 raft_consensus.cc:3060] T d5e5e5bbcf9f43118eea502b69880af5 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:36.874662 1167 tablet_bootstrap.cc:492] T 5fca357d7e24494fb6af4bd73d98833c P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:36.874831 1167 ts_tablet_manager.cc:1403] T 5fca357d7e24494fb6af4bd73d98833c P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.001s
I20251025 07:52:36.875641 1167 raft_consensus.cc:359] T 5fca357d7e24494fb6af4bd73d98833c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.875841 1167 raft_consensus.cc:385] T 5fca357d7e24494fb6af4bd73d98833c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.875929 1167 raft_consensus.cc:740] T 5fca357d7e24494fb6af4bd73d98833c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.878644 1169 tablet_bootstrap.cc:492] T 53198283718d4b45b30df65b5bde96a6 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:36.878790 1169 ts_tablet_manager.cc:1403] T 53198283718d4b45b30df65b5bde96a6 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.000s
I20251025 07:52:36.879575 1169 raft_consensus.cc:359] T 53198283718d4b45b30df65b5bde96a6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.879742 1169 raft_consensus.cc:385] T 53198283718d4b45b30df65b5bde96a6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.879832 1169 raft_consensus.cc:740] T 53198283718d4b45b30df65b5bde96a6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.879901 1167 consensus_queue.cc:260] T 5fca357d7e24494fb6af4bd73d98833c P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.880088 1169 consensus_queue.cc:260] T 53198283718d4b45b30df65b5bde96a6 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.880638 1169 ts_tablet_manager.cc:1434] T 53198283718d4b45b30df65b5bde96a6 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:36.883419 1569 raft_consensus.cc:515] T d5e5e5bbcf9f43118eea502b69880af5 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.884442 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5e5e5bbcf9f43118eea502b69880af5" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:36.884656 776 raft_consensus.cc:3060] T d5e5e5bbcf9f43118eea502b69880af5 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:36.884784 1169 tablet_bootstrap.cc:492] T 9979cae195e849908677d78c6c33be71 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:36.886309 1671 raft_consensus.cc:493] T f170424be2f643e7af499834955472b0 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:36.886423 1169 tablet_bootstrap.cc:654] T 9979cae195e849908677d78c6c33be71 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.886461 1671 raft_consensus.cc:515] T f170424be2f643e7af499834955472b0 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.887133 1671 leader_election.cc:290] T f170424be2f643e7af499834955472b0 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:36.888449 1569 leader_election.cc:290] T d5e5e5bbcf9f43118eea502b69880af5 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:36.888417 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f170424be2f643e7af499834955472b0" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:36.891060 776 raft_consensus.cc:2468] T d5e5e5bbcf9f43118eea502b69880af5 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:36.891366 1670 raft_consensus.cc:493] T ef1854d35a3047e3a9fbe0166ab1295a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:36.891506 1670 raft_consensus.cc:515] T ef1854d35a3047e3a9fbe0166ab1295a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.892050 1670 leader_election.cc:290] T ef1854d35a3047e3a9fbe0166ab1295a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:36.892520 982 leader_election.cc:304] T d5e5e5bbcf9f43118eea502b69880af5 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:36.892768 1569 raft_consensus.cc:2804] T d5e5e5bbcf9f43118eea502b69880af5 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:36.892905 1569 raft_consensus.cc:697] T d5e5e5bbcf9f43118eea502b69880af5 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:36.893164 1569 consensus_queue.cc:237] T d5e5e5bbcf9f43118eea502b69880af5 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.896544 1670 raft_consensus.cc:493] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:36.896700 1670 raft_consensus.cc:515] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.897215 1670 leader_election.cc:290] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:36.899005 712 leader_election.cc:343] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 2aa522477e934c89834026dc480fe7c0
I20251025 07:52:36.892659 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef1854d35a3047e3a9fbe0166ab1295a" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:36.903004 712 leader_election.cc:343] T ef1854d35a3047e3a9fbe0166ab1295a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:36.904898 1167 ts_tablet_manager.cc:1434] T 5fca357d7e24494fb6af4bd73d98833c P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.030s user 0.000s sys 0.002s
I20251025 07:52:36.905092 449 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_2dd1c1390f234ba8ac968b6842dd28be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:36.905860 449 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2dd1c1390f234ba8ac968b6842dd28be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:36.906515 450 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_a3f50508f4b047b6837c67248417b21c" } modify_external_catalogs: true
I20251025 07:52:36.906850 450 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_a3f50508f4b047b6837c67248417b21c" } modify_external_catalogs: true
I20251025 07:52:36.911084 1167 tablet_bootstrap.cc:492] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:36.913506 1167 tablet_bootstrap.cc:654] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.916831 448 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_3d163f45f94f450ebf0447897bdf41a6" } new_table_name: "default.Table_4a86191163f2408f9bcc795cd8d1bbf6" modify_external_catalogs: true
I20251025 07:52:36.913591 1670 raft_consensus.cc:493] T 6ec421eb820b4f40afcff5a5bf80c508 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:36.917789 1670 raft_consensus.cc:515] T 6ec421eb820b4f40afcff5a5bf80c508 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.918524 1670 leader_election.cc:290] T 6ec421eb820b4f40afcff5a5bf80c508 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:36.921900 1167 tablet_bootstrap.cc:492] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:36.922953 1167 ts_tablet_manager.cc:1403] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.002s
I20251025 07:52:36.924028 1167 raft_consensus.cc:359] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.925309 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ec421eb820b4f40afcff5a5bf80c508" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:36.927158 1169 tablet_bootstrap.cc:492] T 9979cae195e849908677d78c6c33be71 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:36.927340 1169 ts_tablet_manager.cc:1403] T 9979cae195e849908677d78c6c33be71 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.043s user 0.009s sys 0.000s
I20251025 07:52:36.927697 455 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_a437a2623d704e3ea3c5b8c65c88b936" } modify_external_catalogs: true
I20251025 07:52:36.927922 455 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_a437a2623d704e3ea3c5b8c65c88b936" } modify_external_catalogs: true
I20251025 07:52:36.928086 1169 raft_consensus.cc:359] T 9979cae195e849908677d78c6c33be71 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.928287 1169 raft_consensus.cc:385] T 9979cae195e849908677d78c6c33be71 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.928355 1169 raft_consensus.cc:740] T 9979cae195e849908677d78c6c33be71 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
W20251025 07:52:36.925894 712 leader_election.cc:343] T 6ec421eb820b4f40afcff5a5bf80c508 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:36.928696 1167 raft_consensus.cc:385] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.928803 1167 raft_consensus.cc:740] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.929385 1169 consensus_queue.cc:260] T 9979cae195e849908677d78c6c33be71 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:36.929940 982 consensus_peers.cc:597] T 3775a2e08d4e40bd9f57e4cda759d535 P db42be7ad3c741a0aa427f27e245993b -> Peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Couldn't send request to peer 23eeffd3de254ce18db5febfde971c6b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3775a2e08d4e40bd9f57e4cda759d535. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:36.930935 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f170424be2f643e7af499834955472b0" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:36.931443 1167 consensus_queue.cc:260] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.932729 1167 ts_tablet_manager.cc:1434] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251025 07:52:36.932992 1169 ts_tablet_manager.cc:1434] T 9979cae195e849908677d78c6c33be71 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251025 07:52:36.933450 1169 tablet_bootstrap.cc:492] T abf94c32ff2945119b710905c075d63c P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:36.933856 1167 tablet_bootstrap.cc:492] T df1248f6dbd94af68c587da7d149e279 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:36.936054 1169 tablet_bootstrap.cc:654] T abf94c32ff2945119b710905c075d63c P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.936076 1167 tablet_bootstrap.cc:654] T df1248f6dbd94af68c587da7d149e279 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.939378 1670 raft_consensus.cc:493] T 70fe9aa2f58d4abea3b3b71066a29653 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:36.939561 1670 raft_consensus.cc:515] T 70fe9aa2f58d4abea3b3b71066a29653 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.939790 449 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_78724078b4574f35879fd2db388fc4f4" } modify_external_catalogs: true
I20251025 07:52:36.940089 449 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_78724078b4574f35879fd2db388fc4f4" } modify_external_catalogs: true
I20251025 07:52:36.940143 1670 leader_election.cc:290] T 70fe9aa2f58d4abea3b3b71066a29653 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:36.940688 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "70fe9aa2f58d4abea3b3b71066a29653" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:36.941149 712 leader_election.cc:343] T 70fe9aa2f58d4abea3b3b71066a29653 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:36.947547 913 raft_consensus.cc:2393] T f170424be2f643e7af499834955472b0 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 23eeffd3de254ce18db5febfde971c6b in this term.
I20251025 07:52:36.948688 983 leader_election.cc:304] T f170424be2f643e7af499834955472b0 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:36.950222 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5e5e5bbcf9f43118eea502b69880af5" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:36.950412 903 raft_consensus.cc:3060] T d5e5e5bbcf9f43118eea502b69880af5 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:36.950601 1167 tablet_bootstrap.cc:492] T df1248f6dbd94af68c587da7d149e279 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:36.950763 1167 ts_tablet_manager.cc:1403] T df1248f6dbd94af68c587da7d149e279 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.002s
I20251025 07:52:36.951480 1167 raft_consensus.cc:359] T df1248f6dbd94af68c587da7d149e279 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.951664 1167 raft_consensus.cc:385] T df1248f6dbd94af68c587da7d149e279 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.951736 1167 raft_consensus.cc:740] T df1248f6dbd94af68c587da7d149e279 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.953681 1167 consensus_queue.cc:260] T df1248f6dbd94af68c587da7d149e279 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.957957 1167 ts_tablet_manager.cc:1434] T df1248f6dbd94af68c587da7d149e279 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251025 07:52:36.959257 1167 tablet_bootstrap.cc:492] T 3abfd52327a04ba88625f391a8658fbb P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:36.961776 1169 tablet_bootstrap.cc:492] T abf94c32ff2945119b710905c075d63c P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:36.961927 1169 ts_tablet_manager.cc:1403] T abf94c32ff2945119b710905c075d63c P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.001s
I20251025 07:52:36.962455 1167 tablet_bootstrap.cc:654] T 3abfd52327a04ba88625f391a8658fbb P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.963030 1169 raft_consensus.cc:359] T abf94c32ff2945119b710905c075d63c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.963249 1169 raft_consensus.cc:385] T abf94c32ff2945119b710905c075d63c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.963351 1169 raft_consensus.cc:740] T abf94c32ff2945119b710905c075d63c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:36.963584 1169 consensus_queue.cc:260] T abf94c32ff2945119b710905c075d63c P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.963874 1671 raft_consensus.cc:3060] T f170424be2f643e7af499834955472b0 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:36.964133 1169 ts_tablet_manager.cc:1434] T abf94c32ff2945119b710905c075d63c P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:36.968919 1671 raft_consensus.cc:2749] T f170424be2f643e7af499834955472b0 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:36.973593 1169 tablet_bootstrap.cc:492] T 2c50ec95e48e49b59a76fac4fc862d3d P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:36.973822 1167 tablet_bootstrap.cc:492] T 3abfd52327a04ba88625f391a8658fbb P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:36.973958 1167 ts_tablet_manager.cc:1403] T 3abfd52327a04ba88625f391a8658fbb P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20251025 07:52:36.974959 1167 raft_consensus.cc:359] T 3abfd52327a04ba88625f391a8658fbb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:36.975189 1167 raft_consensus.cc:385] T 3abfd52327a04ba88625f391a8658fbb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:36.975304 1167 raft_consensus.cc:740] T 3abfd52327a04ba88625f391a8658fbb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
W20251025 07:52:36.977047 712 consensus_peers.cc:597] T cb824bd780524832b2b60f5618a39da5 P 23eeffd3de254ce18db5febfde971c6b -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cb824bd780524832b2b60f5618a39da5. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:36.977494 1166 tablet_bootstrap.cc:492] T b97e39aaac3048ed93dbfe873b757254 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:36.977815 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef1854d35a3047e3a9fbe0166ab1295a" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:36.979246 1167 consensus_queue.cc:260] T 3abfd52327a04ba88625f391a8658fbb P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:36.980224 903 raft_consensus.cc:2468] T d5e5e5bbcf9f43118eea502b69880af5 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:36.981367 1167 ts_tablet_manager.cc:1434] T 3abfd52327a04ba88625f391a8658fbb P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20251025 07:52:36.982695 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "70fe9aa2f58d4abea3b3b71066a29653" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:36.984524 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5]). Expected version 1 got 0
W20251025 07:52:36.985915 710 leader_election.cc:343] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 2aa522477e934c89834026dc480fe7c0
I20251025 07:52:36.986065 710 leader_election.cc:304] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:36.986503 1670 raft_consensus.cc:2749] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:36.986645 710 leader_election.cc:343] T ef1854d35a3047e3a9fbe0166ab1295a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:36.986768 710 leader_election.cc:304] T ef1854d35a3047e3a9fbe0166ab1295a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:36.986917 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3]). Expected version 1 got 0
I20251025 07:52:36.986976 1670 raft_consensus.cc:2749] T ef1854d35a3047e3a9fbe0166ab1295a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:36.987152 710 leader_election.cc:343] T 70fe9aa2f58d4abea3b3b71066a29653 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:36.987231 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_a6c869949b744bc98efb735b0e73e7aa [id=bf77b5ff6f1048dd996d948437f9ff5e]). Expected version 2 got 0
I20251025 07:52:36.987282 710 leader_election.cc:304] T 70fe9aa2f58d4abea3b3b71066a29653 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:36.987501 1670 raft_consensus.cc:2749] T 70fe9aa2f58d4abea3b3b71066a29653 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:36.987915 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f]). Expected version 1 got 0
I20251025 07:52:36.992233 1707 raft_consensus.cc:493] T 9d6a9937d10a4cc2813671dd1bfa1664 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:36.992802 1707 raft_consensus.cc:515] T 9d6a9937d10a4cc2813671dd1bfa1664 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:36.993315 1670 raft_consensus.cc:493] T 509c94f2a422493ea4390ecee576db72 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:36.993453 1707 leader_election.cc:290] T 9d6a9937d10a4cc2813671dd1bfa1664 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:36.993458 1670 raft_consensus.cc:515] T 509c94f2a422493ea4390ecee576db72 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:36.993707 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ec421eb820b4f40afcff5a5bf80c508" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:36.994061 1670 leader_election.cc:290] T 509c94f2a422493ea4390ecee576db72 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:36.994068 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d6a9937d10a4cc2813671dd1bfa1664" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:36.994349 1045 raft_consensus.cc:2393] T 9d6a9937d10a4cc2813671dd1bfa1664 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3aea2869ff814cc7b142012faa274382 in current term 1: Already voted for candidate 23eeffd3de254ce18db5febfde971c6b in this term.
I20251025 07:52:36.994894 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d6a9937d10a4cc2813671dd1bfa1664" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
W20251025 07:52:36.995076 710 leader_election.cc:343] T 6ec421eb820b4f40afcff5a5bf80c508 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:36.995220 710 leader_election.cc:304] T 6ec421eb820b4f40afcff5a5bf80c508 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:36.995411 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7]). Expected version 1 got 0
I20251025 07:52:36.995536 1166 ts_tablet_manager.cc:1403] T b97e39aaac3048ed93dbfe873b757254 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.187s user 0.002s sys 0.006s
I20251025 07:52:36.995616 1670 raft_consensus.cc:2749] T 6ec421eb820b4f40afcff5a5bf80c508 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:36.995813 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "509c94f2a422493ea4390ecee576db72" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:36.996085 847 leader_election.cc:304] T 9d6a9937d10a4cc2813671dd1bfa1664 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:36.996274 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet cb829649f73d4d3f8862d268c953bd86 (table default.Table_ba4b8cc7b930482690b7e192e0edff12 [id=7944225d3ce7439b881bfb9289d9f040]). Expected version 1 got 0
I20251025 07:52:36.996917 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 04d27d54fdb7483b869408af65ff85c9 (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]). Expected version 2 got 0
W20251025 07:52:36.996982 712 leader_election.cc:343] T 509c94f2a422493ea4390ecee576db72 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:36.997443 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 4c155aa7fa5b4d68bbab1918b8432372 (table default.Table_6c5a24747ba14352b0285a41ec2985a4 [id=28bbaad474164a8c8c97a76bb8081791]). Expected version 2 got 0
I20251025 07:52:36.997648 1707 raft_consensus.cc:3060] T 9d6a9937d10a4cc2813671dd1bfa1664 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:36.998062 1169 tablet_bootstrap.cc:654] T 2c50ec95e48e49b59a76fac4fc862d3d P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:36.998078 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "509c94f2a422493ea4390ecee576db72" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:36.999866 710 leader_election.cc:343] T 509c94f2a422493ea4390ecee576db72 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:37.000027 710 leader_election.cc:304] T 509c94f2a422493ea4390ecee576db72 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:37.000425 1670 raft_consensus.cc:2749] T 509c94f2a422493ea4390ecee576db72 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:37.000497 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet aa9d9ab8073945d28d52fb02213f201a (table default.Table_4791672658f043f6b97b9320a531ebb5 [id=dfd20cf9c3e541c4abe23d59e0aa7f2a]). Expected version 1 got 0
I20251025 07:52:37.001437 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 062c0f3e51a3487880cea373f8cf1f54 (table default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de]). Expected version 1 got 0
I20251025 07:52:37.002934 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 292fe04055954afd8c08ac2636b68191 (table default.Table_9fa69b10b5e842639fc0b0a58cd9b6a5 [id=a253ce0253bd4da4814741c482193d8f]). Expected version 1 got 0
I20251025 07:52:37.003739 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 92aa605fabbc4257a6a204c087e7cc7f (table default.Table_78724078b4574f35879fd2db388fc4f4 [id=187bde879b044237ada58158eb000123]). Expected version 2 got 0
I20251025 07:52:37.004344 1166 raft_consensus.cc:359] T b97e39aaac3048ed93dbfe873b757254 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.004554 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]). Expected version 1 got 0
I20251025 07:52:37.005136 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525]). Expected version 1 got 0
I20251025 07:52:37.005659 1707 raft_consensus.cc:2749] T 9d6a9937d10a4cc2813671dd1bfa1664 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:37.005903 1166 raft_consensus.cc:385] T b97e39aaac3048ed93dbfe873b757254 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:37.006006 1166 raft_consensus.cc:740] T b97e39aaac3048ed93dbfe873b757254 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:37.005686 453 catalog_manager.cc:5649] T fd9b4be139d347f4a6a60767029c342a P db42be7ad3c741a0aa427f27e245993b reported cstate change: term changed from 0 to 1, leader changed from <none> to 23eeffd3de254ce18db5febfde971c6b (127.0.104.1). New cstate: current_term: 1 leader_uuid: "23eeffd3de254ce18db5febfde971c6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } }
I20251025 07:52:37.007700 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84]). Expected version 1 got 0
I20251025 07:52:37.008522 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 5dab439bc50d4e869548154e1d222ad6 (table default.Table_ab6798879a554075b3deaa059ad077c7 [id=45325c188eac4d34bb016d01de091f87]). Expected version 1 got 0
I20251025 07:52:37.009090 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 87268e27540c4ec58ce1e62df819f19d (table default.Table_350383f0e98947eba77186e74ad88781 [id=96b0e47c666c4d8f982741aea7b17ae7]). Expected version 2 got 0
I20251025 07:52:37.009594 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_87d38832f85e4965bfab1a077990114f [id=ad812862698d421184ebf501ce79b7be]). Expected version 2 got 0
I20251025 07:52:37.010303 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c]). Expected version 1 got 0
I20251025 07:52:37.010466 1166 consensus_queue.cc:260] T b97e39aaac3048ed93dbfe873b757254 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.011461 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 0325d462441349ef81841f1b0cab2a20 (table default.Table_750ecb905839434fa9a1aa72f87394be [id=9ce3ba77c56a4ef09cb3b40ea0e488f6]). Expected version 1 got 0
I20251025 07:52:37.011610 1166 ts_tablet_manager.cc:1434] T b97e39aaac3048ed93dbfe873b757254 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.016s user 0.001s sys 0.002s
I20251025 07:52:37.013288 453 catalog_manager.cc:5649] T 6897405bbc1c47a4ab0aa3af80e60e3d P db42be7ad3c741a0aa427f27e245993b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } }
I20251025 07:52:37.014874 1166 tablet_bootstrap.cc:492] T 896bd7870d87408e8adfdb7a78998111 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:37.014885 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 6897405bbc1c47a4ab0aa3af80e60e3d (table default.Table_92c0120dd29348618040150dedb14c43 [id=ea892976886644d8aa025ae2edef718d]). Expected version 2 got 0
I20251025 07:52:37.015579 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 8f2c8582fed34c848d9903b8ad8b4057 (table default.Table_438f67f3dc62465999a917986922b3ed [id=7c7fcd265e624d7fab74661db505d9c3]). Expected version 1 got 0
I20251025 07:52:37.016428 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_e90d7f5ff13b40878e5b625c97792c7d [id=fc8180c09d7b4e74b2e161f7acb8fe0e]). Expected version 1 got 0
I20251025 07:52:37.017362 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]). Expected version 1 got 0
I20251025 07:52:37.017568 1166 tablet_bootstrap.cc:654] T 896bd7870d87408e8adfdb7a78998111 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:37.018505 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]). Expected version 3 got 0
I20251025 07:52:37.022310 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236]). Expected version 1 got 0
I20251025 07:52:37.023413 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 36262ab8c6964dd684c01b4db6219a4f (table default.Table_76158f8cf82e4cdb9a90580e46aaea48 [id=f249f30497bc450c887d659721469de8]). Expected version 1 got 0
I20251025 07:52:37.024644 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 2c50ec95e48e49b59a76fac4fc862d3d (table default.Table_32b54e82cbc84d39a0020826f8737d62 [id=c28b0b6acd61490caecfdaab803a9d92]). Expected version 2 got 0
I20251025 07:52:37.025522 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet b97e39aaac3048ed93dbfe873b757254 (table default.Table_99fd36369f9a4e0fb14f234c98635203 [id=c8059ab6cec146efa631d06799d969a9]). Expected version 1 got 0
I20251025 07:52:37.026257 453 catalog_manager.cc:5649] T 8e6d76b1a97945cfb6c3055e1b622f8a P db42be7ad3c741a0aa427f27e245993b reported cstate change: term changed from 0 to 1, leader changed from <none> to db42be7ad3c741a0aa427f27e245993b (127.0.104.3). New cstate: current_term: 1 leader_uuid: "db42be7ad3c741a0aa427f27e245993b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } }
I20251025 07:52:37.027606 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 8e6d76b1a97945cfb6c3055e1b622f8a (table default.Table_10e57b42f97544208675708b1e028846 [id=f5c1be203e2b4831a025871cf8a2ef17]). Expected version 2 got 0
I20251025 07:52:37.027995 1569 raft_consensus.cc:493] T 9979cae195e849908677d78c6c33be71 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:37.028183 1569 raft_consensus.cc:515] T 9979cae195e849908677d78c6c33be71 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.028556 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet b17d2a9ad6be49d29f2413666fba7e69 (table default.Table_fcddb31989ee4236a4981c12779b3f5d [id=f3794fa081364666b944c412415bdba0]). Expected version 3 got 0
I20251025 07:52:37.028775 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3]). Expected version 2 got 0
I20251025 07:52:37.028875 1569 leader_election.cc:290] T 9979cae195e849908677d78c6c33be71 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:37.029140 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 5361f08f50b745eb98bf02e979d7c71e (table default.Table_a3f50508f4b047b6837c67248417b21c [id=572bc5595be947468e24f2a9144f12a4]). Expected version 2 got 0
I20251025 07:52:37.029421 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 509c94f2a422493ea4390ecee576db72 (table default.Table_3f081b7bb4e1420c8121ea4115439f3d [id=696a1285fbfe4038a8b1b38708c8ea49]). Expected version 1 got 0
I20251025 07:52:37.030095 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9979cae195e849908677d78c6c33be71" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:37.030423 776 raft_consensus.cc:2393] T 9979cae195e849908677d78c6c33be71 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:37.030828 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet ab42909afc524f2fa3c1041495d13a59 (table default.Table_7eb8424d5192485da1a2d67bfdb2bbbf [id=40c29df99590408cbd5c583d4fc741eb]). Expected version 2 got 0
I20251025 07:52:37.031172 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet ef1854d35a3047e3a9fbe0166ab1295a (table default.Table_088aad2a46b041bfa29ed8e6929760b1 [id=834669c54dae449790818684fd8659bc]). Expected version 2 got 0
I20251025 07:52:37.031347 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet bc2d16b33ca8426abbd7bd38339c9f1d (table default.Table_4ea985df4c5b415c9570c64506a98cc4 [id=98d2b7797e33485e8872054aa2eb8ec9]). Expected version 1 got 0
I20251025 07:52:37.031558 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e]). Expected version 1 got 0
I20251025 07:52:37.031755 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 4b28215f61574cf48c447dd01955ac77 (table default.Table_a437a2623d704e3ea3c5b8c65c88b936 [id=51b12903e6024e1a9adc2f9fa7400937]). Expected version 3 got 0
I20251025 07:52:37.032066 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]). Expected version 1 got 0
I20251025 07:52:37.032215 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet ef9a2c9f2ccf432ab835396020872208 (table default.Table_2b9b772d4c234ac286bef3658482e490 [id=409fab60f9d744798e6f92b868f0f7c2]). Expected version 1 got 0
I20251025 07:52:37.032810 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9979cae195e849908677d78c6c33be71" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:37.033624 983 leader_election.cc:304] T 9979cae195e849908677d78c6c33be71 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:37.034214 1569 raft_consensus.cc:3060] T 9979cae195e849908677d78c6c33be71 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.037055 1569 raft_consensus.cc:2749] T 9979cae195e849908677d78c6c33be71 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:36.987164 1167 tablet_bootstrap.cc:492] T 50ae9bca6bed4f749d54191ac1543410 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:37.040563 1167 tablet_bootstrap.cc:654] T 50ae9bca6bed4f749d54191ac1543410 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:37.049860 1569 raft_consensus.cc:493] T abf94c32ff2945119b710905c075d63c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:37.050017 1569 raft_consensus.cc:515] T abf94c32ff2945119b710905c075d63c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.050674 1569 leader_election.cc:290] T abf94c32ff2945119b710905c075d63c P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:37.052450 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "abf94c32ff2945119b710905c075d63c" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:37.052665 776 raft_consensus.cc:2468] T abf94c32ff2945119b710905c075d63c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:37.052953 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "abf94c32ff2945119b710905c075d63c" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:37.053126 982 leader_election.cc:304] T abf94c32ff2945119b710905c075d63c P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
W20251025 07:52:37.053555 983 leader_election.cc:343] T abf94c32ff2945119b710905c075d63c P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:37.057994 1169 tablet_bootstrap.cc:492] T 2c50ec95e48e49b59a76fac4fc862d3d P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:37.058205 1169 ts_tablet_manager.cc:1403] T 2c50ec95e48e49b59a76fac4fc862d3d P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.085s user 0.004s sys 0.000s
I20251025 07:52:37.058987 1169 raft_consensus.cc:359] T 2c50ec95e48e49b59a76fac4fc862d3d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.059182 1169 raft_consensus.cc:385] T 2c50ec95e48e49b59a76fac4fc862d3d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:37.059249 1169 raft_consensus.cc:740] T 2c50ec95e48e49b59a76fac4fc862d3d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:37.059469 1169 consensus_queue.cc:260] T 2c50ec95e48e49b59a76fac4fc862d3d P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.059751 1671 raft_consensus.cc:2804] T abf94c32ff2945119b710905c075d63c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:37.059857 1671 raft_consensus.cc:493] T abf94c32ff2945119b710905c075d63c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:37.059947 1671 raft_consensus.cc:3060] T abf94c32ff2945119b710905c075d63c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.060039 1169 ts_tablet_manager.cc:1434] T 2c50ec95e48e49b59a76fac4fc862d3d P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 07:52:37.062728 1169 tablet_bootstrap.cc:492] T 70fe9aa2f58d4abea3b3b71066a29653 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:37.063015 1671 raft_consensus.cc:515] T abf94c32ff2945119b710905c075d63c P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.063674 1671 leader_election.cc:290] T abf94c32ff2945119b710905c075d63c P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:37.064529 1169 tablet_bootstrap.cc:654] T 70fe9aa2f58d4abea3b3b71066a29653 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:37.066386 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "abf94c32ff2945119b710905c075d63c" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:37.066694 776 raft_consensus.cc:3060] T abf94c32ff2945119b710905c075d63c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.072789 1569 raft_consensus.cc:493] T b97e39aaac3048ed93dbfe873b757254 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:37.072992 1569 raft_consensus.cc:515] T b97e39aaac3048ed93dbfe873b757254 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.078816 1569 leader_election.cc:290] T b97e39aaac3048ed93dbfe873b757254 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:37.080507 1169 tablet_bootstrap.cc:492] T 70fe9aa2f58d4abea3b3b71066a29653 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:37.080672 1169 ts_tablet_manager.cc:1403] T 70fe9aa2f58d4abea3b3b71066a29653 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20251025 07:52:37.081332 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b97e39aaac3048ed93dbfe873b757254" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:37.081470 1169 raft_consensus.cc:359] T 70fe9aa2f58d4abea3b3b71066a29653 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.081573 773 raft_consensus.cc:2468] T b97e39aaac3048ed93dbfe873b757254 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:37.081640 1169 raft_consensus.cc:385] T 70fe9aa2f58d4abea3b3b71066a29653 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:37.082018 982 leader_election.cc:304] T b97e39aaac3048ed93dbfe873b757254 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:37.082367 1569 raft_consensus.cc:2804] T b97e39aaac3048ed93dbfe873b757254 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:37.082472 1569 raft_consensus.cc:493] T b97e39aaac3048ed93dbfe873b757254 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:37.082569 1569 raft_consensus.cc:3060] T b97e39aaac3048ed93dbfe873b757254 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.084501 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "abf94c32ff2945119b710905c075d63c" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
W20251025 07:52:37.085572 983 leader_election.cc:343] T abf94c32ff2945119b710905c075d63c P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:37.085668 1569 raft_consensus.cc:515] T b97e39aaac3048ed93dbfe873b757254 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.086246 1569 leader_election.cc:290] T b97e39aaac3048ed93dbfe873b757254 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:37.087213 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b97e39aaac3048ed93dbfe873b757254" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:37.087464 773 raft_consensus.cc:3060] T b97e39aaac3048ed93dbfe873b757254 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.088778 1166 tablet_bootstrap.cc:492] T 896bd7870d87408e8adfdb7a78998111 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:37.088979 1166 ts_tablet_manager.cc:1403] T 896bd7870d87408e8adfdb7a78998111 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.077s user 0.004s sys 0.000s
I20251025 07:52:37.090085 1166 raft_consensus.cc:359] T 896bd7870d87408e8adfdb7a78998111 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.090337 1166 raft_consensus.cc:385] T 896bd7870d87408e8adfdb7a78998111 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:37.090440 1166 raft_consensus.cc:740] T 896bd7870d87408e8adfdb7a78998111 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:37.090870 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b97e39aaac3048ed93dbfe873b757254" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:37.091048 910 raft_consensus.cc:3060] T b97e39aaac3048ed93dbfe873b757254 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.081719 1169 raft_consensus.cc:740] T 70fe9aa2f58d4abea3b3b71066a29653 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:37.094388 1169 consensus_queue.cc:260] T 70fe9aa2f58d4abea3b3b71066a29653 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.094471 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b97e39aaac3048ed93dbfe873b757254" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:37.098273 1169 ts_tablet_manager.cc:1434] T 70fe9aa2f58d4abea3b3b71066a29653 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20251025 07:52:37.099900 1169 tablet_bootstrap.cc:492] T 509c94f2a422493ea4390ecee576db72 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:37.099752 1166 consensus_queue.cc:260] T 896bd7870d87408e8adfdb7a78998111 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.100119 776 raft_consensus.cc:2468] T abf94c32ff2945119b710905c075d63c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:37.100518 910 raft_consensus.cc:2468] T b97e39aaac3048ed93dbfe873b757254 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:37.100643 982 leader_election.cc:304] T abf94c32ff2945119b710905c075d63c P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters: 3aea2869ff814cc7b142012faa274382
I20251025 07:52:37.100971 1671 raft_consensus.cc:2804] T abf94c32ff2945119b710905c075d63c P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:37.102757 1166 ts_tablet_manager.cc:1434] T 896bd7870d87408e8adfdb7a78998111 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20251025 07:52:37.103152 1166 tablet_bootstrap.cc:492] T abf94c32ff2945119b710905c075d63c P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:37.108273 1166 tablet_bootstrap.cc:654] T abf94c32ff2945119b710905c075d63c P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:37.109432 773 raft_consensus.cc:2468] T b97e39aaac3048ed93dbfe873b757254 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:37.101374 1671 raft_consensus.cc:697] T abf94c32ff2945119b710905c075d63c P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:37.112776 1671 consensus_queue.cc:237] T abf94c32ff2945119b710905c075d63c P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.101850 983 leader_election.cc:304] T b97e39aaac3048ed93dbfe873b757254 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:37.122190 1569 raft_consensus.cc:2804] T b97e39aaac3048ed93dbfe873b757254 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:37.123872 1569 raft_consensus.cc:697] T b97e39aaac3048ed93dbfe873b757254 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:37.126696 1569 consensus_queue.cc:237] T b97e39aaac3048ed93dbfe873b757254 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:37.132336 983 consensus_peers.cc:597] T 3775a2e08d4e40bd9f57e4cda759d535 P db42be7ad3c741a0aa427f27e245993b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3775a2e08d4e40bd9f57e4cda759d535. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:37.135485 1169 tablet_bootstrap.cc:654] T 509c94f2a422493ea4390ecee576db72 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:37.147082 1166 tablet_bootstrap.cc:492] T abf94c32ff2945119b710905c075d63c P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:37.147281 1166 ts_tablet_manager.cc:1403] T abf94c32ff2945119b710905c075d63c P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.044s user 0.008s sys 0.003s
I20251025 07:52:37.148283 1166 raft_consensus.cc:359] T abf94c32ff2945119b710905c075d63c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.148532 1166 raft_consensus.cc:385] T abf94c32ff2945119b710905c075d63c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:37.148607 1166 raft_consensus.cc:740] T abf94c32ff2945119b710905c075d63c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:37.148909 1166 consensus_queue.cc:260] T abf94c32ff2945119b710905c075d63c P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.155052 1169 tablet_bootstrap.cc:492] T 509c94f2a422493ea4390ecee576db72 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:37.155287 1169 ts_tablet_manager.cc:1403] T 509c94f2a422493ea4390ecee576db72 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.055s user 0.002s sys 0.002s
I20251025 07:52:37.156149 1169 raft_consensus.cc:359] T 509c94f2a422493ea4390ecee576db72 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.157228 1167 tablet_bootstrap.cc:492] T 50ae9bca6bed4f749d54191ac1543410 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:37.157562 1167 ts_tablet_manager.cc:1403] T 50ae9bca6bed4f749d54191ac1543410 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.170s user 0.010s sys 0.000s
I20251025 07:52:37.158185 1166 ts_tablet_manager.cc:1434] T abf94c32ff2945119b710905c075d63c P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.011s user 0.002s sys 0.002s
I20251025 07:52:37.159101 1167 raft_consensus.cc:359] T 50ae9bca6bed4f749d54191ac1543410 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:37.159389 1167 raft_consensus.cc:385] T 50ae9bca6bed4f749d54191ac1543410 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:37.159514 1167 raft_consensus.cc:740] T 50ae9bca6bed4f749d54191ac1543410 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:37.159830 1167 consensus_queue.cc:260] T 50ae9bca6bed4f749d54191ac1543410 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:37.166513 1166 tablet_bootstrap.cc:492] T 509c94f2a422493ea4390ecee576db72 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:37.169266 1167 ts_tablet_manager.cc:1434] T 50ae9bca6bed4f749d54191ac1543410 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20251025 07:52:37.169559 1169 raft_consensus.cc:385] T 509c94f2a422493ea4390ecee576db72 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:37.175710 1169 raft_consensus.cc:740] T 509c94f2a422493ea4390ecee576db72 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:37.175513 1167 tablet_bootstrap.cc:492] T 8f2c8582fed34c848d9903b8ad8b4057 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:37.177826 1167 tablet_bootstrap.cc:654] T 8f2c8582fed34c848d9903b8ad8b4057 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:37.178104 1169 consensus_queue.cc:260] T 509c94f2a422493ea4390ecee576db72 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.178823 1169 ts_tablet_manager.cc:1434] T 509c94f2a422493ea4390ecee576db72 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20251025 07:52:37.179507 1169 tablet_bootstrap.cc:492] T ef1854d35a3047e3a9fbe0166ab1295a P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:37.181661 1166 tablet_bootstrap.cc:654] T 509c94f2a422493ea4390ecee576db72 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:37.184345 1670 consensus_queue.cc:1048] T c4610b0c62e741eca28c21f63cc3de81 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:37.197325 1011 tablet_service.cc:1505] Processing CreateTablet for tablet a43b006b11e04adfbaf1ad2ea4f04e1c (DEFAULT_TABLE table=default.Table_9df11cfcda61482c85092272ca1f3093 [id=af27951cae5344a3acc34ef0accf5ceb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.197379 893 tablet_service.cc:1505] Processing CreateTablet for tablet a43b006b11e04adfbaf1ad2ea4f04e1c (DEFAULT_TABLE table=default.Table_9df11cfcda61482c85092272ca1f3093 [id=af27951cae5344a3acc34ef0accf5ceb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.197829 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a43b006b11e04adfbaf1ad2ea4f04e1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.197834 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a43b006b11e04adfbaf1ad2ea4f04e1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.200366 1169 tablet_bootstrap.cc:654] T ef1854d35a3047e3a9fbe0166ab1295a P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:37.200498 748 tablet_service.cc:1505] Processing CreateTablet for tablet a43b006b11e04adfbaf1ad2ea4f04e1c (DEFAULT_TABLE table=default.Table_9df11cfcda61482c85092272ca1f3093 [id=af27951cae5344a3acc34ef0accf5ceb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.200973 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a43b006b11e04adfbaf1ad2ea4f04e1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.216219 875 tablet_service.cc:1505] Processing CreateTablet for tablet 623dec4eab1d4666baf2334a01eeb36d (DEFAULT_TABLE table=default.Table_58b693b694cf4930bedf9f4cc6c52c35 [id=00233e389d0a4f1482425635c1ef4a7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.217057 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 623dec4eab1d4666baf2334a01eeb36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.229235 1167 tablet_bootstrap.cc:492] T 8f2c8582fed34c848d9903b8ad8b4057 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:37.229466 1167 ts_tablet_manager.cc:1403] T 8f2c8582fed34c848d9903b8ad8b4057 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.054s user 0.002s sys 0.003s
I20251025 07:52:37.230186 748 tablet_service.cc:1505] Processing CreateTablet for tablet 623dec4eab1d4666baf2334a01eeb36d (DEFAULT_TABLE table=default.Table_58b693b694cf4930bedf9f4cc6c52c35 [id=00233e389d0a4f1482425635c1ef4a7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.230587 1167 raft_consensus.cc:359] T 8f2c8582fed34c848d9903b8ad8b4057 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.230851 1167 raft_consensus.cc:385] T 8f2c8582fed34c848d9903b8ad8b4057 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:37.230980 1167 raft_consensus.cc:740] T 8f2c8582fed34c848d9903b8ad8b4057 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:37.231163 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 623dec4eab1d4666baf2334a01eeb36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.232894 903 raft_consensus.cc:3060] T be5ab0db6ac34c2c9a370445e251eb46 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.234721 1167 consensus_queue.cc:260] T 8f2c8582fed34c848d9903b8ad8b4057 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.235441 1167 ts_tablet_manager.cc:1434] T 8f2c8582fed34c848d9903b8ad8b4057 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20251025 07:52:37.235839 1167 tablet_bootstrap.cc:492] T 410ef150a7014c6c867a021b544a2f4e P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:37.237514 903 raft_consensus.cc:3060] T f3001314b56345f5a70314777cae0125 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.238207 1980 consensus_queue.cc:1048] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:37.239543 1167 tablet_bootstrap.cc:654] T 410ef150a7014c6c867a021b544a2f4e P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:37.241873 1017 tablet_service.cc:1505] Processing CreateTablet for tablet 623dec4eab1d4666baf2334a01eeb36d (DEFAULT_TABLE table=default.Table_58b693b694cf4930bedf9f4cc6c52c35 [id=00233e389d0a4f1482425635c1ef4a7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.242359 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 623dec4eab1d4666baf2334a01eeb36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.242410 1670 consensus_queue.cc:1048] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251025 07:52:37.243358 712 consensus_peers.cc:597] T c4610b0c62e741eca28c21f63cc3de81 P 23eeffd3de254ce18db5febfde971c6b -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c4610b0c62e741eca28c21f63cc3de81. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:37.243614 1670 consensus_queue.cc:1048] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:37.244913 1569 raft_consensus.cc:493] T 896bd7870d87408e8adfdb7a78998111 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:37.251693 1569 raft_consensus.cc:515] T 896bd7870d87408e8adfdb7a78998111 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.252532 1569 leader_election.cc:290] T 896bd7870d87408e8adfdb7a78998111 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:37.254208 1670 raft_consensus.cc:493] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:37.254403 1670 raft_consensus.cc:515] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.254536 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "896bd7870d87408e8adfdb7a78998111" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:37.254741 774 raft_consensus.cc:2468] T 896bd7870d87408e8adfdb7a78998111 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:37.254982 1670 leader_election.cc:290] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:37.255594 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa998b9ca6724c4cb1eb68a3eecb0b11" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:37.255807 982 leader_election.cc:304] T 896bd7870d87408e8adfdb7a78998111 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:37.256039 1671 raft_consensus.cc:493] T 70fe9aa2f58d4abea3b3b71066a29653 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:37.256114 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa998b9ca6724c4cb1eb68a3eecb0b11" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:37.256196 1569 raft_consensus.cc:2804] T 896bd7870d87408e8adfdb7a78998111 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:37.256222 1671 raft_consensus.cc:515] T 70fe9aa2f58d4abea3b3b71066a29653 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.256314 1569 raft_consensus.cc:493] T 896bd7870d87408e8adfdb7a78998111 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:37.256409 1569 raft_consensus.cc:3060] T 896bd7870d87408e8adfdb7a78998111 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.256831 1671 leader_election.cc:290] T 70fe9aa2f58d4abea3b3b71066a29653 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:37.256863 712 leader_election.cc:343] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:37.258424 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "70fe9aa2f58d4abea3b3b71066a29653" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:37.258893 983 leader_election.cc:343] T 70fe9aa2f58d4abea3b3b71066a29653 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:37.260850 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "70fe9aa2f58d4abea3b3b71066a29653" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:37.261112 774 raft_consensus.cc:2468] T 70fe9aa2f58d4abea3b3b71066a29653 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
W20251025 07:52:37.261133 710 leader_election.cc:343] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:37.261315 710 leader_election.cc:304] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:37.261628 982 leader_election.cc:304] T 70fe9aa2f58d4abea3b3b71066a29653 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters: 3aea2869ff814cc7b142012faa274382
I20251025 07:52:37.261904 1980 raft_consensus.cc:2749] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:37.262246 1671 raft_consensus.cc:2804] T 70fe9aa2f58d4abea3b3b71066a29653 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:37.262375 1671 raft_consensus.cc:493] T 70fe9aa2f58d4abea3b3b71066a29653 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:37.262504 1671 raft_consensus.cc:3060] T 70fe9aa2f58d4abea3b3b71066a29653 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.270562 1671 raft_consensus.cc:515] T 70fe9aa2f58d4abea3b3b71066a29653 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.272811 1095 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
I20251025 07:52:37.275241 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "70fe9aa2f58d4abea3b3b71066a29653" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:37.275533 774 raft_consensus.cc:3060] T 70fe9aa2f58d4abea3b3b71066a29653 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.279887 774 raft_consensus.cc:2468] T 70fe9aa2f58d4abea3b3b71066a29653 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:37.280534 982 leader_election.cc:304] T 70fe9aa2f58d4abea3b3b71066a29653 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:37.281015 1612 raft_consensus.cc:2804] T 70fe9aa2f58d4abea3b3b71066a29653 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:37.281195 1612 raft_consensus.cc:697] T 70fe9aa2f58d4abea3b3b71066a29653 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:37.281579 1612 consensus_queue.cc:237] T 70fe9aa2f58d4abea3b3b71066a29653 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.284695 875 tablet_service.cc:1505] Processing CreateTablet for tablet 755f0d6c87044e20a8020f8410940123 (DEFAULT_TABLE table=default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.285179 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 755f0d6c87044e20a8020f8410940123. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.288470 1671 leader_election.cc:290] T 70fe9aa2f58d4abea3b3b71066a29653 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:37.288945 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "70fe9aa2f58d4abea3b3b71066a29653" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
W20251025 07:52:37.289610 983 leader_election.cc:343] T 70fe9aa2f58d4abea3b3b71066a29653 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:37.260414 1569 raft_consensus.cc:515] T 896bd7870d87408e8adfdb7a78998111 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.294072 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "896bd7870d87408e8adfdb7a78998111" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:37.294355 776 raft_consensus.cc:3060] T 896bd7870d87408e8adfdb7a78998111 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.294512 1838 raft_consensus.cc:493] T 07898860aa594a05a211fc89768cb76d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:37.294644 1838 raft_consensus.cc:515] T 07898860aa594a05a211fc89768cb76d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:37.295249 1838 leader_election.cc:290] T 07898860aa594a05a211fc89768cb76d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:37.295955 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07898860aa594a05a211fc89768cb76d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:37.296172 1045 raft_consensus.cc:2468] T 07898860aa594a05a211fc89768cb76d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:37.296666 850 leader_election.cc:304] T 07898860aa594a05a211fc89768cb76d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:37.296957 1838 raft_consensus.cc:2804] T 07898860aa594a05a211fc89768cb76d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:37.297031 1838 raft_consensus.cc:493] T 07898860aa594a05a211fc89768cb76d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:37.297106 1838 raft_consensus.cc:3060] T 07898860aa594a05a211fc89768cb76d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.300283 1838 raft_consensus.cc:515] T 07898860aa594a05a211fc89768cb76d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:37.300443 776 raft_consensus.cc:2468] T 896bd7870d87408e8adfdb7a78998111 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:37.301060 982 leader_election.cc:304] T 896bd7870d87408e8adfdb7a78998111 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:37.301368 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07898860aa594a05a211fc89768cb76d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:37.301431 2050 raft_consensus.cc:2804] T 896bd7870d87408e8adfdb7a78998111 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:37.301570 1045 raft_consensus.cc:3060] T 07898860aa594a05a211fc89768cb76d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.301614 2050 raft_consensus.cc:697] T 896bd7870d87408e8adfdb7a78998111 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:37.301980 2050 consensus_queue.cc:237] T 896bd7870d87408e8adfdb7a78998111 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.304579 1045 raft_consensus.cc:2468] T 07898860aa594a05a211fc89768cb76d P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:37.305251 850 leader_election.cc:304] T 07898860aa594a05a211fc89768cb76d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:37.305516 1561 raft_consensus.cc:2804] T 07898860aa594a05a211fc89768cb76d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:37.305617 1561 raft_consensus.cc:697] T 07898860aa594a05a211fc89768cb76d P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:37.305835 1561 consensus_queue.cc:237] T 07898860aa594a05a211fc89768cb76d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:37.307873 1838 leader_election.cc:290] T 07898860aa594a05a211fc89768cb76d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:37.310194 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07898860aa594a05a211fc89768cb76d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:37.310474 775 raft_consensus.cc:3060] T 07898860aa594a05a211fc89768cb76d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.310528 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "896bd7870d87408e8adfdb7a78998111" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:37.310715 903 raft_consensus.cc:2468] T 896bd7870d87408e8adfdb7a78998111 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:37.314081 775 raft_consensus.cc:2468] T 07898860aa594a05a211fc89768cb76d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:37.316790 1659 raft_consensus.cc:493] T d66ad049d5484228a268b0facb00ecf1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:37.316970 1659 raft_consensus.cc:515] T d66ad049d5484228a268b0facb00ecf1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.317786 1659 leader_election.cc:290] T d66ad049d5484228a268b0facb00ecf1 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:37.317806 1569 leader_election.cc:290] T 896bd7870d87408e8adfdb7a78998111 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:37.318809 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "896bd7870d87408e8adfdb7a78998111" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:37.319128 903 raft_consensus.cc:3060] T 896bd7870d87408e8adfdb7a78998111 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.320986 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d66ad049d5484228a268b0facb00ecf1" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:37.321213 1167 tablet_bootstrap.cc:492] T 410ef150a7014c6c867a021b544a2f4e P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:37.321398 1167 ts_tablet_manager.cc:1403] T 410ef150a7014c6c867a021b544a2f4e P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.086s user 0.009s sys 0.001s
W20251025 07:52:37.321588 712 leader_election.cc:343] T d66ad049d5484228a268b0facb00ecf1 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:37.322093 1166 tablet_bootstrap.cc:492] T 509c94f2a422493ea4390ecee576db72 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:37.322278 1166 ts_tablet_manager.cc:1403] T 509c94f2a422493ea4390ecee576db72 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.156s user 0.004s sys 0.003s
I20251025 07:52:37.322321 1167 raft_consensus.cc:359] T 410ef150a7014c6c867a021b544a2f4e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.322531 1167 raft_consensus.cc:385] T 410ef150a7014c6c867a021b544a2f4e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:37.322614 1167 raft_consensus.cc:740] T 410ef150a7014c6c867a021b544a2f4e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:37.323215 1166 raft_consensus.cc:359] T 509c94f2a422493ea4390ecee576db72 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.323424 1166 raft_consensus.cc:385] T 509c94f2a422493ea4390ecee576db72 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:37.323503 1166 raft_consensus.cc:740] T 509c94f2a422493ea4390ecee576db72 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:37.323741 1166 consensus_queue.cc:260] T 509c94f2a422493ea4390ecee576db72 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.325192 1166 ts_tablet_manager.cc:1434] T 509c94f2a422493ea4390ecee576db72 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251025 07:52:37.325544 1166 tablet_bootstrap.cc:492] T 53198283718d4b45b30df65b5bde96a6 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:37.326087 1167 consensus_queue.cc:260] T 410ef150a7014c6c867a021b544a2f4e P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.326783 1659 consensus_queue.cc:1048] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:37.331094 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07898860aa594a05a211fc89768cb76d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:37.331341 776 raft_consensus.cc:2376] T 07898860aa594a05a211fc89768cb76d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1. Re-sending same reply.
I20251025 07:52:37.334004 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d66ad049d5484228a268b0facb00ecf1" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:37.336089 1167 ts_tablet_manager.cc:1434] T 410ef150a7014c6c867a021b544a2f4e P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.015s user 0.003s sys 0.000s
I20251025 07:52:37.336213 903 raft_consensus.cc:2468] T 896bd7870d87408e8adfdb7a78998111 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
W20251025 07:52:37.339892 710 leader_election.cc:343] T d66ad049d5484228a268b0facb00ecf1 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:37.340056 710 leader_election.cc:304] T d66ad049d5484228a268b0facb00ecf1 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:37.340788 1167 tablet_bootstrap.cc:492] T 22c97b3ee92140c68a7e1b93033cb299 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:37.340855 1659 raft_consensus.cc:2749] T d66ad049d5484228a268b0facb00ecf1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:37.349916 1167 tablet_bootstrap.cc:654] T 22c97b3ee92140c68a7e1b93033cb299 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:37.351058 1659 raft_consensus.cc:493] T 551d07a9242740dcb9366bb2d707a4a8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:37.351253 1659 raft_consensus.cc:515] T 551d07a9242740dcb9366bb2d707a4a8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.351967 1659 leader_election.cc:290] T 551d07a9242740dcb9366bb2d707a4a8 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:37.352820 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "551d07a9242740dcb9366bb2d707a4a8" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:37.353693 712 leader_election.cc:343] T 551d07a9242740dcb9366bb2d707a4a8 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:37.354830 1167 tablet_bootstrap.cc:492] T 22c97b3ee92140c68a7e1b93033cb299 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:37.354987 1167 ts_tablet_manager.cc:1403] T 22c97b3ee92140c68a7e1b93033cb299 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.002s
I20251025 07:52:37.355815 1167 raft_consensus.cc:359] T 22c97b3ee92140c68a7e1b93033cb299 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:37.355993 1167 raft_consensus.cc:385] T 22c97b3ee92140c68a7e1b93033cb299 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:37.356091 1167 raft_consensus.cc:740] T 22c97b3ee92140c68a7e1b93033cb299 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:37.356364 1167 consensus_queue.cc:260] T 22c97b3ee92140c68a7e1b93033cb299 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:37.359679 1167 ts_tablet_manager.cc:1434] T 22c97b3ee92140c68a7e1b93033cb299 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20251025 07:52:37.361423 1167 tablet_bootstrap.cc:492] T 74e2d78c024849b0af5c0bde1f48c36f P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
W20251025 07:52:37.364614 442 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 509c94f2a422493ea4390ecee576db72 (table default.Table_3f081b7bb4e1420c8121ea4115439f3d [id=696a1285fbfe4038a8b1b38708c8ea49], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 509c94f2a422493ea4390ecee576db72 consensus state has no leader
I20251025 07:52:37.365156 822 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
I20251025 07:52:37.365131 893 tablet_service.cc:1505] Processing CreateTablet for tablet c19c97f881cb45218d4a6f6f231aa96e (DEFAULT_TABLE table=default.Table_8129e15bf5c549bd853c04b227a7ef26 [id=e9e8547061ba4ac987d0aa7616c7f3ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.365559 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c19c97f881cb45218d4a6f6f231aa96e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.365449 1011 tablet_service.cc:1505] Processing CreateTablet for tablet 755f0d6c87044e20a8020f8410940123 (DEFAULT_TABLE table=default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.365892 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 755f0d6c87044e20a8020f8410940123. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.371987 1167 tablet_bootstrap.cc:654] T 74e2d78c024849b0af5c0bde1f48c36f P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:37.374297 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "551d07a9242740dcb9366bb2d707a4a8" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:37.374958 710 leader_election.cc:343] T 551d07a9242740dcb9366bb2d707a4a8 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:37.375119 710 leader_election.cc:304] T 551d07a9242740dcb9366bb2d707a4a8 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:37.375525 1659 raft_consensus.cc:2749] T 551d07a9242740dcb9366bb2d707a4a8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:37.375660 1166 tablet_bootstrap.cc:654] T 53198283718d4b45b30df65b5bde96a6 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:37.378201 1169 tablet_bootstrap.cc:492] T ef1854d35a3047e3a9fbe0166ab1295a P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:37.378469 1169 ts_tablet_manager.cc:1403] T ef1854d35a3047e3a9fbe0166ab1295a P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.199s user 0.004s sys 0.000s
I20251025 07:52:37.379549 1169 raft_consensus.cc:359] T ef1854d35a3047e3a9fbe0166ab1295a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.379787 1169 raft_consensus.cc:385] T ef1854d35a3047e3a9fbe0166ab1295a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:37.379904 1169 raft_consensus.cc:740] T ef1854d35a3047e3a9fbe0166ab1295a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:37.380236 1169 consensus_queue.cc:260] T ef1854d35a3047e3a9fbe0166ab1295a P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.380895 1169 ts_tablet_manager.cc:1434] T ef1854d35a3047e3a9fbe0166ab1295a P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:37.381249 1169 tablet_bootstrap.cc:492] T 551d07a9242740dcb9366bb2d707a4a8 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:37.381482 747 tablet_service.cc:1505] Processing CreateTablet for tablet 755f0d6c87044e20a8020f8410940123 (DEFAULT_TABLE table=default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.381911 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 755f0d6c87044e20a8020f8410940123. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.382419 1167 tablet_bootstrap.cc:492] T 74e2d78c024849b0af5c0bde1f48c36f P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:37.382568 1167 ts_tablet_manager.cc:1403] T 74e2d78c024849b0af5c0bde1f48c36f P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.006s
I20251025 07:52:37.383366 1167 raft_consensus.cc:359] T 74e2d78c024849b0af5c0bde1f48c36f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:37.383630 1167 raft_consensus.cc:385] T 74e2d78c024849b0af5c0bde1f48c36f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:37.383703 1167 raft_consensus.cc:740] T 74e2d78c024849b0af5c0bde1f48c36f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:37.385841 1167 consensus_queue.cc:260] T 74e2d78c024849b0af5c0bde1f48c36f P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:37.387389 1167 ts_tablet_manager.cc:1434] T 74e2d78c024849b0af5c0bde1f48c36f P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20251025 07:52:37.387710 1167 tablet_bootstrap.cc:492] T 6a58931e7cd7454fb583f1c70cd2ad15 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:37.391324 1017 tablet_service.cc:1505] Processing CreateTablet for tablet c19c97f881cb45218d4a6f6f231aa96e (DEFAULT_TABLE table=default.Table_8129e15bf5c549bd853c04b227a7ef26 [id=e9e8547061ba4ac987d0aa7616c7f3ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.391862 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c19c97f881cb45218d4a6f6f231aa96e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.393538 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 1d121fe4cde04b969a5620f1d58288aa (DEFAULT_TABLE table=default.Table_56a34b2e5e744f94a5a955e4a796498a [id=e35831cc8e8d47139dd92b95e78c6aee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.394004 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d121fe4cde04b969a5620f1d58288aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.394451 884 tablet_service.cc:1505] Processing CreateTablet for tablet f70091024c034c118fb86578db19a31c (DEFAULT_TABLE table=default.Table_332b308220f64c6dbb5f29711d8463b3 [id=e041946c093b4c789ce175c736cc230b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.394829 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f70091024c034c118fb86578db19a31c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.395112 1018 tablet_service.cc:1505] Processing CreateTablet for tablet f70091024c034c118fb86578db19a31c (DEFAULT_TABLE table=default.Table_332b308220f64c6dbb5f29711d8463b3 [id=e041946c093b4c789ce175c736cc230b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.395467 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f70091024c034c118fb86578db19a31c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.401022 1838 raft_consensus.cc:493] T abf94c32ff2945119b710905c075d63c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:37.401161 1838 raft_consensus.cc:515] T abf94c32ff2945119b710905c075d63c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.401798 1838 leader_election.cc:290] T abf94c32ff2945119b710905c075d63c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:37.409119 1167 tablet_bootstrap.cc:654] T 6a58931e7cd7454fb583f1c70cd2ad15 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:37.416666 2050 raft_consensus.cc:493] T 53198283718d4b45b30df65b5bde96a6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:37.416981 2050 raft_consensus.cc:515] T 53198283718d4b45b30df65b5bde96a6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.418329 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53198283718d4b45b30df65b5bde96a6" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:37.418588 774 raft_consensus.cc:2468] T 53198283718d4b45b30df65b5bde96a6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:37.419008 982 leader_election.cc:304] T 53198283718d4b45b30df65b5bde96a6 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:37.419248 2050 leader_election.cc:290] T 53198283718d4b45b30df65b5bde96a6 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:37.419395 1671 raft_consensus.cc:2804] T 53198283718d4b45b30df65b5bde96a6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:37.419538 1671 raft_consensus.cc:493] T 53198283718d4b45b30df65b5bde96a6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:37.419694 1671 raft_consensus.cc:3060] T 53198283718d4b45b30df65b5bde96a6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.419700 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53198283718d4b45b30df65b5bde96a6" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:37.420169 983 leader_election.cc:343] T 53198283718d4b45b30df65b5bde96a6 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:37.420979 893 tablet_service.cc:1505] Processing CreateTablet for tablet 1d121fe4cde04b969a5620f1d58288aa (DEFAULT_TABLE table=default.Table_56a34b2e5e744f94a5a955e4a796498a [id=e35831cc8e8d47139dd92b95e78c6aee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.421372 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d121fe4cde04b969a5620f1d58288aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.423296 1671 raft_consensus.cc:515] T 53198283718d4b45b30df65b5bde96a6 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.424342 1671 leader_election.cc:290] T 53198283718d4b45b30df65b5bde96a6 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:37.425006 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53198283718d4b45b30df65b5bde96a6" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:37.429101 960 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
I20251025 07:52:37.425206 776 raft_consensus.cc:3060] T 53198283718d4b45b30df65b5bde96a6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.439851 1169 tablet_bootstrap.cc:654] T 551d07a9242740dcb9366bb2d707a4a8 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:37.441133 776 raft_consensus.cc:2468] T 53198283718d4b45b30df65b5bde96a6 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:37.441970 1166 tablet_bootstrap.cc:492] T 53198283718d4b45b30df65b5bde96a6 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:37.442155 1166 ts_tablet_manager.cc:1403] T 53198283718d4b45b30df65b5bde96a6 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.117s user 0.004s sys 0.001s
I20251025 07:52:37.443089 1166 raft_consensus.cc:359] T 53198283718d4b45b30df65b5bde96a6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.443300 1166 raft_consensus.cc:385] T 53198283718d4b45b30df65b5bde96a6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:37.443362 1166 raft_consensus.cc:740] T 53198283718d4b45b30df65b5bde96a6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:37.443668 1166 consensus_queue.cc:260] T 53198283718d4b45b30df65b5bde96a6 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.444674 1166 ts_tablet_manager.cc:1434] T 53198283718d4b45b30df65b5bde96a6 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251025 07:52:37.445027 1166 tablet_bootstrap.cc:492] T 2c50ec95e48e49b59a76fac4fc862d3d P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:37.448078 982 leader_election.cc:304] T 53198283718d4b45b30df65b5bde96a6 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:37.448514 2050 raft_consensus.cc:2804] T 53198283718d4b45b30df65b5bde96a6 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:37.448704 2050 raft_consensus.cc:697] T 53198283718d4b45b30df65b5bde96a6 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:37.448997 2050 consensus_queue.cc:237] T 53198283718d4b45b30df65b5bde96a6 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.450176 1166 tablet_bootstrap.cc:654] T 2c50ec95e48e49b59a76fac4fc862d3d P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:37.451010 439 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:42203, user_credentials={real_user=slave}} blocked reactor thread for 69996.3us
I20251025 07:52:37.452044 747 tablet_service.cc:1505] Processing CreateTablet for tablet c19c97f881cb45218d4a6f6f231aa96e (DEFAULT_TABLE table=default.Table_8129e15bf5c549bd853c04b227a7ef26 [id=e9e8547061ba4ac987d0aa7616c7f3ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.453343 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c19c97f881cb45218d4a6f6f231aa96e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.454767 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53198283718d4b45b30df65b5bde96a6" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:37.454958 903 raft_consensus.cc:3060] T 53198283718d4b45b30df65b5bde96a6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.458606 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "abf94c32ff2945119b710905c075d63c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:37.458941 773 raft_consensus.cc:2393] T abf94c32ff2945119b710905c075d63c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3aea2869ff814cc7b142012faa274382 in current term 1: Already voted for candidate db42be7ad3c741a0aa427f27e245993b in this term.
I20251025 07:52:37.460310 1169 tablet_bootstrap.cc:492] T 551d07a9242740dcb9366bb2d707a4a8 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:37.460489 1169 ts_tablet_manager.cc:1403] T 551d07a9242740dcb9366bb2d707a4a8 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.079s user 0.001s sys 0.003s
I20251025 07:52:37.460754 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "abf94c32ff2945119b710905c075d63c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:37.461369 850 leader_election.cc:304] T abf94c32ff2945119b710905c075d63c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:37.461514 1169 raft_consensus.cc:359] T 551d07a9242740dcb9366bb2d707a4a8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.461689 1169 raft_consensus.cc:385] T 551d07a9242740dcb9366bb2d707a4a8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:37.461753 1169 raft_consensus.cc:740] T 551d07a9242740dcb9366bb2d707a4a8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:37.461762 744 tablet_service.cc:1505] Processing CreateTablet for tablet 1d121fe4cde04b969a5620f1d58288aa (DEFAULT_TABLE table=default.Table_56a34b2e5e744f94a5a955e4a796498a [id=e35831cc8e8d47139dd92b95e78c6aee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.461958 1561 raft_consensus.cc:3060] T abf94c32ff2945119b710905c075d63c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.464025 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d121fe4cde04b969a5620f1d58288aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.465930 1670 raft_consensus.cc:493] T df1248f6dbd94af68c587da7d149e279 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:37.466163 1670 raft_consensus.cc:515] T df1248f6dbd94af68c587da7d149e279 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.467185 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df1248f6dbd94af68c587da7d149e279" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:37.467677 712 leader_election.cc:343] T df1248f6dbd94af68c587da7d149e279 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:37.462018 1169 consensus_queue.cc:260] T 551d07a9242740dcb9366bb2d707a4a8 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.470413 1169 ts_tablet_manager.cc:1434] T 551d07a9242740dcb9366bb2d707a4a8 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20251025 07:52:37.470691 875 tablet_service.cc:1505] Processing CreateTablet for tablet a758a518c37b461f85e7b98e87020fc1 (DEFAULT_TABLE table=default.Table_ab87db32c7024d709eee5476d53f17ce [id=b1458cb701e14086a5a7fd82db235571]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.471105 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a758a518c37b461f85e7b98e87020fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.473243 1169 tablet_bootstrap.cc:492] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:37.473752 744 tablet_service.cc:1505] Processing CreateTablet for tablet f70091024c034c118fb86578db19a31c (DEFAULT_TABLE table=default.Table_332b308220f64c6dbb5f29711d8463b3 [id=e041946c093b4c789ce175c736cc230b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.474248 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f70091024c034c118fb86578db19a31c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.474660 1561 raft_consensus.cc:2749] T abf94c32ff2945119b710905c075d63c P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:37.475950 1169 tablet_bootstrap.cc:654] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:37.476881 903 raft_consensus.cc:2468] T 53198283718d4b45b30df65b5bde96a6 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:37.477694 1166 tablet_bootstrap.cc:492] T 2c50ec95e48e49b59a76fac4fc862d3d P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:37.477861 1166 ts_tablet_manager.cc:1403] T 2c50ec95e48e49b59a76fac4fc862d3d P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.000s
I20251025 07:52:37.479609 1670 leader_election.cc:290] T df1248f6dbd94af68c587da7d149e279 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:37.479904 1166 raft_consensus.cc:359] T 2c50ec95e48e49b59a76fac4fc862d3d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.480062 1166 raft_consensus.cc:385] T 2c50ec95e48e49b59a76fac4fc862d3d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:37.480123 1166 raft_consensus.cc:740] T 2c50ec95e48e49b59a76fac4fc862d3d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:37.480325 1166 consensus_queue.cc:260] T 2c50ec95e48e49b59a76fac4fc862d3d P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.480540 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df1248f6dbd94af68c587da7d149e279" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:37.480804 1166 ts_tablet_manager.cc:1434] T 2c50ec95e48e49b59a76fac4fc862d3d P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20251025 07:52:37.481036 710 leader_election.cc:343] T df1248f6dbd94af68c587da7d149e279 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:37.481141 1166 tablet_bootstrap.cc:492] T 70fe9aa2f58d4abea3b3b71066a29653 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:37.481171 710 leader_election.cc:304] T df1248f6dbd94af68c587da7d149e279 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:37.481699 1980 raft_consensus.cc:2749] T df1248f6dbd94af68c587da7d149e279 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:37.483263 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_a6c869949b744bc98efb735b0e73e7aa [id=bf77b5ff6f1048dd996d948437f9ff5e]). Expected version 2 got 0
I20251025 07:52:37.483578 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3]). Expected version 1 got 0
I20251025 07:52:37.483912 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5]). Expected version 1 got 0
I20251025 07:52:37.484748 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f]). Expected version 1 got 0
I20251025 07:52:37.485201 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7]). Expected version 1 got 0
I20251025 07:52:37.486896 1011 tablet_service.cc:1505] Processing CreateTablet for tablet a758a518c37b461f85e7b98e87020fc1 (DEFAULT_TABLE table=default.Table_ab87db32c7024d709eee5476d53f17ce [id=b1458cb701e14086a5a7fd82db235571]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.487514 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a758a518c37b461f85e7b98e87020fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.488806 884 tablet_service.cc:1505] Processing CreateTablet for tablet f43d68799e824bfa9eabc27bffeda3c7 (DEFAULT_TABLE table=default.Table_f32fe407bbe34695884f8ee432f0c947 [id=1128ec9706ac4d879b18b1037a3a5a91]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.489181 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f43d68799e824bfa9eabc27bffeda3c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.494849 1017 tablet_service.cc:1505] Processing CreateTablet for tablet f43d68799e824bfa9eabc27bffeda3c7 (DEFAULT_TABLE table=default.Table_f32fe407bbe34695884f8ee432f0c947 [id=1128ec9706ac4d879b18b1037a3a5a91]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.495400 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f43d68799e824bfa9eabc27bffeda3c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.502033 1167 tablet_bootstrap.cc:492] T 6a58931e7cd7454fb583f1c70cd2ad15 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:37.502233 1167 ts_tablet_manager.cc:1403] T 6a58931e7cd7454fb583f1c70cd2ad15 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.115s user 0.001s sys 0.008s
I20251025 07:52:37.503000 1167 raft_consensus.cc:359] T 6a58931e7cd7454fb583f1c70cd2ad15 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:37.503193 1167 raft_consensus.cc:385] T 6a58931e7cd7454fb583f1c70cd2ad15 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:37.503281 1167 raft_consensus.cc:740] T 6a58931e7cd7454fb583f1c70cd2ad15 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:37.503518 1167 consensus_queue.cc:260] T 6a58931e7cd7454fb583f1c70cd2ad15 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:37.504009 1167 ts_tablet_manager.cc:1434] T 6a58931e7cd7454fb583f1c70cd2ad15 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:37.506809 1167 tablet_bootstrap.cc:492] T 92aa605fabbc4257a6a204c087e7cc7f P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:37.509258 1167 tablet_bootstrap.cc:654] T 92aa605fabbc4257a6a204c087e7cc7f P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:37.511478 1166 tablet_bootstrap.cc:654] T 70fe9aa2f58d4abea3b3b71066a29653 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:37.512944 748 tablet_service.cc:1505] Processing CreateTablet for tablet f43d68799e824bfa9eabc27bffeda3c7 (DEFAULT_TABLE table=default.Table_f32fe407bbe34695884f8ee432f0c947 [id=1128ec9706ac4d879b18b1037a3a5a91]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.513267 1169 tablet_bootstrap.cc:492] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:37.513355 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f43d68799e824bfa9eabc27bffeda3c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.513536 1169 ts_tablet_manager.cc:1403] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.001s
I20251025 07:52:37.514307 1018 tablet_service.cc:1505] Processing CreateTablet for tablet f65ae042d1da472f9e256c125ce7898a (DEFAULT_TABLE table=default.Table_74277c3abac14953b2eb73c850e3e727 [id=afdcd9fdb01f4cad85633186bed13682]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.514505 1169 raft_consensus.cc:359] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.514691 1169 raft_consensus.cc:385] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:37.514734 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f65ae042d1da472f9e256c125ce7898a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.514781 1169 raft_consensus.cc:740] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:37.515033 1169 consensus_queue.cc:260] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.515616 1169 ts_tablet_manager.cc:1434] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:37.518421 1169 tablet_bootstrap.cc:492] T 6ec421eb820b4f40afcff5a5bf80c508 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:37.520087 1670 consensus_queue.cc:1048] T c88c46e022ff41a58b02538f2151e260 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:37.521221 884 tablet_service.cc:1505] Processing CreateTablet for tablet f65ae042d1da472f9e256c125ce7898a (DEFAULT_TABLE table=default.Table_74277c3abac14953b2eb73c850e3e727 [id=afdcd9fdb01f4cad85633186bed13682]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.521417 1169 tablet_bootstrap.cc:654] T 6ec421eb820b4f40afcff5a5bf80c508 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:37.521725 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f65ae042d1da472f9e256c125ce7898a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.526409 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 04d27d54fdb7483b869408af65ff85c9 (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]). Expected version 2 got 0
I20251025 07:52:37.526863 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 4c155aa7fa5b4d68bbab1918b8432372 (table default.Table_6c5a24747ba14352b0285a41ec2985a4 [id=28bbaad474164a8c8c97a76bb8081791]). Expected version 2 got 0
I20251025 07:52:37.527173 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet cb829649f73d4d3f8862d268c953bd86 (table default.Table_ba4b8cc7b930482690b7e192e0edff12 [id=7944225d3ce7439b881bfb9289d9f040]). Expected version 1 got 0
I20251025 07:52:37.528591 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 062c0f3e51a3487880cea373f8cf1f54 (table default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de]). Expected version 1 got 0
I20251025 07:52:37.529038 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 292fe04055954afd8c08ac2636b68191 (table default.Table_9fa69b10b5e842639fc0b0a58cd9b6a5 [id=a253ce0253bd4da4814741c482193d8f]). Expected version 1 got 0
I20251025 07:52:37.529462 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet aa9d9ab8073945d28d52fb02213f201a (table default.Table_4791672658f043f6b97b9320a531ebb5 [id=dfd20cf9c3e541c4abe23d59e0aa7f2a]). Expected version 1 got 0
I20251025 07:52:37.529904 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet b17d2a9ad6be49d29f2413666fba7e69 (table default.Table_fcddb31989ee4236a4981c12779b3f5d [id=f3794fa081364666b944c412415bdba0]). Expected version 3 got 0
I20251025 07:52:37.530287 747 tablet_service.cc:1505] Processing CreateTablet for tablet f65ae042d1da472f9e256c125ce7898a (DEFAULT_TABLE table=default.Table_74277c3abac14953b2eb73c850e3e727 [id=afdcd9fdb01f4cad85633186bed13682]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.530740 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f65ae042d1da472f9e256c125ce7898a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.530789 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84]). Expected version 1 got 0
I20251025 07:52:37.533798 1166 tablet_bootstrap.cc:492] T 70fe9aa2f58d4abea3b3b71066a29653 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:37.535276 744 tablet_service.cc:1505] Processing CreateTablet for tablet a758a518c37b461f85e7b98e87020fc1 (DEFAULT_TABLE table=default.Table_ab87db32c7024d709eee5476d53f17ce [id=b1458cb701e14086a5a7fd82db235571]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.535696 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a758a518c37b461f85e7b98e87020fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.540294 1018 tablet_service.cc:1505] Processing CreateTablet for tablet c087a467850541c9aecc5c82311682cb (DEFAULT_TABLE table=default.Table_aa4cf07c18a3424f91688e2a2ddc97b9 [id=ba53c5c52b8445fdad649a93edb51204]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.540871 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c087a467850541c9aecc5c82311682cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.544085 1670 raft_consensus.cc:493] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:37.544378 1670 raft_consensus.cc:515] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:37.545699 712 leader_election.cc:343] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 87a93778620f4c84b3b3f7a3d7d7e87f
I20251025 07:52:37.546433 1670 leader_election.cc:290] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:37.546784 710 leader_election.cc:343] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 87a93778620f4c84b3b3f7a3d7d7e87f
I20251025 07:52:37.546908 710 leader_election.cc:304] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:37.547323 1670 raft_consensus.cc:2749] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:37.547767 738 tablet_service.cc:1505] Processing CreateTablet for tablet c087a467850541c9aecc5c82311682cb (DEFAULT_TABLE table=default.Table_aa4cf07c18a3424f91688e2a2ddc97b9 [id=ba53c5c52b8445fdad649a93edb51204]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.548278 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c087a467850541c9aecc5c82311682cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.550711 1166 ts_tablet_manager.cc:1403] T 70fe9aa2f58d4abea3b3b71066a29653 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.070s user 0.000s sys 0.003s
I20251025 07:52:37.551679 1166 raft_consensus.cc:359] T 70fe9aa2f58d4abea3b3b71066a29653 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.551960 1166 raft_consensus.cc:385] T 70fe9aa2f58d4abea3b3b71066a29653 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:37.552034 1166 raft_consensus.cc:740] T 70fe9aa2f58d4abea3b3b71066a29653 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:37.552254 1166 consensus_queue.cc:260] T 70fe9aa2f58d4abea3b3b71066a29653 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.552789 1166 ts_tablet_manager.cc:1434] T 70fe9aa2f58d4abea3b3b71066a29653 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 07:52:37.556257 1167 tablet_bootstrap.cc:492] T 92aa605fabbc4257a6a204c087e7cc7f P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:37.556511 1167 ts_tablet_manager.cc:1403] T 92aa605fabbc4257a6a204c087e7cc7f P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.003s
I20251025 07:52:37.557451 1167 raft_consensus.cc:359] T 92aa605fabbc4257a6a204c087e7cc7f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:37.557708 1167 raft_consensus.cc:385] T 92aa605fabbc4257a6a204c087e7cc7f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:37.557844 1167 raft_consensus.cc:740] T 92aa605fabbc4257a6a204c087e7cc7f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:37.558277 1167 consensus_queue.cc:260] T 92aa605fabbc4257a6a204c087e7cc7f P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:37.538633 884 tablet_service.cc:1505] Processing CreateTablet for tablet c087a467850541c9aecc5c82311682cb (DEFAULT_TABLE table=default.Table_aa4cf07c18a3424f91688e2a2ddc97b9 [id=ba53c5c52b8445fdad649a93edb51204]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.542171 1561 consensus_queue.cc:1048] T 9979cae195e849908677d78c6c33be71 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:37.561075 1166 tablet_bootstrap.cc:492] T 6ec421eb820b4f40afcff5a5bf80c508 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:37.562482 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 5dab439bc50d4e869548154e1d222ad6 (table default.Table_ab6798879a554075b3deaa059ad077c7 [id=45325c188eac4d34bb016d01de091f87]). Expected version 1 got 0
I20251025 07:52:37.562830 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_87d38832f85e4965bfab1a077990114f [id=ad812862698d421184ebf501ce79b7be]). Expected version 2 got 0
I20251025 07:52:37.563262 452 catalog_manager.cc:5649] T 87268e27540c4ec58ce1e62df819f19d P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } }
I20251025 07:52:37.563988 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 87268e27540c4ec58ce1e62df819f19d (table default.Table_350383f0e98947eba77186e74ad88781 [id=96b0e47c666c4d8f982741aea7b17ae7]). Expected version 2 got 0
I20251025 07:52:37.564903 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 6897405bbc1c47a4ab0aa3af80e60e3d (table default.Table_92c0120dd29348618040150dedb14c43 [id=ea892976886644d8aa025ae2edef718d]). Expected version 2 got 0
I20251025 07:52:37.565702 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]). Expected version 1 got 0
I20251025 07:52:37.565881 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c087a467850541c9aecc5c82311682cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.566367 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 0325d462441349ef81841f1b0cab2a20 (table default.Table_750ecb905839434fa9a1aa72f87394be [id=9ce3ba77c56a4ef09cb3b40ea0e488f6]). Expected version 1 got 0
I20251025 07:52:37.569797 1166 tablet_bootstrap.cc:654] T 6ec421eb820b4f40afcff5a5bf80c508 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:37.571255 1011 tablet_service.cc:1505] Processing CreateTablet for tablet b928bc638aaa4700ba7a6478a5097c5b (DEFAULT_TABLE table=default.Table_d5ae5bc2cb804015ba50415cd3f62847 [id=951b49e5f0ca41488c2e92be79288158]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.571663 1612 raft_consensus.cc:493] T ef1854d35a3047e3a9fbe0166ab1295a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:37.571735 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b928bc638aaa4700ba7a6478a5097c5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.571835 1612 raft_consensus.cc:515] T ef1854d35a3047e3a9fbe0166ab1295a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.571971 893 tablet_service.cc:1505] Processing CreateTablet for tablet b928bc638aaa4700ba7a6478a5097c5b (DEFAULT_TABLE table=default.Table_d5ae5bc2cb804015ba50415cd3f62847 [id=951b49e5f0ca41488c2e92be79288158]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.572306 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b928bc638aaa4700ba7a6478a5097c5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.573086 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef1854d35a3047e3a9fbe0166ab1295a" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:37.573361 776 raft_consensus.cc:2468] T ef1854d35a3047e3a9fbe0166ab1295a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:37.573917 1612 leader_election.cc:290] T ef1854d35a3047e3a9fbe0166ab1295a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:37.574692 982 leader_election.cc:304] T ef1854d35a3047e3a9fbe0166ab1295a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:37.575084 1671 raft_consensus.cc:2804] T ef1854d35a3047e3a9fbe0166ab1295a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:37.575181 1671 raft_consensus.cc:493] T ef1854d35a3047e3a9fbe0166ab1295a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:37.575299 1671 raft_consensus.cc:3060] T ef1854d35a3047e3a9fbe0166ab1295a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.577019 738 tablet_service.cc:1505] Processing CreateTablet for tablet b928bc638aaa4700ba7a6478a5097c5b (DEFAULT_TABLE table=default.Table_d5ae5bc2cb804015ba50415cd3f62847 [id=951b49e5f0ca41488c2e92be79288158]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.577450 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b928bc638aaa4700ba7a6478a5097c5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.580067 1167 ts_tablet_manager.cc:1434] T 92aa605fabbc4257a6a204c087e7cc7f P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20251025 07:52:37.580471 1671 raft_consensus.cc:515] T ef1854d35a3047e3a9fbe0166ab1295a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.582013 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef1854d35a3047e3a9fbe0166ab1295a" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:37.582278 776 raft_consensus.cc:3060] T ef1854d35a3047e3a9fbe0166ab1295a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.583264 1167 tablet_bootstrap.cc:492] T 80ee75935d7e461d9d4066fac9819bef P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:37.584664 1671 leader_election.cc:290] T ef1854d35a3047e3a9fbe0166ab1295a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:37.585664 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef1854d35a3047e3a9fbe0166ab1295a" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:37.585659 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef1854d35a3047e3a9fbe0166ab1295a" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
W20251025 07:52:37.586922 983 leader_election.cc:343] T ef1854d35a3047e3a9fbe0166ab1295a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:37.588100 1670 consensus_queue.cc:1048] T c88c46e022ff41a58b02538f2151e260 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:37.592801 1561 consensus_queue.cc:1048] T 9979cae195e849908677d78c6c33be71 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:37.593794 1017 tablet_service.cc:1505] Processing CreateTablet for tablet 80124cae4fb64858a1e39f99c5dd0419 (DEFAULT_TABLE table=default.Table_68c19f91d7094d5ab0ab4f77f74a937e [id=867342b181604482af90da2976ef21a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.594446 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80124cae4fb64858a1e39f99c5dd0419. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.595402 888 tablet_service.cc:1505] Processing CreateTablet for tablet 99ea11a2729442918a0a6485d0d83cac (DEFAULT_TABLE table=default.Table_a0665a4cb38b40109df3991cb2f71ed6 [id=dbd92b3210f24fc2b8146dcfcb18833a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.595785 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99ea11a2729442918a0a6485d0d83cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.601924 740 tablet_service.cc:1505] Processing CreateTablet for tablet 80124cae4fb64858a1e39f99c5dd0419 (DEFAULT_TABLE table=default.Table_68c19f91d7094d5ab0ab4f77f74a937e [id=867342b181604482af90da2976ef21a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.602414 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80124cae4fb64858a1e39f99c5dd0419. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.604735 1169 tablet_bootstrap.cc:492] T 6ec421eb820b4f40afcff5a5bf80c508 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:37.604944 1169 ts_tablet_manager.cc:1403] T 6ec421eb820b4f40afcff5a5bf80c508 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.087s user 0.004s sys 0.000s
I20251025 07:52:37.605994 1169 raft_consensus.cc:359] T 6ec421eb820b4f40afcff5a5bf80c508 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.606285 1169 raft_consensus.cc:385] T 6ec421eb820b4f40afcff5a5bf80c508 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:37.606369 1169 raft_consensus.cc:740] T 6ec421eb820b4f40afcff5a5bf80c508 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:37.606721 1169 consensus_queue.cc:260] T 6ec421eb820b4f40afcff5a5bf80c508 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.608121 1169 ts_tablet_manager.cc:1434] T 6ec421eb820b4f40afcff5a5bf80c508 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251025 07:52:37.608561 875 tablet_service.cc:1505] Processing CreateTablet for tablet 80124cae4fb64858a1e39f99c5dd0419 (DEFAULT_TABLE table=default.Table_68c19f91d7094d5ab0ab4f77f74a937e [id=867342b181604482af90da2976ef21a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.609757 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80124cae4fb64858a1e39f99c5dd0419. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.610237 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 99ea11a2729442918a0a6485d0d83cac (DEFAULT_TABLE table=default.Table_a0665a4cb38b40109df3991cb2f71ed6 [id=dbd92b3210f24fc2b8146dcfcb18833a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.613397 744 tablet_service.cc:1505] Processing CreateTablet for tablet 99ea11a2729442918a0a6485d0d83cac (DEFAULT_TABLE table=default.Table_a0665a4cb38b40109df3991cb2f71ed6 [id=dbd92b3210f24fc2b8146dcfcb18833a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.613843 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99ea11a2729442918a0a6485d0d83cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.613832 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_e90d7f5ff13b40878e5b625c97792c7d [id=fc8180c09d7b4e74b2e161f7acb8fe0e]). Expected version 1 got 0
I20251025 07:52:37.614509 1561 raft_consensus.cc:493] T 2c50ec95e48e49b59a76fac4fc862d3d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:37.614651 1561 raft_consensus.cc:515] T 2c50ec95e48e49b59a76fac4fc862d3d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.615543 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3]). Expected version 2 got 0
I20251025 07:52:37.616019 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c50ec95e48e49b59a76fac4fc862d3d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:37.616129 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 8f2c8582fed34c848d9903b8ad8b4057 (table default.Table_438f67f3dc62465999a917986922b3ed [id=7c7fcd265e624d7fab74661db505d9c3]). Expected version 1 got 0
I20251025 07:52:37.616247 774 raft_consensus.cc:2468] T 2c50ec95e48e49b59a76fac4fc862d3d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:37.616673 847 leader_election.cc:304] T 2c50ec95e48e49b59a76fac4fc862d3d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:37.616945 1707 raft_consensus.cc:2804] T 2c50ec95e48e49b59a76fac4fc862d3d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:37.617033 1707 raft_consensus.cc:493] T 2c50ec95e48e49b59a76fac4fc862d3d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:37.617111 1707 raft_consensus.cc:3060] T 2c50ec95e48e49b59a76fac4fc862d3d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.617147 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236]). Expected version 1 got 0
I20251025 07:52:37.617525 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c]). Expected version 1 got 0
I20251025 07:52:37.618206 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]). Expected version 3 got 0
I20251025 07:52:37.618515 894 tablet_service.cc:1505] Processing CreateTablet for tablet a8fddd7cd7f543b190fb608a331bb3ac (DEFAULT_TABLE table=default.Table_58bd2fb570eb4cfebf3ce98477e23525 [id=f4668f4d18304a619724e52d76d6c915]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.618882 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8fddd7cd7f543b190fb608a331bb3ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.619731 1707 raft_consensus.cc:515] T 2c50ec95e48e49b59a76fac4fc862d3d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.620697 1707 leader_election.cc:290] T 2c50ec95e48e49b59a76fac4fc862d3d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:37.621337 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c50ec95e48e49b59a76fac4fc862d3d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:37.621562 774 raft_consensus.cc:3060] T 2c50ec95e48e49b59a76fac4fc862d3d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.625140 774 raft_consensus.cc:2468] T 2c50ec95e48e49b59a76fac4fc862d3d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:37.625677 847 leader_election.cc:304] T 2c50ec95e48e49b59a76fac4fc862d3d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:37.626214 1838 raft_consensus.cc:2804] T 2c50ec95e48e49b59a76fac4fc862d3d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:37.626740 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525]). Expected version 1 got 0
I20251025 07:52:37.627446 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99ea11a2729442918a0a6485d0d83cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.633265 1166 tablet_bootstrap.cc:492] T 6ec421eb820b4f40afcff5a5bf80c508 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:37.633468 1166 ts_tablet_manager.cc:1403] T 6ec421eb820b4f40afcff5a5bf80c508 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.073s user 0.005s sys 0.000s
I20251025 07:52:37.634557 1166 raft_consensus.cc:359] T 6ec421eb820b4f40afcff5a5bf80c508 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.634794 1166 raft_consensus.cc:385] T 6ec421eb820b4f40afcff5a5bf80c508 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:37.634862 1166 raft_consensus.cc:740] T 6ec421eb820b4f40afcff5a5bf80c508 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:37.635106 1166 consensus_queue.cc:260] T 6ec421eb820b4f40afcff5a5bf80c508 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.635552 1166 ts_tablet_manager.cc:1434] T 6ec421eb820b4f40afcff5a5bf80c508 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:37.635867 1166 tablet_bootstrap.cc:492] T 551d07a9242740dcb9366bb2d707a4a8 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:37.637221 1980 consensus_queue.cc:1048] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:37.638043 1166 tablet_bootstrap.cc:654] T 551d07a9242740dcb9366bb2d707a4a8 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:37.642402 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 36262ab8c6964dd684c01b4db6219a4f (table default.Table_76158f8cf82e4cdb9a90580e46aaea48 [id=f249f30497bc450c887d659721469de8]). Expected version 1 got 0
I20251025 07:52:37.643168 776 raft_consensus.cc:2468] T ef1854d35a3047e3a9fbe0166ab1295a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:37.643224 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 2c50ec95e48e49b59a76fac4fc862d3d (table default.Table_32b54e82cbc84d39a0020826f8737d62 [id=c28b0b6acd61490caecfdaab803a9d92]). Expected version 2 got 0
I20251025 07:52:37.643803 1561 leader_election.cc:290] T 2c50ec95e48e49b59a76fac4fc862d3d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:37.643842 982 leader_election.cc:304] T ef1854d35a3047e3a9fbe0166ab1295a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters: 3aea2869ff814cc7b142012faa274382
I20251025 07:52:37.644151 1671 raft_consensus.cc:2804] T ef1854d35a3047e3a9fbe0166ab1295a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:37.644297 1671 raft_consensus.cc:697] T ef1854d35a3047e3a9fbe0166ab1295a P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:37.644506 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet b97e39aaac3048ed93dbfe873b757254 (table default.Table_99fd36369f9a4e0fb14f234c98635203 [id=c8059ab6cec146efa631d06799d969a9]). Expected version 1 got 0
I20251025 07:52:37.644578 1671 consensus_queue.cc:237] T ef1854d35a3047e3a9fbe0166ab1295a P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.644799 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 8e6d76b1a97945cfb6c3055e1b622f8a (table default.Table_10e57b42f97544208675708b1e028846 [id=f5c1be203e2b4831a025871cf8a2ef17]). Expected version 2 got 0
I20251025 07:52:37.645463 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 509c94f2a422493ea4390ecee576db72 (table default.Table_3f081b7bb4e1420c8121ea4115439f3d [id=696a1285fbfe4038a8b1b38708c8ea49]). Expected version 1 got 0
I20251025 07:52:37.645619 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet ab42909afc524f2fa3c1041495d13a59 (table default.Table_7eb8424d5192485da1a2d67bfdb2bbbf [id=40c29df99590408cbd5c583d4fc741eb]). Expected version 2 got 0
I20251025 07:52:37.647724 1169 tablet_bootstrap.cc:492] T 5361f08f50b745eb98bf02e979d7c71e P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:37.650214 1016 tablet_service.cc:1505] Processing CreateTablet for tablet a8fddd7cd7f543b190fb608a331bb3ac (DEFAULT_TABLE table=default.Table_58bd2fb570eb4cfebf3ce98477e23525 [id=f4668f4d18304a619724e52d76d6c915]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.650656 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8fddd7cd7f543b190fb608a331bb3ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.652806 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e]). Expected version 1 got 0
I20251025 07:52:37.653071 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet bc2d16b33ca8426abbd7bd38339c9f1d (table default.Table_4ea985df4c5b415c9570c64506a98cc4 [id=98d2b7797e33485e8872054aa2eb8ec9]). Expected version 1 got 0
I20251025 07:52:37.653447 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet ef1854d35a3047e3a9fbe0166ab1295a (table default.Table_088aad2a46b041bfa29ed8e6929760b1 [id=834669c54dae449790818684fd8659bc]). Expected version 2 got 0
I20251025 07:52:37.653841 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]). Expected version 1 got 0
I20251025 07:52:37.654008 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet ef9a2c9f2ccf432ab835396020872208 (table default.Table_2b9b772d4c234ac286bef3658482e490 [id=409fab60f9d744798e6f92b868f0f7c2]). Expected version 1 got 0
I20251025 07:52:37.654181 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]). Expected version 1 got 0
I20251025 07:52:37.655586 1659 raft_consensus.cc:493] T 92aa605fabbc4257a6a204c087e7cc7f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:37.655771 1659 raft_consensus.cc:515] T 92aa605fabbc4257a6a204c087e7cc7f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:37.656371 1659 leader_election.cc:290] T 92aa605fabbc4257a6a204c087e7cc7f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:37.657385 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "92aa605fabbc4257a6a204c087e7cc7f" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:37.658682 712 leader_election.cc:343] T 92aa605fabbc4257a6a204c087e7cc7f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:37.659482 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "92aa605fabbc4257a6a204c087e7cc7f" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:37.660645 710 leader_election.cc:343] T 92aa605fabbc4257a6a204c087e7cc7f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:37.660849 710 leader_election.cc:304] T 92aa605fabbc4257a6a204c087e7cc7f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:37.661433 1980 raft_consensus.cc:2749] T 92aa605fabbc4257a6a204c087e7cc7f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:37.663450 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c50ec95e48e49b59a76fac4fc862d3d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:37.663762 1045 raft_consensus.cc:3060] T 2c50ec95e48e49b59a76fac4fc862d3d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.676764 740 tablet_service.cc:1505] Processing CreateTablet for tablet a8fddd7cd7f543b190fb608a331bb3ac (DEFAULT_TABLE table=default.Table_58bd2fb570eb4cfebf3ce98477e23525 [id=f4668f4d18304a619724e52d76d6c915]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.678514 1166 tablet_bootstrap.cc:492] T 551d07a9242740dcb9366bb2d707a4a8 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:37.678687 1166 ts_tablet_manager.cc:1403] T 551d07a9242740dcb9366bb2d707a4a8 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.000s
I20251025 07:52:37.679400 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8fddd7cd7f543b190fb608a331bb3ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.679723 1045 raft_consensus.cc:2468] T 2c50ec95e48e49b59a76fac4fc862d3d P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:37.679740 1166 raft_consensus.cc:359] T 551d07a9242740dcb9366bb2d707a4a8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.679978 1166 raft_consensus.cc:385] T 551d07a9242740dcb9366bb2d707a4a8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:37.680051 1166 raft_consensus.cc:740] T 551d07a9242740dcb9366bb2d707a4a8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:37.680281 1166 consensus_queue.cc:260] T 551d07a9242740dcb9366bb2d707a4a8 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.680907 1166 ts_tablet_manager.cc:1434] T 551d07a9242740dcb9366bb2d707a4a8 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 07:52:37.681288 1166 tablet_bootstrap.cc:492] T ef1854d35a3047e3a9fbe0166ab1295a P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:37.685474 1167 tablet_bootstrap.cc:654] T 80ee75935d7e461d9d4066fac9819bef P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:37.693373 1169 tablet_bootstrap.cc:654] T 5361f08f50b745eb98bf02e979d7c71e P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:37.697494 1017 tablet_service.cc:1505] Processing CreateTablet for tablet 872cb7512d104827b1f1f186d33a4a44 (DEFAULT_TABLE table=default.Table_d5c65dce8b1b4214ba519d964012bfc0 [id=d4be274b36a845a99a4b96b8846248fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.698060 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 872cb7512d104827b1f1f186d33a4a44. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.702064 1169 tablet_bootstrap.cc:492] T 5361f08f50b745eb98bf02e979d7c71e P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:37.702263 1169 ts_tablet_manager.cc:1403] T 5361f08f50b745eb98bf02e979d7c71e P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.055s user 0.004s sys 0.000s
I20251025 07:52:37.703186 1169 raft_consensus.cc:359] T 5361f08f50b745eb98bf02e979d7c71e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.703382 1169 raft_consensus.cc:385] T 5361f08f50b745eb98bf02e979d7c71e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:37.703480 1169 raft_consensus.cc:740] T 5361f08f50b745eb98bf02e979d7c71e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:37.704875 1011 tablet_service.cc:1505] Processing CreateTablet for tablet f33ff0d79e8846c5b58453f097d2ea88 (DEFAULT_TABLE table=default.Table_6eb139482bc940989a606a0d8745d54f [id=616b3a99164f47fbb918c4e91d0c2e14]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.705299 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f33ff0d79e8846c5b58453f097d2ea88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.709165 1018 tablet_service.cc:1505] Processing CreateTablet for tablet 3a0f50c8000144a4acb5219484b01800 (DEFAULT_TABLE table=default.Table_646f48a21c194af3a1611e5e12b2e11b [id=6f395892b09240fea02482b3199cfb5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.709661 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a0f50c8000144a4acb5219484b01800. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.717480 1169 consensus_queue.cc:260] T 5361f08f50b745eb98bf02e979d7c71e P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.690060 894 tablet_service.cc:1505] Processing CreateTablet for tablet 872cb7512d104827b1f1f186d33a4a44 (DEFAULT_TABLE table=default.Table_d5c65dce8b1b4214ba519d964012bfc0 [id=d4be274b36a845a99a4b96b8846248fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.722615 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 872cb7512d104827b1f1f186d33a4a44. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.728595 1166 tablet_bootstrap.cc:654] T ef1854d35a3047e3a9fbe0166ab1295a P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:37.699664 884 tablet_service.cc:1505] Processing CreateTablet for tablet f33ff0d79e8846c5b58453f097d2ea88 (DEFAULT_TABLE table=default.Table_6eb139482bc940989a606a0d8745d54f [id=616b3a99164f47fbb918c4e91d0c2e14]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.708377 893 tablet_service.cc:1505] Processing CreateTablet for tablet 3a0f50c8000144a4acb5219484b01800 (DEFAULT_TABLE table=default.Table_646f48a21c194af3a1611e5e12b2e11b [id=6f395892b09240fea02482b3199cfb5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.734622 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a0f50c8000144a4acb5219484b01800. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.736979 1980 raft_consensus.cc:493] T 5fca357d7e24494fb6af4bd73d98833c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:37.737236 1980 raft_consensus.cc:515] T 5fca357d7e24494fb6af4bd73d98833c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.738306 1980 leader_election.cc:290] T 5fca357d7e24494fb6af4bd73d98833c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:37.738474 1980 raft_consensus.cc:493] T 3abfd52327a04ba88625f391a8658fbb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:37.739195 748 tablet_service.cc:1505] Processing CreateTablet for tablet 872cb7512d104827b1f1f186d33a4a44 (DEFAULT_TABLE table=default.Table_d5c65dce8b1b4214ba519d964012bfc0 [id=d4be274b36a845a99a4b96b8846248fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.739554 1980 raft_consensus.cc:515] T 3abfd52327a04ba88625f391a8658fbb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:37.743829 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f33ff0d79e8846c5b58453f097d2ea88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.744277 1980 leader_election.cc:290] T 3abfd52327a04ba88625f391a8658fbb P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:37.745010 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 872cb7512d104827b1f1f186d33a4a44. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.716825 1838 raft_consensus.cc:697] T 2c50ec95e48e49b59a76fac4fc862d3d P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
W20251025 07:52:37.717211 850 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:45417, user_credentials={real_user=slave}} blocked reactor thread for 89863.1us
I20251025 07:52:37.748049 1838 consensus_queue.cc:237] T 2c50ec95e48e49b59a76fac4fc862d3d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.751600 2106 consensus_queue.cc:1048] T c59f6e6eb9594a8e83f109a09b034045 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:37.764410 1166 tablet_bootstrap.cc:492] T ef1854d35a3047e3a9fbe0166ab1295a P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:37.764564 1166 ts_tablet_manager.cc:1403] T ef1854d35a3047e3a9fbe0166ab1295a P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.083s user 0.002s sys 0.002s
I20251025 07:52:37.765226 1166 raft_consensus.cc:359] T ef1854d35a3047e3a9fbe0166ab1295a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.765388 1166 raft_consensus.cc:385] T ef1854d35a3047e3a9fbe0166ab1295a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:37.765437 1166 raft_consensus.cc:740] T ef1854d35a3047e3a9fbe0166ab1295a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:37.774365 1166 consensus_queue.cc:260] T ef1854d35a3047e3a9fbe0166ab1295a P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.778971 1166 ts_tablet_manager.cc:1434] T ef1854d35a3047e3a9fbe0166ab1295a P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20251025 07:52:37.779244 1166 tablet_bootstrap.cc:492] T 410ef150a7014c6c867a021b544a2f4e P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:37.780939 1166 tablet_bootstrap.cc:654] T 410ef150a7014c6c867a021b544a2f4e P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:37.781906 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:43563, user_credentials={real_user=slave}} blocked reactor thread for 61856us
I20251025 07:52:37.782524 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5fca357d7e24494fb6af4bd73d98833c" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:37.782894 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3abfd52327a04ba88625f391a8658fbb" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:37.783764 1980 raft_consensus.cc:493] T 8f2c8582fed34c848d9903b8ad8b4057 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:37.783923 1980 raft_consensus.cc:515] T 8f2c8582fed34c848d9903b8ad8b4057 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.784572 1980 leader_election.cc:290] T 8f2c8582fed34c848d9903b8ad8b4057 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:37.785887 710 leader_election.cc:343] T 5fca357d7e24494fb6af4bd73d98833c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:37.786131 710 leader_election.cc:343] T 3abfd52327a04ba88625f391a8658fbb P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:37.786473 1980 consensus_queue.cc:1048] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251025 07:52:37.787096 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:45417, user_credentials={real_user=slave}} blocked reactor thread for 68776.8us
I20251025 07:52:37.787250 2105 consensus_queue.cc:1048] T db99a1e68daf4e948a3d66ad276cc5f9 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:37.789503 894 tablet_service.cc:1505] Processing CreateTablet for tablet bae843ed66fb4dcc8af9ab55693439ac (DEFAULT_TABLE table=default.Table_8d0e78b0bb8d433f9ed8e49d24d7e95d [id=8be005a6c0ba404da6ebed5da21f7498]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.790007 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bae843ed66fb4dcc8af9ab55693439ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.794041 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f2c8582fed34c848d9903b8ad8b4057" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:37.742195 2050 raft_consensus.cc:493] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:37.794293 2050 raft_consensus.cc:515] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.794937 2050 leader_election.cc:290] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:37.749387 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c50ec95e48e49b59a76fac4fc862d3d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:37.795156 1045 raft_consensus.cc:2376] T 2c50ec95e48e49b59a76fac4fc862d3d P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1. Re-sending same reply.
I20251025 07:52:37.788441 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3abfd52327a04ba88625f391a8658fbb" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:37.788807 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f2c8582fed34c848d9903b8ad8b4057" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:37.792470 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5fca357d7e24494fb6af4bd73d98833c" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:37.795984 1044 raft_consensus.cc:2468] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:37.796263 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5fca357d7e24494fb6af4bd73d98833c" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:37.796469 776 raft_consensus.cc:2468] T 5fca357d7e24494fb6af4bd73d98833c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:37.796883 982 leader_election.cc:304] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:37.797235 2050 raft_consensus.cc:2804] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:37.797322 2050 raft_consensus.cc:493] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:37.797430 2050 raft_consensus.cc:3060] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.800905 2050 raft_consensus.cc:515] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.801388 1017 tablet_service.cc:1505] Processing CreateTablet for tablet bae843ed66fb4dcc8af9ab55693439ac (DEFAULT_TABLE table=default.Table_8d0e78b0bb8d433f9ed8e49d24d7e95d [id=8be005a6c0ba404da6ebed5da21f7498]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.801796 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bae843ed66fb4dcc8af9ab55693439ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.802088 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5fca357d7e24494fb6af4bd73d98833c" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:37.802349 776 raft_consensus.cc:3060] T 5fca357d7e24494fb6af4bd73d98833c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.805060 776 raft_consensus.cc:2468] T 5fca357d7e24494fb6af4bd73d98833c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:37.805963 982 leader_election.cc:304] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:37.806727 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5fca357d7e24494fb6af4bd73d98833c" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:37.807286 983 leader_election.cc:343] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:37.807361 2050 leader_election.cc:290] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:37.807577 2050 raft_consensus.cc:2804] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:37.807880 2050 raft_consensus.cc:697] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:37.807940 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5fca357d7e24494fb6af4bd73d98833c" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
W20251025 07:52:37.808491 983 leader_election.cc:343] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:37.808763 2050 consensus_queue.cc:237] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.810130 1838 consensus_queue.cc:1048] T db99a1e68daf4e948a3d66ad276cc5f9 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:37.810982 1569 raft_consensus.cc:493] T 509c94f2a422493ea4390ecee576db72 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:37.811170 1569 raft_consensus.cc:515] T 509c94f2a422493ea4390ecee576db72 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.811586 1169 ts_tablet_manager.cc:1434] T 5361f08f50b745eb98bf02e979d7c71e P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.109s user 0.002s sys 0.000s
I20251025 07:52:37.811776 1569 leader_election.cc:290] T 509c94f2a422493ea4390ecee576db72 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:37.812050 1169 tablet_bootstrap.cc:492] T 3abfd52327a04ba88625f391a8658fbb P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:37.813015 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "509c94f2a422493ea4390ecee576db72" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:37.813160 910 raft_consensus.cc:2468] T 509c94f2a422493ea4390ecee576db72 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:37.813669 983 leader_election.cc:304] T 509c94f2a422493ea4390ecee576db72 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:37.813958 1569 raft_consensus.cc:2804] T 509c94f2a422493ea4390ecee576db72 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:37.814062 1569 raft_consensus.cc:493] T 509c94f2a422493ea4390ecee576db72 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:37.814165 1569 raft_consensus.cc:3060] T 509c94f2a422493ea4390ecee576db72 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.817809 1569 raft_consensus.cc:515] T 509c94f2a422493ea4390ecee576db72 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.819109 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "509c94f2a422493ea4390ecee576db72" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:37.819341 776 raft_consensus.cc:2468] T 509c94f2a422493ea4390ecee576db72 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:37.820324 1569 leader_election.cc:290] T 509c94f2a422493ea4390ecee576db72 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:37.820932 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "509c94f2a422493ea4390ecee576db72" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:37.820935 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "509c94f2a422493ea4390ecee576db72" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:37.821116 910 raft_consensus.cc:3060] T 509c94f2a422493ea4390ecee576db72 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.821142 776 raft_consensus.cc:3060] T 509c94f2a422493ea4390ecee576db72 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.824084 776 raft_consensus.cc:2468] T 509c94f2a422493ea4390ecee576db72 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:37.824343 910 raft_consensus.cc:2468] T 509c94f2a422493ea4390ecee576db72 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:37.824499 982 leader_election.cc:304] T 509c94f2a422493ea4390ecee576db72 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:37.824766 1569 raft_consensus.cc:2804] T 509c94f2a422493ea4390ecee576db72 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:37.824941 1569 raft_consensus.cc:697] T 509c94f2a422493ea4390ecee576db72 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:37.819411 1167 tablet_bootstrap.cc:492] T 80ee75935d7e461d9d4066fac9819bef P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:37.825213 1167 ts_tablet_manager.cc:1403] T 80ee75935d7e461d9d4066fac9819bef P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.242s user 0.003s sys 0.004s
I20251025 07:52:37.825681 1569 consensus_queue.cc:237] T 509c94f2a422493ea4390ecee576db72 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.826572 744 tablet_service.cc:1505] Processing CreateTablet for tablet f33ff0d79e8846c5b58453f097d2ea88 (DEFAULT_TABLE table=default.Table_6eb139482bc940989a606a0d8745d54f [id=616b3a99164f47fbb918c4e91d0c2e14]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.827092 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f33ff0d79e8846c5b58453f097d2ea88. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:37.830580 712 leader_election.cc:343] T 3abfd52327a04ba88625f391a8658fbb P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:37.837419 1169 tablet_bootstrap.cc:654] T 3abfd52327a04ba88625f391a8658fbb P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:37.837675 712 leader_election.cc:304] T 3abfd52327a04ba88625f391a8658fbb P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:37.839118 712 leader_election.cc:343] T 8f2c8582fed34c848d9903b8ad8b4057 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:37.840281 712 leader_election.cc:304] T 5fca357d7e24494fb6af4bd73d98833c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters: 3aea2869ff814cc7b142012faa274382
I20251025 07:52:37.835708 1167 raft_consensus.cc:359] T 80ee75935d7e461d9d4066fac9819bef P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:37.841254 1167 raft_consensus.cc:385] T 80ee75935d7e461d9d4066fac9819bef P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:37.841423 1167 raft_consensus.cc:740] T 80ee75935d7e461d9d4066fac9819bef P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:37.841729 1167 consensus_queue.cc:260] T 80ee75935d7e461d9d4066fac9819bef P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:37.838346 740 tablet_service.cc:1505] Processing CreateTablet for tablet 3a0f50c8000144a4acb5219484b01800 (DEFAULT_TABLE table=default.Table_646f48a21c194af3a1611e5e12b2e11b [id=6f395892b09240fea02482b3199cfb5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.842505 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a0f50c8000144a4acb5219484b01800. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.831810 2117 consensus_queue.cc:1048] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20251025 07:52:37.832304 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:43563, user_credentials={real_user=slave}} blocked reactor thread for 45883.9us
I20251025 07:52:37.830858 2122 consensus_queue.cc:1048] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:37.839445 1670 raft_consensus.cc:2749] T 3abfd52327a04ba88625f391a8658fbb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:37.851020 710 leader_election.cc:343] T 8f2c8582fed34c848d9903b8ad8b4057 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:37.854843 710 leader_election.cc:304] T 8f2c8582fed34c848d9903b8ad8b4057 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:37.859539 1659 raft_consensus.cc:2764] T 5fca357d7e24494fb6af4bd73d98833c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20251025 07:52:37.860528 738 tablet_service.cc:1505] Processing CreateTablet for tablet bae843ed66fb4dcc8af9ab55693439ac (DEFAULT_TABLE table=default.Table_8d0e78b0bb8d433f9ed8e49d24d7e95d [id=8be005a6c0ba404da6ebed5da21f7498]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:37.860949 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bae843ed66fb4dcc8af9ab55693439ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:37.869132 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f]). Expected version 1 got 0
I20251025 07:52:37.869535 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 4c155aa7fa5b4d68bbab1918b8432372 (table default.Table_6c5a24747ba14352b0285a41ec2985a4 [id=28bbaad474164a8c8c97a76bb8081791]). Expected version 2 got 0
I20251025 07:52:37.871585 1659 raft_consensus.cc:2749] T 8f2c8582fed34c848d9903b8ad8b4057 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:37.873623 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7]). Expected version 1 got 0
I20251025 07:52:37.873898 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 04d27d54fdb7483b869408af65ff85c9 (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]). Expected version 2 got 0
I20251025 07:52:37.874159 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet cb829649f73d4d3f8862d268c953bd86 (table default.Table_ba4b8cc7b930482690b7e192e0edff12 [id=7944225d3ce7439b881bfb9289d9f040]). Expected version 1 got 0
I20251025 07:52:37.874794 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 062c0f3e51a3487880cea373f8cf1f54 (table default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de]). Expected version 1 got 0
I20251025 07:52:37.875535 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet aa9d9ab8073945d28d52fb02213f201a (table default.Table_4791672658f043f6b97b9320a531ebb5 [id=dfd20cf9c3e541c4abe23d59e0aa7f2a]). Expected version 1 got 0
I20251025 07:52:37.878360 1167 ts_tablet_manager.cc:1434] T 80ee75935d7e461d9d4066fac9819bef P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.053s user 0.002s sys 0.000s
I20251025 07:52:37.866194 1569 consensus_queue.cc:1048] T d5e5e5bbcf9f43118eea502b69880af5 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:37.894878 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]). Expected version 1 got 0
I20251025 07:52:37.895109 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet ef9a2c9f2ccf432ab835396020872208 (table default.Table_2b9b772d4c234ac286bef3658482e490 [id=409fab60f9d744798e6f92b868f0f7c2]). Expected version 1 got 0
I20251025 07:52:37.895308 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]). Expected version 1 got 0
I20251025 07:52:37.895510 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_a6c869949b744bc98efb735b0e73e7aa [id=bf77b5ff6f1048dd996d948437f9ff5e]). Expected version 2 got 0
I20251025 07:52:37.895854 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet ef1854d35a3047e3a9fbe0166ab1295a (table default.Table_088aad2a46b041bfa29ed8e6929760b1 [id=834669c54dae449790818684fd8659bc]). Expected version 2 got 0
I20251025 07:52:37.896109 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet bc2d16b33ca8426abbd7bd38339c9f1d (table default.Table_4ea985df4c5b415c9570c64506a98cc4 [id=98d2b7797e33485e8872054aa2eb8ec9]). Expected version 1 got 0
I20251025 07:52:37.896342 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e]). Expected version 1 got 0
I20251025 07:52:37.896492 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet ab42909afc524f2fa3c1041495d13a59 (table default.Table_7eb8424d5192485da1a2d67bfdb2bbbf [id=40c29df99590408cbd5c583d4fc741eb]). Expected version 2 got 0
I20251025 07:52:37.896961 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525]). Expected version 1 got 0
I20251025 07:52:37.897128 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84]). Expected version 1 got 0
I20251025 07:52:37.897351 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 5dab439bc50d4e869548154e1d222ad6 (table default.Table_ab6798879a554075b3deaa059ad077c7 [id=45325c188eac4d34bb016d01de091f87]). Expected version 1 got 0
I20251025 07:52:37.897591 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_87d38832f85e4965bfab1a077990114f [id=ad812862698d421184ebf501ce79b7be]). Expected version 2 got 0
I20251025 07:52:37.898288 1167 tablet_bootstrap.cc:492] T ef9a2c9f2ccf432ab835396020872208 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:37.900300 1838 consensus_queue.cc:1048] T c59f6e6eb9594a8e83f109a09b034045 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:37.898125 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]). Expected version 1 got 0
I20251025 07:52:37.904987 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5]). Expected version 1 got 0
I20251025 07:52:37.905557 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 6897405bbc1c47a4ab0aa3af80e60e3d (table default.Table_92c0120dd29348618040150dedb14c43 [id=ea892976886644d8aa025ae2edef718d]). Expected version 2 got 0
I20251025 07:52:37.907847 2105 raft_consensus.cc:493] T 70fe9aa2f58d4abea3b3b71066a29653 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:37.907989 2105 raft_consensus.cc:515] T 70fe9aa2f58d4abea3b3b71066a29653 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.908535 2105 leader_election.cc:290] T 70fe9aa2f58d4abea3b3b71066a29653 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:37.909670 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 0325d462441349ef81841f1b0cab2a20 (table default.Table_750ecb905839434fa9a1aa72f87394be [id=9ce3ba77c56a4ef09cb3b40ea0e488f6]). Expected version 1 got 0
I20251025 07:52:37.910210 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "70fe9aa2f58d4abea3b3b71066a29653" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:37.910554 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "70fe9aa2f58d4abea3b3b71066a29653" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:37.910835 774 raft_consensus.cc:2393] T 70fe9aa2f58d4abea3b3b71066a29653 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3aea2869ff814cc7b142012faa274382 in current term 1: Already voted for candidate db42be7ad3c741a0aa427f27e245993b in this term.
I20251025 07:52:37.910185 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c]). Expected version 1 got 0
I20251025 07:52:37.911340 847 leader_election.cc:304] T 70fe9aa2f58d4abea3b3b71066a29653 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:37.911768 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236]). Expected version 1 got 0
I20251025 07:52:37.911922 1838 raft_consensus.cc:3060] T 70fe9aa2f58d4abea3b3b71066a29653 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.911998 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]). Expected version 3 got 0
I20251025 07:52:37.912331 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_e90d7f5ff13b40878e5b625c97792c7d [id=fc8180c09d7b4e74b2e161f7acb8fe0e]). Expected version 1 got 0
I20251025 07:52:37.915743 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3]). Expected version 2 got 0
I20251025 07:52:37.916399 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 87268e27540c4ec58ce1e62df819f19d (table default.Table_350383f0e98947eba77186e74ad88781 [id=96b0e47c666c4d8f982741aea7b17ae7]). Expected version 2 got 0
I20251025 07:52:37.916631 1838 raft_consensus.cc:2749] T 70fe9aa2f58d4abea3b3b71066a29653 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:37.916667 451 catalog_manager.cc:5649] T 8ff73ffc42434f74a4949f156771d5a4 P 23eeffd3de254ce18db5febfde971c6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 23eeffd3de254ce18db5febfde971c6b (127.0.104.1). New cstate: current_term: 1 leader_uuid: "23eeffd3de254ce18db5febfde971c6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: UNKNOWN } } }
I20251025 07:52:37.917230 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 8f2c8582fed34c848d9903b8ad8b4057 (table default.Table_438f67f3dc62465999a917986922b3ed [id=7c7fcd265e624d7fab74661db505d9c3]). Expected version 1 got 0
I20251025 07:52:37.917503 1167 tablet_bootstrap.cc:654] T ef9a2c9f2ccf432ab835396020872208 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:37.917816 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 36262ab8c6964dd684c01b4db6219a4f (table default.Table_76158f8cf82e4cdb9a90580e46aaea48 [id=f249f30497bc450c887d659721469de8]). Expected version 1 got 0
I20251025 07:52:37.926076 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 2c50ec95e48e49b59a76fac4fc862d3d (table default.Table_32b54e82cbc84d39a0020826f8737d62 [id=c28b0b6acd61490caecfdaab803a9d92]). Expected version 2 got 0
I20251025 07:52:37.926359 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3]). Expected version 1 got 0
I20251025 07:52:37.926599 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet b97e39aaac3048ed93dbfe873b757254 (table default.Table_99fd36369f9a4e0fb14f234c98635203 [id=c8059ab6cec146efa631d06799d969a9]). Expected version 1 got 0
I20251025 07:52:37.926868 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 8e6d76b1a97945cfb6c3055e1b622f8a (table default.Table_10e57b42f97544208675708b1e028846 [id=f5c1be203e2b4831a025871cf8a2ef17]). Expected version 2 got 0
I20251025 07:52:37.930927 1169 tablet_bootstrap.cc:492] T 3abfd52327a04ba88625f391a8658fbb P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:37.934219 1169 ts_tablet_manager.cc:1403] T 3abfd52327a04ba88625f391a8658fbb P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.119s user 0.004s sys 0.000s
I20251025 07:52:37.938446 1169 raft_consensus.cc:359] T 3abfd52327a04ba88625f391a8658fbb P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:37.939491 1569 raft_consensus.cc:493] T 6ec421eb820b4f40afcff5a5bf80c508 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:37.939630 1569 raft_consensus.cc:515] T 6ec421eb820b4f40afcff5a5bf80c508 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.940256 1569 leader_election.cc:290] T 6ec421eb820b4f40afcff5a5bf80c508 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:37.940879 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ec421eb820b4f40afcff5a5bf80c508" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:37.941069 913 raft_consensus.cc:2468] T 6ec421eb820b4f40afcff5a5bf80c508 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:37.941741 983 leader_election.cc:304] T 6ec421eb820b4f40afcff5a5bf80c508 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:37.942591 2050 raft_consensus.cc:493] T 5361f08f50b745eb98bf02e979d7c71e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:37.942711 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ec421eb820b4f40afcff5a5bf80c508" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:37.942735 2050 raft_consensus.cc:515] T 5361f08f50b745eb98bf02e979d7c71e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.942879 773 raft_consensus.cc:2468] T 6ec421eb820b4f40afcff5a5bf80c508 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:37.943301 2050 leader_election.cc:290] T 5361f08f50b745eb98bf02e979d7c71e P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:37.943444 2050 raft_consensus.cc:2804] T 6ec421eb820b4f40afcff5a5bf80c508 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:37.943536 2050 raft_consensus.cc:493] T 6ec421eb820b4f40afcff5a5bf80c508 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:37.943614 2050 raft_consensus.cc:3060] T 6ec421eb820b4f40afcff5a5bf80c508 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.943738 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5361f08f50b745eb98bf02e979d7c71e" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:37.943912 776 raft_consensus.cc:2468] T 5361f08f50b745eb98bf02e979d7c71e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:37.944270 982 leader_election.cc:304] T 5361f08f50b745eb98bf02e979d7c71e P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:37.944554 1612 consensus_queue.cc:1048] T d5e5e5bbcf9f43118eea502b69880af5 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:37.944973 1612 raft_consensus.cc:2804] T 5361f08f50b745eb98bf02e979d7c71e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:37.945056 1612 raft_consensus.cc:493] T 5361f08f50b745eb98bf02e979d7c71e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:37.945181 1612 raft_consensus.cc:3060] T 5361f08f50b745eb98bf02e979d7c71e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.946408 1169 raft_consensus.cc:385] T 3abfd52327a04ba88625f391a8658fbb P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:37.946528 1169 raft_consensus.cc:740] T 3abfd52327a04ba88625f391a8658fbb P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:37.946825 1169 consensus_queue.cc:260] T 3abfd52327a04ba88625f391a8658fbb P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:37.947454 1169 ts_tablet_manager.cc:1434] T 3abfd52327a04ba88625f391a8658fbb P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20251025 07:52:37.948067 1612 raft_consensus.cc:515] T 5361f08f50b745eb98bf02e979d7c71e P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.948635 2050 raft_consensus.cc:515] T 6ec421eb820b4f40afcff5a5bf80c508 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.948724 1612 leader_election.cc:290] T 5361f08f50b745eb98bf02e979d7c71e P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:37.927130 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet b17d2a9ad6be49d29f2413666fba7e69 (table default.Table_fcddb31989ee4236a4981c12779b3f5d [id=f3794fa081364666b944c412415bdba0]). Expected version 3 got 0
I20251025 07:52:37.949086 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 292fe04055954afd8c08ac2636b68191 (table default.Table_9fa69b10b5e842639fc0b0a58cd9b6a5 [id=a253ce0253bd4da4814741c482193d8f]). Expected version 1 got 0
I20251025 07:52:37.949138 2050 leader_election.cc:290] T 6ec421eb820b4f40afcff5a5bf80c508 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:37.949379 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5361f08f50b745eb98bf02e979d7c71e" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:37.949731 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 509c94f2a422493ea4390ecee576db72 (table default.Table_3f081b7bb4e1420c8121ea4115439f3d [id=696a1285fbfe4038a8b1b38708c8ea49]). Expected version 1 got 0
I20251025 07:52:37.949790 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5361f08f50b745eb98bf02e979d7c71e" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:37.949998 449 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet 92aa605fabbc4257a6a204c087e7cc7f
I20251025 07:52:37.950088 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ec421eb820b4f40afcff5a5bf80c508" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:37.950242 903 raft_consensus.cc:3060] T 6ec421eb820b4f40afcff5a5bf80c508 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.950801 449 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:47610 (request call id 32) took 1692 ms. Trace:
I20251025 07:52:37.952008 1169 tablet_bootstrap.cc:492] T fa998b9ca6724c4cb1eb68a3eecb0b11 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:37.950943 449 rpcz_store.cc:276] 1025 07:52:36.257928 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:36.938441 (+680513us) service_pool.cc:225] Handling call
1025 07:52:36.939730 (+ 1289us) spinlock_profiling.cc:243] Waited 1.02 ms on lock 0x616000041254. stack: 00007fb90f7779f6 00007fb90f7777b0 00007fb90e7533ad 0000000000983f04 0000000000983ecd 00007fb91ea756bd 00007fb91ea757b3 00007fb91b640441 00007fb9232f69dd 00007fb923484256 00007fb919f9b486 00007fb919f9b1b5 00007fb91671327e 00007fb91671215e 00007fb916718395 00007fb916718cbf
1025 07:52:36.940281 (+ 551us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:36.940409 (+ 128us) catalog_manager.cc:2815] Modifying in-memory table state
1025 07:52:36.940476 (+ 67us) catalog_manager.cc:2824] Locking tablets
1025 07:52:37.055198 (+114722us) catalog_manager.cc:2838] Removing table and tablets from system table
1025 07:52:37.077173 (+ 21975us) catalog_manager.cc:2856] Removing table from by-name map
1025 07:52:37.077310 (+ 137us) catalog_manager.cc:2872] Committing in-memory state
1025 07:52:37.949926 (+872616us) catalog_manager.cc:2876] Aborting table tasks
1025 07:52:37.950723 (+ 797us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:37.057295 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:37.057428 (+ 133us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:37.057436 (+ 8us) write_op.cc:435] Acquired schema lock
1025 07:52:37.057439 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:37.057575 (+ 136us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:37.057589 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:37.057596 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:37.057731 (+ 135us) tablet.cc:699] Row locks acquired
1025 07:52:37.057735 (+ 4us) write_op.cc:260] PREPARE: finished
1025 07:52:37.057794 (+ 59us) write_op.cc:270] Start()
1025 07:52:37.057828 (+ 34us) write_op.cc:276] Timestamp: P: 1761378757057737 usec, L: 0
1025 07:52:37.057833 (+ 5us) op_driver.cc:348] REPLICATION: starting
1025 07:52:37.058020 (+ 187us) log.cc:844] Serialized 756 byte log entry
1025 07:52:37.075176 (+ 17156us) op_driver.cc:464] REPLICATION: finished
1025 07:52:37.076190 (+ 1014us) write_op.cc:301] APPLY: starting
1025 07:52:37.076204 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:37.076240 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:37.076281 (+ 41us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:37.076345 (+ 64us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:37.076380 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1025 07:52:37.076391 (+ 11us) write_op.cc:312] APPLY: finished
1025 07:52:37.076454 (+ 63us) log.cc:844] Serialized 29 byte log entry
1025 07:52:37.076513 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:37.076532 (+ 19us) write_op.cc:454] Released schema lock
1025 07:52:37.076540 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:37.076611 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:37.076617 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":2338048,"child_traces":[["op",{"apply.queue_time_us":975,"apply.run_cpu_time_us":410,"apply.run_wall_time_us":410,"num_ops":2,"prepare.queue_time_us":1308,"prepare.run_cpu_time_us":828,"prepare.run_wall_time_us":827,"raft.queue_time_us":2767,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":6,"replication_time_us":17320}]]}
I20251025 07:52:37.952970 903 raft_consensus.cc:2468] T 6ec421eb820b4f40afcff5a5bf80c508 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:37.953855 1169 tablet_bootstrap.cc:654] T fa998b9ca6724c4cb1eb68a3eecb0b11 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:37.954531 983 leader_election.cc:343] T 5361f08f50b745eb98bf02e979d7c71e P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:37.954891 983 leader_election.cc:343] T 5361f08f50b745eb98bf02e979d7c71e P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:37.955090 983 leader_election.cc:304] T 6ec421eb820b4f40afcff5a5bf80c508 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:37.955981 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 92aa605fabbc4257a6a204c087e7cc7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:37.957543 1166 tablet_bootstrap.cc:492] T 410ef150a7014c6c867a021b544a2f4e P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:37.957459 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ec421eb820b4f40afcff5a5bf80c508" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:37.957690 772 raft_consensus.cc:3060] T 6ec421eb820b4f40afcff5a5bf80c508 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.957695 1166 ts_tablet_manager.cc:1403] T 410ef150a7014c6c867a021b544a2f4e P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.179s user 0.011s sys 0.011s
I20251025 07:52:37.958590 1166 raft_consensus.cc:359] T 410ef150a7014c6c867a021b544a2f4e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.958783 1166 raft_consensus.cc:385] T 410ef150a7014c6c867a021b544a2f4e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:37.958854 1166 raft_consensus.cc:740] T 410ef150a7014c6c867a021b544a2f4e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:37.960340 772 raft_consensus.cc:2468] T 6ec421eb820b4f40afcff5a5bf80c508 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:37.957480 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5361f08f50b745eb98bf02e979d7c71e" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:37.960695 775 raft_consensus.cc:3060] T 5361f08f50b745eb98bf02e979d7c71e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.963358 775 raft_consensus.cc:2468] T 5361f08f50b745eb98bf02e979d7c71e P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:37.963413 1612 raft_consensus.cc:2804] T 6ec421eb820b4f40afcff5a5bf80c508 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:37.963570 1612 raft_consensus.cc:697] T 6ec421eb820b4f40afcff5a5bf80c508 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:37.963819 1569 raft_consensus.cc:493] T 551d07a9242740dcb9366bb2d707a4a8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:37.963979 1569 raft_consensus.cc:515] T 551d07a9242740dcb9366bb2d707a4a8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.963900 1612 consensus_queue.cc:237] T 6ec421eb820b4f40afcff5a5bf80c508 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.964643 1569 leader_election.cc:290] T 551d07a9242740dcb9366bb2d707a4a8 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:37.965437 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "551d07a9242740dcb9366bb2d707a4a8" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:37.965615 903 raft_consensus.cc:2468] T 551d07a9242740dcb9366bb2d707a4a8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:37.965608 982 leader_election.cc:304] T 5361f08f50b745eb98bf02e979d7c71e P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters: 3aea2869ff814cc7b142012faa274382
I20251025 07:52:37.966117 983 leader_election.cc:304] T 551d07a9242740dcb9366bb2d707a4a8 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:37.966327 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "551d07a9242740dcb9366bb2d707a4a8" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:37.966506 775 raft_consensus.cc:2468] T 551d07a9242740dcb9366bb2d707a4a8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:37.966931 1569 raft_consensus.cc:2804] T 5361f08f50b745eb98bf02e979d7c71e P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:37.967080 1569 raft_consensus.cc:697] T 5361f08f50b745eb98bf02e979d7c71e P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:37.967133 1166 consensus_queue.cc:260] T 410ef150a7014c6c867a021b544a2f4e P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.967362 1569 consensus_queue.cc:237] T 5361f08f50b745eb98bf02e979d7c71e P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.967741 1166 ts_tablet_manager.cc:1434] T 410ef150a7014c6c867a021b544a2f4e P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251025 07:52:37.968024 1671 raft_consensus.cc:2804] T 551d07a9242740dcb9366bb2d707a4a8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:37.968124 1671 raft_consensus.cc:493] T 551d07a9242740dcb9366bb2d707a4a8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:37.968217 1671 raft_consensus.cc:3060] T 551d07a9242740dcb9366bb2d707a4a8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.969813 2157 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:42203 every 8 connection(s)
I20251025 07:52:37.970480 1830 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:42203
I20251025 07:52:37.971086 1671 raft_consensus.cc:515] T 551d07a9242740dcb9366bb2d707a4a8 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.972028 1671 leader_election.cc:290] T 551d07a9242740dcb9366bb2d707a4a8 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:37.972715 1830 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/data/info.pb
I20251025 07:52:37.973227 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "551d07a9242740dcb9366bb2d707a4a8" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:37.973416 903 raft_consensus.cc:3060] T 551d07a9242740dcb9366bb2d707a4a8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.975579 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRBHlYt/build/asan/bin/kudu as pid 1830
I20251025 07:52:37.976922 903 raft_consensus.cc:2468] T 551d07a9242740dcb9366bb2d707a4a8 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:37.977545 983 leader_election.cc:304] T 551d07a9242740dcb9366bb2d707a4a8 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:37.978351 1671 raft_consensus.cc:2804] T 551d07a9242740dcb9366bb2d707a4a8 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:37.978533 1671 raft_consensus.cc:697] T 551d07a9242740dcb9366bb2d707a4a8 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:37.978894 1671 consensus_queue.cc:237] T 551d07a9242740dcb9366bb2d707a4a8 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.980904 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "551d07a9242740dcb9366bb2d707a4a8" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:37.981175 773 raft_consensus.cc:3060] T 551d07a9242740dcb9366bb2d707a4a8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:37.981523 1169 tablet_bootstrap.cc:492] T fa998b9ca6724c4cb1eb68a3eecb0b11 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:37.981684 1169 ts_tablet_manager.cc:1403] T fa998b9ca6724c4cb1eb68a3eecb0b11 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20251025 07:52:37.982712 1169 raft_consensus.cc:359] T fa998b9ca6724c4cb1eb68a3eecb0b11 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:37.982887 1169 raft_consensus.cc:385] T fa998b9ca6724c4cb1eb68a3eecb0b11 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:37.982980 1169 raft_consensus.cc:740] T fa998b9ca6724c4cb1eb68a3eecb0b11 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:37.984643 1659 raft_consensus.cc:493] T 50ae9bca6bed4f749d54191ac1543410 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:37.984869 1659 raft_consensus.cc:515] T 50ae9bca6bed4f749d54191ac1543410 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:37.985694 1659 leader_election.cc:290] T 50ae9bca6bed4f749d54191ac1543410 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:37.986371 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50ae9bca6bed4f749d54191ac1543410" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:37.986809 1166 tablet_bootstrap.cc:492] T 8f2c8582fed34c848d9903b8ad8b4057 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
W20251025 07:52:37.987157 710 leader_election.cc:343] T 50ae9bca6bed4f749d54191ac1543410 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:37.990679 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50ae9bca6bed4f749d54191ac1543410" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:37.996009 1167 tablet_bootstrap.cc:492] T ef9a2c9f2ccf432ab835396020872208 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:37.996181 1167 ts_tablet_manager.cc:1403] T ef9a2c9f2ccf432ab835396020872208 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.098s user 0.002s sys 0.001s
I20251025 07:52:37.996964 1167 raft_consensus.cc:359] T ef9a2c9f2ccf432ab835396020872208 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:37.997108 1167 raft_consensus.cc:385] T ef9a2c9f2ccf432ab835396020872208 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:37.997151 1167 raft_consensus.cc:740] T ef9a2c9f2ccf432ab835396020872208 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:37.998400 1167 consensus_queue.cc:260] T ef9a2c9f2ccf432ab835396020872208 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:37.998596 773 raft_consensus.cc:2468] T 551d07a9242740dcb9366bb2d707a4a8 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:37.999542 1167 ts_tablet_manager.cc:1434] T ef9a2c9f2ccf432ab835396020872208 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251025 07:52:37.999778 1167 tablet_bootstrap.cc:492] T fe5033e5ff404a5ebc6c8443d7548327 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:38.001578 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 92aa605fabbc4257a6a204c087e7cc7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.002779 1167 tablet_bootstrap.cc:654] T fe5033e5ff404a5ebc6c8443d7548327 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:38.003671 712 leader_election.cc:343] T 50ae9bca6bed4f749d54191ac1543410 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:38.004139 712 leader_election.cc:304] T 50ae9bca6bed4f749d54191ac1543410 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:38.004781 1659 raft_consensus.cc:2749] T 50ae9bca6bed4f749d54191ac1543410 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:38.012466 1166 tablet_bootstrap.cc:654] T 8f2c8582fed34c848d9903b8ad8b4057 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:38.015542 1167 tablet_bootstrap.cc:492] T fe5033e5ff404a5ebc6c8443d7548327 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:38.015678 1167 ts_tablet_manager.cc:1403] T fe5033e5ff404a5ebc6c8443d7548327 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20251025 07:52:38.016336 1167 raft_consensus.cc:359] T fe5033e5ff404a5ebc6c8443d7548327 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.016498 1167 raft_consensus.cc:385] T fe5033e5ff404a5ebc6c8443d7548327 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:38.016542 1167 raft_consensus.cc:740] T fe5033e5ff404a5ebc6c8443d7548327 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:38.016746 1167 consensus_queue.cc:260] T fe5033e5ff404a5ebc6c8443d7548327 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.017259 1167 ts_tablet_manager.cc:1434] T fe5033e5ff404a5ebc6c8443d7548327 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:38.017505 1167 tablet_bootstrap.cc:492] T 36262ab8c6964dd684c01b4db6219a4f P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:38.023016 455 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet 4b28215f61574cf48c447dd01955ac77
W20251025 07:52:38.023276 454 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5dab439bc50d4e869548154e1d222ad6 consensus state has no leader
I20251025 07:52:38.023733 455 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:47610 (request call id 31) took 1765 ms. Trace:
I20251025 07:52:38.023838 1659 raft_consensus.cc:493] T 22c97b3ee92140c68a7e1b93033cb299 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:38.023960 1659 raft_consensus.cc:515] T 22c97b3ee92140c68a7e1b93033cb299 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.023819 455 rpcz_store.cc:276] 1025 07:52:36.257836 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:36.926727 (+668891us) service_pool.cc:225] Handling call
1025 07:52:36.928028 (+ 1301us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:36.928210 (+ 182us) catalog_manager.cc:2815] Modifying in-memory table state
1025 07:52:36.928306 (+ 96us) catalog_manager.cc:2824] Locking tablets
1025 07:52:37.054080 (+125774us) catalog_manager.cc:2838] Removing table and tablets from system table
1025 07:52:37.077363 (+ 23283us) catalog_manager.cc:2856] Removing table from by-name map
1025 07:52:37.077443 (+ 80us) catalog_manager.cc:2872] Committing in-memory state
1025 07:52:38.022937 (+945494us) catalog_manager.cc:2876] Aborting table tasks
1025 07:52:38.023674 (+ 737us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:37.056247 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:37.056425 (+ 178us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:37.056434 (+ 9us) write_op.cc:435] Acquired schema lock
1025 07:52:37.056438 (+ 4us) tablet.cc:660] Decoding operations
1025 07:52:37.056602 (+ 164us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:37.056622 (+ 20us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:37.056629 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:37.056759 (+ 130us) tablet.cc:699] Row locks acquired
1025 07:52:37.056764 (+ 5us) write_op.cc:260] PREPARE: finished
1025 07:52:37.056805 (+ 41us) write_op.cc:270] Start()
1025 07:52:37.056845 (+ 40us) write_op.cc:276] Timestamp: P: 1761378757056770 usec, L: 0
1025 07:52:37.056851 (+ 6us) op_driver.cc:348] REPLICATION: starting
1025 07:52:37.057116 (+ 265us) log.cc:844] Serialized 756 byte log entry
1025 07:52:37.075086 (+ 17970us) op_driver.cc:464] REPLICATION: finished
1025 07:52:37.075498 (+ 412us) write_op.cc:301] APPLY: starting
1025 07:52:37.075521 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:37.075585 (+ 64us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:37.075622 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:37.075721 (+ 99us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:37.075769 (+ 48us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1025 07:52:37.075782 (+ 13us) write_op.cc:312] APPLY: finished
1025 07:52:37.075891 (+ 109us) log.cc:844] Serialized 29 byte log entry
1025 07:52:37.075970 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:37.075998 (+ 28us) write_op.cc:454] Released schema lock
1025 07:52:37.076008 (+ 10us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:37.076098 (+ 90us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:37.076104 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":1820544,"child_traces":[["op",{"apply.queue_time_us":335,"apply.run_cpu_time_us":587,"apply.run_wall_time_us":587,"num_ops":2,"prepare.queue_time_us":1224,"prepare.run_cpu_time_us":998,"prepare.run_wall_time_us":998,"raft.queue_time_us":4065,"raft.run_cpu_time_us":355,"raft.run_wall_time_us":356,"replication_time_us":18190}]]}
I20251025 07:52:38.024564 1659 leader_election.cc:290] T 22c97b3ee92140c68a7e1b93033cb299 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:38.025105 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22c97b3ee92140c68a7e1b93033cb299" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:38.025568 712 leader_election.cc:343] T 22c97b3ee92140c68a7e1b93033cb299 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:38.026690 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b28215f61574cf48c447dd01955ac77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.026890 1167 tablet_bootstrap.cc:654] T 36262ab8c6964dd684c01b4db6219a4f P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:38.027259 744 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b28215f61574cf48c447dd01955ac77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.035328 2166 tablet_replica.cc:333] T 92aa605fabbc4257a6a204c087e7cc7f P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:38.035676 2166 raft_consensus.cc:2243] T 92aa605fabbc4257a6a204c087e7cc7f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:38.035914 2166 raft_consensus.cc:2272] T 92aa605fabbc4257a6a204c087e7cc7f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:38.035946 450 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet 5361f08f50b745eb98bf02e979d7c71e
I20251025 07:52:38.036617 450 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:47610 (request call id 29) took 1778 ms. Trace:
I20251025 07:52:38.036573 1569 consensus_queue.cc:1048] T b97e39aaac3048ed93dbfe873b757254 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:38.036994 1169 consensus_queue.cc:260] T fa998b9ca6724c4cb1eb68a3eecb0b11 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.037146 2166 ts_tablet_manager.cc:1916] T 92aa605fabbc4257a6a204c087e7cc7f P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:38.036748 450 rpcz_store.cc:276] 1025 07:52:36.257680 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:36.900090 (+642410us) service_pool.cc:225] Handling call
1025 07:52:36.906416 (+ 6326us) spinlock_profiling.cc:243] Waited 5.89 ms on lock 0x616000041254. stack: 00007fb90f7779f6 00007fb90f7777b0 00007fb90e7533ad 0000000000983f04 0000000000983ecd 00007fb91ea756bd 00007fb91ea757b3 00007fb91b640441 00007fb9232f69dd 00007fb923484256 00007fb919f9b486 00007fb919f9b1b5 00007fb91671327e 00007fb91671215e 00007fb916718395 00007fb916718cbf
1025 07:52:36.907326 (+ 910us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:36.907578 (+ 252us) catalog_manager.cc:2815] Modifying in-memory table state
1025 07:52:36.907668 (+ 90us) catalog_manager.cc:2824] Locking tablets
1025 07:52:37.054051 (+146383us) catalog_manager.cc:2838] Removing table and tablets from system table
1025 07:52:37.077501 (+ 23450us) catalog_manager.cc:2856] Removing table from by-name map
1025 07:52:37.077566 (+ 65us) catalog_manager.cc:2872] Committing in-memory state
1025 07:52:38.035886 (+958320us) catalog_manager.cc:2876] Aborting table tasks
1025 07:52:38.036546 (+ 660us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:37.058184 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:37.058298 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:37.058305 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:37.058309 (+ 4us) tablet.cc:660] Decoding operations
1025 07:52:37.058422 (+ 113us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:37.058434 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:37.058441 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:37.058557 (+ 116us) tablet.cc:699] Row locks acquired
1025 07:52:37.058562 (+ 5us) write_op.cc:260] PREPARE: finished
1025 07:52:37.058593 (+ 31us) write_op.cc:270] Start()
1025 07:52:37.058623 (+ 30us) write_op.cc:276] Timestamp: P: 1761378757058561 usec, L: 0
1025 07:52:37.058628 (+ 5us) op_driver.cc:348] REPLICATION: starting
1025 07:52:37.058790 (+ 162us) log.cc:844] Serialized 756 byte log entry
1025 07:52:37.075254 (+ 16464us) op_driver.cc:464] REPLICATION: finished
1025 07:52:37.076688 (+ 1434us) write_op.cc:301] APPLY: starting
1025 07:52:37.076701 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:37.076736 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:37.076762 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:37.076820 (+ 58us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:37.076853 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1025 07:52:37.076864 (+ 11us) write_op.cc:312] APPLY: finished
1025 07:52:37.076919 (+ 55us) log.cc:844] Serialized 29 byte log entry
1025 07:52:37.076958 (+ 39us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:37.076977 (+ 19us) write_op.cc:454] Released schema lock
1025 07:52:37.076983 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:37.142197 (+ 65214us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:37.142209 (+ 12us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":13547776,"child_traces":[["op",{"apply.queue_time_us":1395,"apply.run_cpu_time_us":355,"apply.run_wall_time_us":355,"num_ops":2,"prepare.queue_time_us":415,"prepare.run_cpu_time_us":11419,"prepare.run_wall_time_us":83990,"raft.queue_time_us":1047,"raft.run_cpu_time_us":10,"raft.run_wall_time_us":9,"replication_time_us":16604,"thread_start_us":82681,"threads_started":1}]]}
I20251025 07:52:38.037576 1169 ts_tablet_manager.cc:1434] T fa998b9ca6724c4cb1eb68a3eecb0b11 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.056s user 0.002s sys 0.000s
I20251025 07:52:38.037899 1169 tablet_bootstrap.cc:492] T d66ad049d5484228a268b0facb00ecf1 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:38.039049 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 5361f08f50b745eb98bf02e979d7c71e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.049862 1169 tablet_bootstrap.cc:654] T d66ad049d5484228a268b0facb00ecf1 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:38.054337 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 5361f08f50b745eb98bf02e979d7c71e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:58412
W20251025 07:52:38.055052 448 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 22f7db660616457b83b24e947a0b0bae consensus state has no leader
W20251025 07:52:38.057696 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 4b28215f61574cf48c447dd01955ac77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b28215f61574cf48c447dd01955ac77 already in progress: creating tablet
I20251025 07:52:38.061954 2169 tablet_replica.cc:333] T 5361f08f50b745eb98bf02e979d7c71e P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:38.062601 2169 raft_consensus.cc:2243] T 5361f08f50b745eb98bf02e979d7c71e P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:38.063287 2169 pending_rounds.cc:70] T 5361f08f50b745eb98bf02e979d7c71e P db42be7ad3c741a0aa427f27e245993b: Trying to abort 1 pending ops.
I20251025 07:52:38.063721 2169 pending_rounds.cc:77] T 5361f08f50b745eb98bf02e979d7c71e P db42be7ad3c741a0aa427f27e245993b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214607392677376000 op_type: NO_OP noop_request { }
I20251025 07:52:38.064105 2169 raft_consensus.cc:2889] T 5361f08f50b745eb98bf02e979d7c71e P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 07:52:38.064430 2169 raft_consensus.cc:2272] T 5361f08f50b745eb98bf02e979d7c71e P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:38.065809 2169 ts_tablet_manager.cc:1916] T 5361f08f50b745eb98bf02e979d7c71e P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:38.066447 453 catalog_manager.cc:6194] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: default.Table_cd9190d6cb714bdbb763dd934b33c10a [id=1b5db2f5053c4cb3a3b40eb109d5d5d9] alter complete (version 3)
I20251025 07:52:38.069269 1569 raft_consensus.cc:493] T 3abfd52327a04ba88625f391a8658fbb P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:38.069474 1569 raft_consensus.cc:515] T 3abfd52327a04ba88625f391a8658fbb P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.069814 2166 ts_tablet_manager.cc:1929] T 92aa605fabbc4257a6a204c087e7cc7f P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:38.070350 1569 leader_election.cc:290] T 3abfd52327a04ba88625f391a8658fbb P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:38.070686 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3abfd52327a04ba88625f391a8658fbb" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:38.070856 774 raft_consensus.cc:2468] T 3abfd52327a04ba88625f391a8658fbb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:38.071298 982 leader_election.cc:304] T 3abfd52327a04ba88625f391a8658fbb P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:38.071592 1569 raft_consensus.cc:2804] T 3abfd52327a04ba88625f391a8658fbb P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:38.071704 1569 raft_consensus.cc:493] T 3abfd52327a04ba88625f391a8658fbb P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:38.071799 1569 raft_consensus.cc:3060] T 3abfd52327a04ba88625f391a8658fbb P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:38.073494 1167 tablet_bootstrap.cc:492] T 36262ab8c6964dd684c01b4db6219a4f P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:38.073639 1167 ts_tablet_manager.cc:1403] T 36262ab8c6964dd684c01b4db6219a4f P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.011s
I20251025 07:52:38.027673 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b28215f61574cf48c447dd01955ac77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.074389 1167 raft_consensus.cc:359] T 36262ab8c6964dd684c01b4db6219a4f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.074529 1167 raft_consensus.cc:385] T 36262ab8c6964dd684c01b4db6219a4f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251025 07:52:38.074566 446 catalog_manager.cc:5101] could not convert tablet's cd23b8094d7e472eba6bd170248858a9 schema from PB: OK
I20251025 07:52:38.074582 1167 raft_consensus.cc:740] T 36262ab8c6964dd684c01b4db6219a4f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:38.074769 1167 consensus_queue.cc:260] T 36262ab8c6964dd684c01b4db6219a4f P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:38.075083 447 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 830fdbc6812e416690d0db42a73af503 consensus state has no leader
I20251025 07:52:38.075281 1167 ts_tablet_manager.cc:1434] T 36262ab8c6964dd684c01b4db6219a4f P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:38.075570 1167 tablet_bootstrap.cc:492] T 292fe04055954afd8c08ac2636b68191 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:38.077545 1167 tablet_bootstrap.cc:654] T 292fe04055954afd8c08ac2636b68191 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:38.034693 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22c97b3ee92140c68a7e1b93033cb299" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:38.049938 1838 raft_consensus.cc:493] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:38.079548 1838 raft_consensus.cc:515] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.080289 1838 leader_election.cc:290] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:38.052830 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 5361f08f50b745eb98bf02e979d7c71e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.064047 2105 consensus_queue.cc:1048] T 07898860aa594a05a211fc89768cb76d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:38.070937 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3abfd52327a04ba88625f391a8658fbb" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:38.082043 1659 raft_consensus.cc:493] T 410ef150a7014c6c867a021b544a2f4e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:38.082197 1659 raft_consensus.cc:515] T 410ef150a7014c6c867a021b544a2f4e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:38.082564 983 leader_election.cc:343] T 3abfd52327a04ba88625f391a8658fbb P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:38.082842 1659 leader_election.cc:290] T 410ef150a7014c6c867a021b544a2f4e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:38.083465 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "410ef150a7014c6c867a021b544a2f4e" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:38.083949 712 leader_election.cc:343] T 410ef150a7014c6c867a021b544a2f4e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:38.084589 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "410ef150a7014c6c867a021b544a2f4e" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:38.082209 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b17d2a9ad6be49d29f2413666fba7e69" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:38.084762 910 raft_consensus.cc:2468] T 410ef150a7014c6c867a021b544a2f4e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:38.084792 774 raft_consensus.cc:2468] T b17d2a9ad6be49d29f2413666fba7e69 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:38.085337 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b17d2a9ad6be49d29f2413666fba7e69" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:38.085670 1046 raft_consensus.cc:2468] T b17d2a9ad6be49d29f2413666fba7e69 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:38.086174 850 leader_election.cc:304] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:38.086432 2105 raft_consensus.cc:2804] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 07:52:38.086514 2105 raft_consensus.cc:493] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:38.086580 2105 raft_consensus.cc:3060] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Advancing to term 2
I20251025 07:52:38.087118 710 leader_election.cc:304] T 410ef150a7014c6c867a021b544a2f4e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:38.087487 710 leader_election.cc:343] T 22c97b3ee92140c68a7e1b93033cb299 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:38.087572 710 leader_election.cc:304] T 22c97b3ee92140c68a7e1b93033cb299 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:38.087847 2124 raft_consensus.cc:2804] T 410ef150a7014c6c867a021b544a2f4e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:38.087900 2124 raft_consensus.cc:493] T 410ef150a7014c6c867a021b544a2f4e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:38.087949 2124 raft_consensus.cc:3060] T 410ef150a7014c6c867a021b544a2f4e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:38.089442 2105 raft_consensus.cc:515] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.090868 2105 leader_election.cc:290] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 2 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:38.091542 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b17d2a9ad6be49d29f2413666fba7e69" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:38.091806 1046 raft_consensus.cc:3060] T b17d2a9ad6be49d29f2413666fba7e69 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Advancing to term 2
I20251025 07:52:38.092365 1659 raft_consensus.cc:2749] T 22c97b3ee92140c68a7e1b93033cb299 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:38.093901 1612 consensus_queue.cc:1048] T abf94c32ff2945119b710905c075d63c P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:38.093941 1838 consensus_queue.cc:1048] T 07898860aa594a05a211fc89768cb76d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:38.098984 1169 tablet_bootstrap.cc:492] T d66ad049d5484228a268b0facb00ecf1 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:38.099023 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b17d2a9ad6be49d29f2413666fba7e69" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:38.099166 1169 ts_tablet_manager.cc:1403] T d66ad049d5484228a268b0facb00ecf1 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.061s user 0.003s sys 0.001s
I20251025 07:52:38.099239 774 raft_consensus.cc:3060] T b17d2a9ad6be49d29f2413666fba7e69 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Advancing to term 2
I20251025 07:52:38.099924 1169 raft_consensus.cc:359] T d66ad049d5484228a268b0facb00ecf1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.100155 1169 raft_consensus.cc:385] T d66ad049d5484228a268b0facb00ecf1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:38.100252 1169 raft_consensus.cc:740] T d66ad049d5484228a268b0facb00ecf1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:38.106559 2124 raft_consensus.cc:515] T 410ef150a7014c6c867a021b544a2f4e P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.107394 2124 leader_election.cc:290] T 410ef150a7014c6c867a021b544a2f4e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:38.107949 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "410ef150a7014c6c867a021b544a2f4e" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:38.108565 712 leader_election.cc:343] T 410ef150a7014c6c867a021b544a2f4e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:38.108971 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "410ef150a7014c6c867a021b544a2f4e" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:38.109141 910 raft_consensus.cc:3060] T 410ef150a7014c6c867a021b544a2f4e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:38.112082 910 raft_consensus.cc:2468] T 410ef150a7014c6c867a021b544a2f4e P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:38.112785 710 leader_election.cc:304] T 410ef150a7014c6c867a021b544a2f4e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:38.113544 2124 raft_consensus.cc:2804] T 410ef150a7014c6c867a021b544a2f4e P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:38.114053 2124 raft_consensus.cc:697] T 410ef150a7014c6c867a021b544a2f4e P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:38.118813 1167 tablet_bootstrap.cc:492] T 292fe04055954afd8c08ac2636b68191 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:38.119040 1167 ts_tablet_manager.cc:1403] T 292fe04055954afd8c08ac2636b68191 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.044s user 0.012s sys 0.002s
I20251025 07:52:38.117221 2124 consensus_queue.cc:237] T 410ef150a7014c6c867a021b544a2f4e P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.119974 1167 raft_consensus.cc:359] T 292fe04055954afd8c08ac2636b68191 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.120517 1167 raft_consensus.cc:385] T 292fe04055954afd8c08ac2636b68191 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:38.120725 1167 raft_consensus.cc:740] T 292fe04055954afd8c08ac2636b68191 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:38.121188 1167 consensus_queue.cc:260] T 292fe04055954afd8c08ac2636b68191 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.122601 961 heartbeater.cc:344] Connected to a master server at 127.0.104.61:42203
I20251025 07:52:38.122113 774 raft_consensus.cc:2468] T b17d2a9ad6be49d29f2413666fba7e69 P 23eeffd3de254ce18db5febfde971c6b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 2.
I20251025 07:52:38.124541 847 leader_election.cc:304] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:38.124854 2159 sys_catalog.cc:263] Verifying existing consensus state
I20251025 07:52:38.125165 1167 ts_tablet_manager.cc:1434] T 292fe04055954afd8c08ac2636b68191 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251025 07:52:38.125901 1167 tablet_bootstrap.cc:492] T e54fe7cbcffa40c4b04144b47bee2b48 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:38.125308 2105 raft_consensus.cc:2804] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [term 2 FOLLOWER]: Leader election won for term 2
I20251025 07:52:38.127071 2105 raft_consensus.cc:697] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [term 2 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:38.130563 1167 tablet_bootstrap.cc:654] T e54fe7cbcffa40c4b04144b47bee2b48 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:38.131003 2105 consensus_queue.cc:237] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.070106 2166 log.cc:1199] T 92aa605fabbc4257a6a204c087e7cc7f P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/92aa605fabbc4257a6a204c087e7cc7f
I20251025 07:52:38.143436 2166 ts_tablet_manager.cc:1950] T 92aa605fabbc4257a6a204c087e7cc7f P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
W20251025 07:52:38.149125 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 92aa605fabbc4257a6a204c087e7cc7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 92aa605fabbc4257a6a204c087e7cc7f already in progress: creating tablet
I20251025 07:52:38.151239 823 heartbeater.cc:344] Connected to a master server at 127.0.104.61:42203
I20251025 07:52:38.154286 2159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Bootstrap starting.
I20251025 07:52:38.154671 1166 tablet_bootstrap.cc:492] T 8f2c8582fed34c848d9903b8ad8b4057 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:38.154816 1166 ts_tablet_manager.cc:1403] T 8f2c8582fed34c848d9903b8ad8b4057 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.168s user 0.003s sys 0.002s
I20251025 07:52:38.158656 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 92aa605fabbc4257a6a204c087e7cc7f (table default.Table_78724078b4574f35879fd2db388fc4f4 [id=187bde879b044237ada58158eb000123]) successfully deleted
W20251025 07:52:38.160059 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 4b28215f61574cf48c447dd01955ac77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b28215f61574cf48c447dd01955ac77 already in progress: creating tablet
I20251025 07:52:38.160683 2166 tablet_replica.cc:333] T 5361f08f50b745eb98bf02e979d7c71e P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:38.161275 2166 raft_consensus.cc:2243] T 5361f08f50b745eb98bf02e979d7c71e P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:38.161870 2166 raft_consensus.cc:2272] T 5361f08f50b745eb98bf02e979d7c71e P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:38.162706 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 4b28215f61574cf48c447dd01955ac77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b28215f61574cf48c447dd01955ac77 already in progress: creating tablet
W20251025 07:52:38.163636 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 5361f08f50b745eb98bf02e979d7c71e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5361f08f50b745eb98bf02e979d7c71e already in progress: creating tablet
I20251025 07:52:38.185981 1166 raft_consensus.cc:359] T 8f2c8582fed34c848d9903b8ad8b4057 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.186213 1166 raft_consensus.cc:385] T 8f2c8582fed34c848d9903b8ad8b4057 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:38.186285 1166 raft_consensus.cc:740] T 8f2c8582fed34c848d9903b8ad8b4057 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:38.186560 1166 consensus_queue.cc:260] T 8f2c8582fed34c848d9903b8ad8b4057 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.187060 1166 ts_tablet_manager.cc:1434] T 8f2c8582fed34c848d9903b8ad8b4057 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.032s user 0.001s sys 0.001s
I20251025 07:52:38.200964 454 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:47610 (request call id 20) took 1944 ms. Trace:
I20251025 07:52:38.201148 454 rpcz_store.cc:276] 1025 07:52:36.256684 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:36.311028 (+ 54344us) service_pool.cc:225] Handling call
1025 07:52:36.311582 (+ 554us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:36.311900 (+ 318us) catalog_manager.cc:3708] Apply alter schema
1025 07:52:36.312097 (+ 197us) catalog_manager.cc:3740] Acquired catalog manager lock
1025 07:52:36.312168 (+ 71us) catalog_manager.cc:3898] Updating metadata on disk
1025 07:52:36.753153 (+440985us) catalog_manager.cc:3929] Committing alterations to in-memory state
1025 07:52:38.200908 (+1447755us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:36.312853 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:36.312946 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:36.312953 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:36.312956 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:36.313052 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:36.313065 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:36.313070 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:36.313131 (+ 61us) tablet.cc:699] Row locks acquired
1025 07:52:36.313134 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:36.313164 (+ 30us) write_op.cc:270] Start()
1025 07:52:36.313197 (+ 33us) write_op.cc:276] Timestamp: P: 1761378756313152 usec, L: 0
1025 07:52:36.313201 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:36.313383 (+ 182us) log.cc:844] Serialized 405 byte log entry
1025 07:52:36.749510 (+436127us) op_driver.cc:464] REPLICATION: finished
1025 07:52:36.751506 (+ 1996us) write_op.cc:301] APPLY: starting
1025 07:52:36.751517 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:36.751541 (+ 24us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:36.751558 (+ 17us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:36.751594 (+ 36us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:36.751615 (+ 21us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:36.751621 (+ 6us) write_op.cc:312] APPLY: finished
1025 07:52:36.751678 (+ 57us) log.cc:844] Serialized 23 byte log entry
1025 07:52:36.751724 (+ 46us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:36.751737 (+ 13us) write_op.cc:454] Released schema lock
1025 07:52:36.751742 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1964,"apply.run_cpu_time_us":289,"apply.run_wall_time_us":289,"num_ops":1,"prepare.queue_time_us":106,"prepare.run_cpu_time_us":628,"prepare.run_wall_time_us":628,"raft.queue_time_us":65,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":3,"replication_time_us":436282,"thread_start_us":51706,"threads_started":1}]]}
I20251025 07:52:38.202610 454 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_8773c967a71644baba838946c154aee2" } modify_external_catalogs: true
I20251025 07:52:38.202879 454 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_8773c967a71644baba838946c154aee2" } modify_external_catalogs: true
I20251025 07:52:38.203233 454 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:47610 (request call id 33) took 1945 ms. Trace:
I20251025 07:52:38.203392 454 rpcz_store.cc:276] 1025 07:52:36.258010 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.202450 (+1944440us) service_pool.cc:225] Handling call
1025 07:52:38.203063 (+ 613us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:38.203200 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 07:52:38.203938 454 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_192549a3e67840f3aefc31029cf2ff52" } modify_external_catalogs: true
I20251025 07:52:38.204176 454 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_192549a3e67840f3aefc31029cf2ff52" } modify_external_catalogs: true
I20251025 07:52:38.204607 454 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:47610 (request call id 34) took 1933 ms. Trace:
I20251025 07:52:38.204747 454 rpcz_store.cc:276] 1025 07:52:36.271227 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.203740 (+1932513us) service_pool.cc:225] Handling call
1025 07:52:38.204323 (+ 583us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:38.204581 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 07:52:38.205368 454 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_05d7a0bdab8b438591303fb459a61a68" } new_table_name: "default.Table_350383f0e98947eba77186e74ad88781" modify_external_catalogs: true
I20251025 07:52:38.120261 1169 consensus_queue.cc:260] T d66ad049d5484228a268b0facb00ecf1 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.219296 1166 tablet_bootstrap.cc:492] T 50ae9bca6bed4f749d54191ac1543410 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:38.222419 1166 tablet_bootstrap.cc:654] T 50ae9bca6bed4f749d54191ac1543410 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:38.229014 1569 raft_consensus.cc:515] T 3abfd52327a04ba88625f391a8658fbb P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.235392 2050 consensus_queue.cc:1048] T 70fe9aa2f58d4abea3b3b71066a29653 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.023s
I20251025 07:52:38.240581 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3abfd52327a04ba88625f391a8658fbb" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:38.240765 773 raft_consensus.cc:3060] T 3abfd52327a04ba88625f391a8658fbb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:38.243523 773 raft_consensus.cc:2468] T 3abfd52327a04ba88625f391a8658fbb P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:38.244292 982 leader_election.cc:304] T 3abfd52327a04ba88625f391a8658fbb P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:38.253865 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f0c4f0e8f014e59be9e46d2afe0be9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.254607 894 tablet_service.cc:1505] Processing CreateTablet for tablet 8e049e460f50482aa920e0c76740fd79 (DEFAULT_TABLE table=default.Table_2dd1c1390f234ba8ac968b6842dd28be [id=1a843288ea7c49bfa3f7758fc1fcf9a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:38.255169 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e049e460f50482aa920e0c76740fd79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:38.256600 748 tablet_service.cc:1505] Processing CreateTablet for tablet 8e049e460f50482aa920e0c76740fd79 (DEFAULT_TABLE table=default.Table_2dd1c1390f234ba8ac968b6842dd28be [id=1a843288ea7c49bfa3f7758fc1fcf9a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:38.257062 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e049e460f50482aa920e0c76740fd79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:38.259433 2050 raft_consensus.cc:2804] T 3abfd52327a04ba88625f391a8658fbb P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:38.259565 2050 raft_consensus.cc:697] T 3abfd52327a04ba88625f391a8658fbb P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:38.259799 2050 consensus_queue.cc:237] T 3abfd52327a04ba88625f391a8658fbb P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.257207 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3abfd52327a04ba88625f391a8658fbb" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:38.254289 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 80ee75935d7e461d9d4066fac9819bef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.267460 875 tablet_service.cc:1505] Processing CreateTablet for tablet 8390711f8ab84170be461b9eeb4c361c (DEFAULT_TABLE table=default.Table_b3ca6007437c4040be247798121808d8 [id=88d28fc51ba94fb4aef4eca8ee82f809]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:38.267828 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8390711f8ab84170be461b9eeb4c361c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:38.269050 1011 tablet_service.cc:1552] Processing DeleteTablet for tablet 92aa605fabbc4257a6a204c087e7cc7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.271169 748 tablet_service.cc:1505] Processing CreateTablet for tablet 8390711f8ab84170be461b9eeb4c361c (DEFAULT_TABLE table=default.Table_b3ca6007437c4040be247798121808d8 [id=88d28fc51ba94fb4aef4eca8ee82f809]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:38.272222 2106 raft_consensus.cc:493] T 8f2c8582fed34c848d9903b8ad8b4057 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:38.272349 2106 raft_consensus.cc:515] T 8f2c8582fed34c848d9903b8ad8b4057 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.272964 2106 leader_election.cc:290] T 8f2c8582fed34c848d9903b8ad8b4057 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:38.273707 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8390711f8ab84170be461b9eeb4c361c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:38.273851 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f2c8582fed34c848d9903b8ad8b4057" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:38.274016 774 raft_consensus.cc:2468] T 8f2c8582fed34c848d9903b8ad8b4057 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:38.274484 847 leader_election.cc:304] T 8f2c8582fed34c848d9903b8ad8b4057 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:38.277698 738 tablet_service.cc:1505] Processing CreateTablet for tablet ff3daed6c9af47e3976b82b4b9a41086 (DEFAULT_TABLE table=default.Table_6caa55a17907446d81c7f0656b329cdd [id=5b6faf06c72745f68f7c478e50a304d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:38.278057 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff3daed6c9af47e3976b82b4b9a41086. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:38.270224 983 leader_election.cc:343] T 3abfd52327a04ba88625f391a8658fbb P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:38.271988 1096 heartbeater.cc:344] Connected to a master server at 127.0.104.61:42203
I20251025 07:52:38.281246 1166 tablet_bootstrap.cc:492] T 50ae9bca6bed4f749d54191ac1543410 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:38.281502 1166 ts_tablet_manager.cc:1403] T 50ae9bca6bed4f749d54191ac1543410 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.062s user 0.004s sys 0.000s
I20251025 07:52:38.282519 1166 raft_consensus.cc:359] T 50ae9bca6bed4f749d54191ac1543410 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.282790 1166 raft_consensus.cc:385] T 50ae9bca6bed4f749d54191ac1543410 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:38.282883 1166 raft_consensus.cc:740] T 50ae9bca6bed4f749d54191ac1543410 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:38.283106 1166 consensus_queue.cc:260] T 50ae9bca6bed4f749d54191ac1543410 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.283839 1659 raft_consensus.cc:493] T 74e2d78c024849b0af5c0bde1f48c36f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:38.283967 1659 raft_consensus.cc:515] T 74e2d78c024849b0af5c0bde1f48c36f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.284420 1166 ts_tablet_manager.cc:1434] T 50ae9bca6bed4f749d54191ac1543410 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251025 07:52:38.284555 1659 leader_election.cc:290] T 74e2d78c024849b0af5c0bde1f48c36f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:38.285530 2051 consensus_queue.cc:1048] T 70fe9aa2f58d4abea3b3b71066a29653 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:38.286715 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74e2d78c024849b0af5c0bde1f48c36f" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:38.287583 712 leader_election.cc:343] T 74e2d78c024849b0af5c0bde1f48c36f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:38.287756 2051 consensus_queue.cc:1048] T abf94c32ff2945119b710905c075d63c P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:38.274539 1569 leader_election.cc:290] T 3abfd52327a04ba88625f391a8658fbb P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:38.288805 744 tablet_service.cc:1505] Processing CreateTablet for tablet 3c41741ea55842f289bd44762a59c595 (DEFAULT_TABLE table=default.Table_be8af62c5969471b9e7b869c6820db0c [id=5ebbedc74c894c32940ac4492e95909b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:38.289172 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c41741ea55842f289bd44762a59c595. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:38.290800 2106 raft_consensus.cc:2804] T 8f2c8582fed34c848d9903b8ad8b4057 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:38.290884 2106 raft_consensus.cc:493] T 8f2c8582fed34c848d9903b8ad8b4057 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:38.290952 2106 raft_consensus.cc:3060] T 8f2c8582fed34c848d9903b8ad8b4057 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:38.291479 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74e2d78c024849b0af5c0bde1f48c36f" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:38.292584 710 leader_election.cc:343] T 74e2d78c024849b0af5c0bde1f48c36f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:38.292712 710 leader_election.cc:304] T 74e2d78c024849b0af5c0bde1f48c36f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:38.292953 2122 raft_consensus.cc:2749] T 74e2d78c024849b0af5c0bde1f48c36f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:38.294605 740 tablet_service.cc:1505] Processing CreateTablet for tablet a0ac349532534848a9016ad5660e85ae (DEFAULT_TABLE table=default.Table_e72e2534c5c44fd299ed3ec3a367a34c [id=c0903d54e7114c7d83233474e7345a3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:38.294955 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0ac349532534848a9016ad5660e85ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:38.296427 1046 raft_consensus.cc:2468] T b17d2a9ad6be49d29f2413666fba7e69 P db42be7ad3c741a0aa427f27e245993b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 2.
I20251025 07:52:38.204981 446 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:47610 (request call id 27) took 1947 ms. Trace:
I20251025 07:52:38.305022 446 rpcz_store.cc:276] 1025 07:52:36.257486 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:36.864361 (+606875us) service_pool.cc:225] Handling call
1025 07:52:36.864769 (+ 408us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:36.864992 (+ 223us) catalog_manager.cc:3708] Apply alter schema
1025 07:52:36.865172 (+ 180us) catalog_manager.cc:3740] Acquired catalog manager lock
1025 07:52:36.865242 (+ 70us) catalog_manager.cc:3898] Updating metadata on disk
1025 07:52:36.883523 (+ 18281us) catalog_manager.cc:3929] Committing alterations to in-memory state
1025 07:52:38.204936 (+1321413us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:36.870894 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:36.870986 (+ 92us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:36.870991 (+ 5us) write_op.cc:435] Acquired schema lock
1025 07:52:36.870993 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:36.871067 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:36.871076 (+ 9us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:36.871080 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:36.871126 (+ 46us) tablet.cc:699] Row locks acquired
1025 07:52:36.871128 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:36.871152 (+ 24us) write_op.cc:270] Start()
1025 07:52:36.871174 (+ 22us) write_op.cc:276] Timestamp: P: 1761378756871124 usec, L: 0
1025 07:52:36.871177 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:36.871306 (+ 129us) log.cc:844] Serialized 405 byte log entry
1025 07:52:36.876989 (+ 5683us) op_driver.cc:464] REPLICATION: finished
1025 07:52:36.882999 (+ 6010us) write_op.cc:301] APPLY: starting
1025 07:52:36.883011 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:36.883043 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:36.883078 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:36.883125 (+ 47us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:36.883151 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:36.883158 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:36.883245 (+ 87us) log.cc:844] Serialized 23 byte log entry
1025 07:52:36.883416 (+ 171us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:36.883430 (+ 14us) write_op.cc:454] Released schema lock
1025 07:52:36.883435 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:36.907135 (+ 23700us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:36.907143 (+ 8us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":5972,"apply.run_cpu_time_us":357,"apply.run_wall_time_us":484,"num_ops":1,"prepare.queue_time_us":5285,"prepare.run_cpu_time_us":6098,"prepare.run_wall_time_us":36215,"raft.queue_time_us":2812,"raft.run_cpu_time_us":8,"raft.run_wall_time_us":6,"replication_time_us":5798,"thread_start_us":32942,"threads_started":1}]]}
I20251025 07:52:38.305694 2106 raft_consensus.cc:515] T 8f2c8582fed34c848d9903b8ad8b4057 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.306185 446 master_service.cc:954] ReplaceTablet: received request to replace tablet b947ecc352e54f4caa73c6ca97372256 from {username='slave'} at 127.0.0.1:47610
I20251025 07:52:38.205651 447 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:47610 (request call id 17) took 1949 ms. Trace:
I20251025 07:52:38.206900 448 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:47610 (request call id 30) took 1948 ms. Trace:
W20251025 07:52:38.242208 453 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 6d3c9954693d4e8782ee0128ef066aa9 consensus state has no leader
W20251025 07:52:38.307083 453 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 9a8e2658a81e436298fe2190726080d4 consensus state has no leader
I20251025 07:52:38.306618 448 rpcz_store.cc:276] 1025 07:52:36.257767 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:36.915477 (+657710us) service_pool.cc:225] Handling call
1025 07:52:36.916760 (+ 1283us) spinlock_profiling.cc:243] Waited 992 us on lock 0x616000041254. stack: 00007fb90f7779f6 00007fb90f7777b0 00007fb90e7533ad 0000000000983f04 0000000000983ecd 00007fb91ea756bd 00007fb91ea757b3 00007fb91b640441 00007fb9232f69dd 00007fb9234854f6 00007fb919f9d146 00007fb919f9ce75 00007fb91671327e 00007fb91671215e 00007fb916718395 00007fb916718cbf
1025 07:52:36.917027 (+ 267us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:36.917255 (+ 228us) catalog_manager.cc:3708] Apply alter schema
1025 07:52:36.917464 (+ 209us) catalog_manager.cc:3740] Acquired catalog manager lock
1025 07:52:36.917585 (+ 121us) catalog_manager.cc:3898] Updating metadata on disk
1025 07:52:36.945639 (+ 28054us) catalog_manager.cc:3929] Committing alterations to in-memory state
1025 07:52:38.206695 (+1261056us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:36.940672 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:36.940800 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:36.940805 (+ 5us) write_op.cc:435] Acquired schema lock
1025 07:52:36.940807 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:36.940906 (+ 99us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:36.940919 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:36.940924 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:36.940982 (+ 58us) tablet.cc:699] Row locks acquired
1025 07:52:36.940984 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:36.941010 (+ 26us) write_op.cc:270] Start()
1025 07:52:36.941058 (+ 48us) write_op.cc:276] Timestamp: P: 1761378756940981 usec, L: 0
1025 07:52:36.941062 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:36.941222 (+ 160us) log.cc:844] Serialized 405 byte log entry
1025 07:52:36.944324 (+ 3102us) op_driver.cc:464] REPLICATION: finished
1025 07:52:36.945003 (+ 679us) write_op.cc:301] APPLY: starting
1025 07:52:36.945019 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:36.945057 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:36.945088 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:36.945134 (+ 46us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:36.945164 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:36.945172 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:36.945238 (+ 66us) log.cc:844] Serialized 23 byte log entry
1025 07:52:36.945429 (+ 191us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:36.945559 (+ 130us) write_op.cc:454] Released schema lock
1025 07:52:36.945566 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:36.950224 (+ 4658us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:36.950232 (+ 8us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":2281728,"child_traces":[["op",{"apply.queue_time_us":628,"apply.run_cpu_time_us":396,"apply.run_wall_time_us":727,"num_ops":1,"prepare.queue_time_us":22592,"prepare.run_cpu_time_us":664,"prepare.run_wall_time_us":9544,"raft.queue_time_us":627,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":548,"replication_time_us":3235}]]}
I20251025 07:52:38.307593 448 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_4f427d19e85b4ce19136b8a82cc5ef76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:38.308239 448 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f427d19e85b4ce19136b8a82cc5ef76 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:38.308372 453 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 72618756f756484e895e885ef12545c8 consensus state has no leader
W20251025 07:52:38.308887 453 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 30418858bf95438686bdfe64ac733144 consensus state has no leader
W20251025 07:52:38.309096 453 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet cb829649f73d4d3f8862d268c953bd86 (table default.Table_ba4b8cc7b930482690b7e192e0edff12 [id=7944225d3ce7439b881bfb9289d9f040], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet cb829649f73d4d3f8862d268c953bd86 consensus state has no leader
W20251025 07:52:38.309469 453 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 04d27d54fdb7483b869408af65ff85c9 (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 04d27d54fdb7483b869408af65ff85c9 consensus state has no leader
W20251025 07:52:38.309734 453 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 4c155aa7fa5b4d68bbab1918b8432372 (table default.Table_6c5a24747ba14352b0285a41ec2985a4 [id=28bbaad474164a8c8c97a76bb8081791], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 4c155aa7fa5b4d68bbab1918b8432372 consensus state has no leader
W20251025 07:52:38.246007 452 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_a6c869949b744bc98efb735b0e73e7aa [id=bf77b5ff6f1048dd996d948437f9ff5e], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet d46673a283fe42b78ddeb99fbd2678eb consensus state has no leader
I20251025 07:52:38.310573 2122 raft_consensus.cc:493] T 6a58931e7cd7454fb583f1c70cd2ad15 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:38.310696 2122 raft_consensus.cc:515] T 6a58931e7cd7454fb583f1c70cd2ad15 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.311437 2122 leader_election.cc:290] T 6a58931e7cd7454fb583f1c70cd2ad15 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
W20251025 07:52:38.312865 452 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 062c0f3e51a3487880cea373f8cf1f54 (table default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 062c0f3e51a3487880cea373f8cf1f54 consensus state has no leader
W20251025 07:52:38.313313 452 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 292fe04055954afd8c08ac2636b68191 (table default.Table_9fa69b10b5e842639fc0b0a58cd9b6a5 [id=a253ce0253bd4da4814741c482193d8f], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 292fe04055954afd8c08ac2636b68191 consensus state has no leader
W20251025 07:52:38.313654 452 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet aa9d9ab8073945d28d52fb02213f201a (table default.Table_4791672658f043f6b97b9320a531ebb5 [id=dfd20cf9c3e541c4abe23d59e0aa7f2a], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet aa9d9ab8073945d28d52fb02213f201a consensus state has no leader
W20251025 07:52:38.313930 452 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet b17d2a9ad6be49d29f2413666fba7e69 (table default.Table_fcddb31989ee4236a4981c12779b3f5d [id=f3794fa081364666b944c412415bdba0], current schema version=3): Not found: failed to reset TS proxy: No leader found: Tablet b17d2a9ad6be49d29f2413666fba7e69 consensus state has no leader
I20251025 07:52:38.313989 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a58931e7cd7454fb583f1c70cd2ad15" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:38.257579 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 4f0c4f0e8f014e59be9e46d2afe0be9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f0c4f0e8f014e59be9e46d2afe0be9d already in progress: creating tablet
W20251025 07:52:38.314497 712 leader_election.cc:343] T 6a58931e7cd7454fb583f1c70cd2ad15 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:38.314493 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:43563, user_credentials={real_user=slave}} blocked reactor thread for 56970.8us
I20251025 07:52:38.316004 2122 raft_consensus.cc:493] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:38.316161 2122 raft_consensus.cc:515] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:38.316470 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 80ee75935d7e461d9d4066fac9819bef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80ee75935d7e461d9d4066fac9819bef already in progress: creating tablet
I20251025 07:52:38.316819 2122 leader_election.cc:290] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:38.317754 884 tablet_service.cc:1505] Processing CreateTablet for tablet a0ac349532534848a9016ad5660e85ae (DEFAULT_TABLE table=default.Table_e72e2534c5c44fd299ed3ec3a367a34c [id=c0903d54e7114c7d83233474e7345a3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:38.318419 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0ac349532534848a9016ad5660e85ae. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:38.320293 453 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 92aa605fabbc4257a6a204c087e7cc7f (table default.Table_78724078b4574f35879fd2db388fc4f4 [id=187bde879b044237ada58158eb000123], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 92aa605fabbc4257a6a204c087e7cc7f consensus state has no leader
W20251025 07:52:38.320493 453 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet c0c6324cbae34dcebade3ef9f89a7150 consensus state has no leader
W20251025 07:52:38.320681 453 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 141e413d2faa4c67b4c822192e631417 consensus state has no leader
W20251025 07:52:38.321820 453 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet fa826b85f0c94150a17d4e417dd3d62f consensus state has no leader
I20251025 07:52:38.322827 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e6544202d4f489d82f22d6073ef240d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.323022 891 tablet_service.cc:1552] Processing DeleteTablet for tablet dda0b26e4f1a44928bc4b84f333901c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.323154 885 tablet_service.cc:1552] Processing DeleteTablet for tablet ffe7c8e1d6c44872b7b65e512764a56a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.323292 878 tablet_service.cc:1552] Processing DeleteTablet for tablet db99a1e68daf4e948a3d66ad276cc5f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.324180 893 tablet_service.cc:1505] Processing CreateTablet for tablet 3c41741ea55842f289bd44762a59c595 (DEFAULT_TABLE table=default.Table_be8af62c5969471b9e7b869c6820db0c [id=5ebbedc74c894c32940ac4492e95909b]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:38.325485 452 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_87d38832f85e4965bfab1a077990114f [id=ad812862698d421184ebf501ce79b7be], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet c501ef5d70e742598c1a902efe47feb1 consensus state has no leader
I20251025 07:52:38.327960 2166 ts_tablet_manager.cc:1916] T 5361f08f50b745eb98bf02e979d7c71e P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:38.327967 448 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 37) took 1405 ms. Trace:
I20251025 07:52:38.328100 448 rpcz_store.cc:276] 1025 07:52:36.922413 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.307290 (+1384877us) service_pool.cc:225] Handling call
1025 07:52:38.308326 (+ 1036us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:38.308772 (+ 446us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:38.327722 (+ 18950us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:38.327824 (+ 102us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:38.327908 (+ 84us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:38.309331 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:38.309489 (+ 158us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:38.309496 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:38.309499 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:38.309582 (+ 83us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:38.309596 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:38.309601 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:38.309671 (+ 70us) tablet.cc:699] Row locks acquired
1025 07:52:38.309673 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:38.309703 (+ 30us) write_op.cc:270] Start()
1025 07:52:38.309738 (+ 35us) write_op.cc:276] Timestamp: P: 1761378758309690 usec, L: 0
1025 07:52:38.309742 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:38.309936 (+ 194us) log.cc:844] Serialized 433 byte log entry
1025 07:52:38.324116 (+ 14180us) op_driver.cc:464] REPLICATION: finished
1025 07:52:38.324232 (+ 116us) write_op.cc:301] APPLY: starting
1025 07:52:38.324251 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:38.324299 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:38.324331 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:38.324520 (+ 189us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:38.324557 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:38.324565 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:38.324657 (+ 92us) log.cc:844] Serialized 29 byte log entry
1025 07:52:38.324722 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.324748 (+ 26us) write_op.cc:454] Released schema lock
1025 07:52:38.324754 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:38.324850 (+ 96us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.324855 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":63,"apply.run_cpu_time_us":605,"apply.run_wall_time_us":605,"num_ops":2,"prepare.queue_time_us":50,"prepare.run_cpu_time_us":703,"prepare.run_wall_time_us":704,"raft.queue_time_us":2422,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":239,"replication_time_us":14306}]]}
I20251025 07:52:38.328933 448 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_4b11b64671124da3aa3ec46fe2cb9983"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:38.329581 448 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b11b64671124da3aa3ec46fe2cb9983 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:38.329823 2106 leader_election.cc:290] T 8f2c8582fed34c848d9903b8ad8b4057 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:38.329903 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 80fe3255ed614270bd8352a2853fbc43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.330061 891 tablet_service.cc:1552] Processing DeleteTablet for tablet ce1ebb1ec14b4801930a72765bca1a7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.331614 875 tablet_service.cc:1505] Processing CreateTablet for tablet ff3daed6c9af47e3976b82b4b9a41086 (DEFAULT_TABLE table=default.Table_6caa55a17907446d81c7f0656b329cdd [id=5b6faf06c72745f68f7c478e50a304d8]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:38.332268 712 leader_election.cc:343] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 5cba9c05dd424cc788febc614a34819a
W20251025 07:52:38.334076 452 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet e8402fd33ba842e981c6c68787c7c548 consensus state has no leader
W20251025 07:52:38.334465 452 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 0325d462441349ef81841f1b0cab2a20 (table default.Table_750ecb905839434fa9a1aa72f87394be [id=9ce3ba77c56a4ef09cb3b40ea0e488f6], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 0325d462441349ef81841f1b0cab2a20 consensus state has no leader
I20251025 07:52:38.334942 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b6383c99e9e4c90a374da9ccd830a6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.335258 1166 tablet_bootstrap.cc:492] T d66ad049d5484228a268b0facb00ecf1 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:38.335672 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 91728987303b41fd840eb56f3a311c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.336911 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a58931e7cd7454fb583f1c70cd2ad15" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:38.337679 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c41741ea55842f289bd44762a59c595. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:38.337680 710 leader_election.cc:343] T 6a58931e7cd7454fb583f1c70cd2ad15 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:38.337782 710 leader_election.cc:304] T 6a58931e7cd7454fb583f1c70cd2ad15 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:38.338227 710 leader_election.cc:343] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 5cba9c05dd424cc788febc614a34819a
I20251025 07:52:38.338313 710 leader_election.cc:304] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:38.336645 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f2c8582fed34c848d9903b8ad8b4057" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:38.339043 776 raft_consensus.cc:3060] T 8f2c8582fed34c848d9903b8ad8b4057 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:38.339056 1659 raft_consensus.cc:2749] T 6a58931e7cd7454fb583f1c70cd2ad15 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:38.339273 2122 raft_consensus.cc:2749] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:38.334820 452 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_e90d7f5ff13b40878e5b625c97792c7d [id=fc8180c09d7b4e74b2e161f7acb8fe0e], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet df0d86ad301245c1a6bcb44805936d25 consensus state has no leader
W20251025 07:52:38.341315 453 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet ed243540f2d746b980c25687fbca7414 consensus state has no leader
I20251025 07:52:38.343186 448 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 38) took 1387 ms. Trace:
I20251025 07:52:38.343356 448 rpcz_store.cc:276] 1025 07:52:36.955429 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.328732 (+1373303us) service_pool.cc:225] Handling call
1025 07:52:38.329673 (+ 941us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:38.330257 (+ 584us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:38.342885 (+ 12628us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:38.342995 (+ 110us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:38.343098 (+ 103us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:38.331025 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:38.331313 (+ 288us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:38.331319 (+ 6us) write_op.cc:435] Acquired schema lock
1025 07:52:38.331321 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:38.331407 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:38.331421 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:38.331425 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:38.331495 (+ 70us) tablet.cc:699] Row locks acquired
1025 07:52:38.331497 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:38.331528 (+ 31us) write_op.cc:270] Start()
1025 07:52:38.331556 (+ 28us) write_op.cc:276] Timestamp: P: 1761378758331513 usec, L: 0
1025 07:52:38.331560 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:38.331713 (+ 153us) log.cc:844] Serialized 433 byte log entry
1025 07:52:38.340226 (+ 8513us) op_driver.cc:464] REPLICATION: finished
1025 07:52:38.340324 (+ 98us) write_op.cc:301] APPLY: starting
1025 07:52:38.340342 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:38.340384 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:38.340415 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:38.340521 (+ 106us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:38.340552 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:38.340560 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:38.340630 (+ 70us) log.cc:844] Serialized 29 byte log entry
1025 07:52:38.340683 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.340703 (+ 20us) write_op.cc:454] Released schema lock
1025 07:52:38.340710 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:38.340768 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.340772 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":50,"apply.run_cpu_time_us":438,"apply.run_wall_time_us":437,"num_ops":2,"prepare.queue_time_us":303,"prepare.run_cpu_time_us":628,"prepare.run_wall_time_us":764,"replication_time_us":8636}]]}
I20251025 07:52:38.343963 1166 tablet_bootstrap.cc:654] T d66ad049d5484228a268b0facb00ecf1 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:38.344431 448 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_1a207ab585e8456bad8fc10a6244d621"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:38.345456 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 050a5d3b5f934dd19bf2e221f2fe0780 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.345649 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 04449cc97f404a19b6a1b20dea074f95 with delete_type TABLET_DATA_DELETED (replaced by tablet f70091024c034c118fb86578db19a31c) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.312915 447 rpcz_store.cc:276] 1025 07:52:36.256190 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:36.264155 (+ 7965us) service_pool.cc:225] Handling call
1025 07:52:36.264941 (+ 786us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:36.265406 (+ 465us) catalog_manager.cc:3708] Apply alter schema
1025 07:52:36.265643 (+ 237us) catalog_manager.cc:3740] Acquired catalog manager lock
1025 07:52:36.265790 (+ 147us) catalog_manager.cc:3898] Updating metadata on disk
1025 07:52:36.718966 (+453176us) catalog_manager.cc:3929] Committing alterations to in-memory state
1025 07:52:38.205585 (+1486619us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:36.280783 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:36.280872 (+ 89us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:36.280877 (+ 5us) write_op.cc:435] Acquired schema lock
1025 07:52:36.280879 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:36.280970 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:36.280990 (+ 20us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:36.280998 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:36.281059 (+ 61us) tablet.cc:699] Row locks acquired
1025 07:52:36.281064 (+ 5us) write_op.cc:260] PREPARE: finished
1025 07:52:36.281096 (+ 32us) write_op.cc:270] Start()
1025 07:52:36.281127 (+ 31us) write_op.cc:276] Timestamp: P: 1761378756281081 usec, L: 0
1025 07:52:36.281130 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:36.281252 (+ 122us) log.cc:844] Serialized 405 byte log entry
1025 07:52:36.712996 (+431744us) op_driver.cc:464] REPLICATION: finished
1025 07:52:36.718217 (+ 5221us) write_op.cc:301] APPLY: starting
1025 07:52:36.718236 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:36.718289 (+ 53us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:36.718331 (+ 42us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:36.718392 (+ 61us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:36.718435 (+ 43us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:36.718457 (+ 22us) write_op.cc:312] APPLY: finished
1025 07:52:36.718558 (+ 101us) log.cc:844] Serialized 23 byte log entry
1025 07:52:36.718841 (+ 283us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:36.718864 (+ 23us) write_op.cc:454] Released schema lock
1025 07:52:36.718873 (+ 9us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:36.719177 (+ 304us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:36.719184 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":5174,"apply.run_cpu_time_us":531,"apply.run_wall_time_us":950,"num_ops":1,"prepare.queue_time_us":14361,"prepare.run_cpu_time_us":542,"prepare.run_wall_time_us":594,"raft.queue_time_us":35,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":431845}]]}
I20251025 07:52:38.347018 447 master_service.cc:954] ReplaceTablet: received request to replace tablet ffe7c8e1d6c44872b7b65e512764a56a from {username='slave'} at 127.0.0.1:47610
I20251025 07:52:38.347290 1167 tablet_bootstrap.cc:492] T e54fe7cbcffa40c4b04144b47bee2b48 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:38.347443 1167 ts_tablet_manager.cc:1403] T e54fe7cbcffa40c4b04144b47bee2b48 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.222s user 0.010s sys 0.034s
I20251025 07:52:38.347569 447 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:47610 (request call id 40) took 1310 ms. Trace:
I20251025 07:52:38.347743 447 rpcz_store.cc:276] 1025 07:52:37.037510 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.346932 (+1309422us) service_pool.cc:225] Handling call
1025 07:52:38.347478 (+ 546us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 07:52:38.348182 447 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_617594400ddd4cf4a8743365e27af63f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:38.348268 1167 raft_consensus.cc:359] T e54fe7cbcffa40c4b04144b47bee2b48 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.348443 1167 raft_consensus.cc:385] T e54fe7cbcffa40c4b04144b47bee2b48 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251025 07:52:38.344700 452 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 37c7938e8c5e41f08506aff6747801c9 consensus state has no leader
I20251025 07:52:38.348498 1167 raft_consensus.cc:740] T e54fe7cbcffa40c4b04144b47bee2b48 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:38.348766 891 tablet_service.cc:1552] Processing DeleteTablet for tablet ebc824ec4c60411dab1a2ea4a385148e with delete_type TABLET_DATA_DELETED (replaced by tablet 29dbaf9149134f548a974a5779f7da67) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.348977 1167 consensus_queue.cc:260] T e54fe7cbcffa40c4b04144b47bee2b48 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
W20251025 07:52:38.349066 447 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_617594400ddd4cf4a8743365e27af63f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:38.349500 1167 ts_tablet_manager.cc:1434] T e54fe7cbcffa40c4b04144b47bee2b48 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:38.349777 1167 tablet_bootstrap.cc:492] T 34315f64b89148efbf1f20a43d5c5efb P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:38.349933 2146 tablet_replica.cc:333] T db99a1e68daf4e948a3d66ad276cc5f9 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
W20251025 07:52:38.349975 448 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a207ab585e8456bad8fc10a6244d621 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:38.351154 452 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 9903e80a26fd469a8744dc855534ba75 consensus state has no leader
I20251025 07:52:38.352206 892 tablet_service.cc:1552] Processing DeleteTablet for tablet aaef24ccdfba45fe8204327723df834e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.352977 776 raft_consensus.cc:2468] T 8f2c8582fed34c848d9903b8ad8b4057 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
W20251025 07:52:38.343823 453 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 8f2c8582fed34c848d9903b8ad8b4057 (table default.Table_438f67f3dc62465999a917986922b3ed [id=7c7fcd265e624d7fab74661db505d9c3], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 8f2c8582fed34c848d9903b8ad8b4057 consensus state has no leader
I20251025 07:52:38.353984 883 tablet_service.cc:1552] Processing DeleteTablet for tablet f97acc9a638b4d5a968dcc3412a384bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.354573 885 tablet_service.cc:1552] Processing DeleteTablet for tablet d37758f606344127b3db74c87babe69c with delete_type TABLET_DATA_DELETED (replaced by tablet 07898860aa594a05a211fc89768cb76d) from {username='slave'} at 127.0.0.1:59616
W20251025 07:52:38.355298 452 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69], current schema version=3): Not found: failed to reset TS proxy: No leader found: Tablet 48b73e0c0dce4be686d05f991a21bd19 consensus state has no leader
I20251025 07:52:38.357301 1167 tablet_bootstrap.cc:654] T 34315f64b89148efbf1f20a43d5c5efb P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:38.357401 886 tablet_service.cc:1552] Processing DeleteTablet for tablet d9bfd7c0c3ad4e5b98b6708793d61d41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.357547 2146 raft_consensus.cc:2243] T db99a1e68daf4e948a3d66ad276cc5f9 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:38.359306 2169 ts_tablet_manager.cc:1929] T 5361f08f50b745eb98bf02e979d7c71e P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:38.359534 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 0f5f6ca6c3cd4d369690e425387a4344 with delete_type TABLET_DATA_DELETED (replaced by tablet 22fc90c704f64d2f9f4b8c54dee45576) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.360198 878 tablet_service.cc:1552] Processing DeleteTablet for tablet f43ef51ee45d47a99ec732b49d50d3b4 with delete_type TABLET_DATA_DELETED (replaced by tablet e429b94326cc4827a03968fd92de38e0) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.360378 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 410ef150a7014c6c867a021b544a2f4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.360543 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b940fa5ae26427dafbcfd26372754b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.360677 886 tablet_service.cc:1552] Processing DeleteTablet for tablet 26c849d376d143c6bef0c8e0e1d7d03a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.360832 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 88eb51447e0f4080baa49dd902deb24f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.360963 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff3daed6c9af47e3976b82b4b9a41086. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:38.365458 442 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 0325d462441349ef81841f1b0cab2a20 (table default.Table_750ecb905839434fa9a1aa72f87394be [id=9ce3ba77c56a4ef09cb3b40ea0e488f6], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0325d462441349ef81841f1b0cab2a20 consensus state has no leader
I20251025 07:52:38.365698 2169 log.cc:1199] T 5361f08f50b745eb98bf02e979d7c71e P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/5361f08f50b745eb98bf02e979d7c71e
I20251025 07:52:38.367091 1166 tablet_bootstrap.cc:492] T d66ad049d5484228a268b0facb00ecf1 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:38.367251 1166 ts_tablet_manager.cc:1403] T d66ad049d5484228a268b0facb00ecf1 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20251025 07:52:38.367511 2169 ts_tablet_manager.cc:1950] T 5361f08f50b745eb98bf02e979d7c71e P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:38.367897 883 tablet_service.cc:1552] Processing DeleteTablet for tablet b3a0af893a8947848e12093d0a7f0229 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.370450 889 tablet_service.cc:1552] Processing DeleteTablet for tablet f656945454fe4921ab819c772a089eb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.370630 847 leader_election.cc:304] T 8f2c8582fed34c848d9903b8ad8b4057 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:38.371088 891 tablet_service.cc:1552] Processing DeleteTablet for tablet e54fe7cbcffa40c4b04144b47bee2b48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.371232 876 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f26667f053a4f16b5da330c0b5e40f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 34315f64b89148efbf1f20a43d5c5efb) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.369423 892 tablet_service.cc:1552] Processing DeleteTablet for tablet f661d5909657455581dcb5e6eadd3c0b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.371869 2106 raft_consensus.cc:2804] T 8f2c8582fed34c848d9903b8ad8b4057 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
W20251025 07:52:38.372663 453 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 36262ab8c6964dd684c01b4db6219a4f (table default.Table_76158f8cf82e4cdb9a90580e46aaea48 [id=f249f30497bc450c887d659721469de8], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 36262ab8c6964dd684c01b4db6219a4f consensus state has no leader
W20251025 07:52:38.373672 453 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 2c50ec95e48e49b59a76fac4fc862d3d (table default.Table_32b54e82cbc84d39a0020826f8737d62 [id=c28b0b6acd61490caecfdaab803a9d92], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 2c50ec95e48e49b59a76fac4fc862d3d consensus state has no leader
W20251025 07:52:38.373860 453 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet b97e39aaac3048ed93dbfe873b757254 (table default.Table_99fd36369f9a4e0fb14f234c98635203 [id=c8059ab6cec146efa631d06799d969a9], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet b97e39aaac3048ed93dbfe873b757254 consensus state has no leader
I20251025 07:52:38.374100 2166 ts_tablet_manager.cc:1929] T 5361f08f50b745eb98bf02e979d7c71e P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:38.374327 2166 log.cc:1199] T 5361f08f50b745eb98bf02e979d7c71e P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/5361f08f50b745eb98bf02e979d7c71e
I20251025 07:52:38.374701 2166 ts_tablet_manager.cc:1950] T 5361f08f50b745eb98bf02e979d7c71e P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:38.375610 1659 raft_consensus.cc:493] T fe5033e5ff404a5ebc6c8443d7548327 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 07:52:38.375802 453 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 5361f08f50b745eb98bf02e979d7c71e (table default.Table_a3f50508f4b047b6837c67248417b21c [id=572bc5595be947468e24f2a9144f12a4], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 5361f08f50b745eb98bf02e979d7c71e consensus state has no leader
I20251025 07:52:38.375764 1659 raft_consensus.cc:515] T fe5033e5ff404a5ebc6c8443d7548327 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.376431 1659 leader_election.cc:290] T fe5033e5ff404a5ebc6c8443d7548327 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:38.376893 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5033e5ff404a5ebc6c8443d7548327" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:38.377427 712 leader_election.cc:343] T fe5033e5ff404a5ebc6c8443d7548327 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:38.379987 454 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:47610 (request call id 35) took 2108 ms. Trace:
I20251025 07:52:38.380117 454 rpcz_store.cc:276] 1025 07:52:36.271340 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.205227 (+1933887us) service_pool.cc:225] Handling call
1025 07:52:38.379761 (+174534us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:38.379914 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 07:52:38.380470 454 master_service.cc:954] ReplaceTablet: received request to replace tablet 3c4970c8a491460e8b4b1c29fde83e4c from {username='slave'} at 127.0.0.1:47610
I20251025 07:52:38.381523 447 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 41) took 1329 ms. Trace:
I20251025 07:52:38.381841 2146 raft_consensus.cc:2272] T db99a1e68daf4e948a3d66ad276cc5f9 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:38.381672 447 rpcz_store.cc:276] 1025 07:52:37.051667 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.347984 (+1296317us) service_pool.cc:225] Handling call
1025 07:52:38.373089 (+ 25105us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:38.373641 (+ 552us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:38.381215 (+ 7574us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:38.381361 (+ 146us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:38.381472 (+ 111us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:38.374289 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:38.374410 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:38.374418 (+ 8us) write_op.cc:435] Acquired schema lock
1025 07:52:38.374421 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:38.374547 (+ 126us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:38.374565 (+ 18us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:38.374573 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:38.374668 (+ 95us) tablet.cc:699] Row locks acquired
1025 07:52:38.374671 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:38.374710 (+ 39us) write_op.cc:270] Start()
1025 07:52:38.374751 (+ 41us) write_op.cc:276] Timestamp: P: 1761378758374694 usec, L: 0
1025 07:52:38.374755 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:38.374983 (+ 228us) log.cc:844] Serialized 433 byte log entry
1025 07:52:38.380230 (+ 5247us) op_driver.cc:464] REPLICATION: finished
1025 07:52:38.380499 (+ 269us) write_op.cc:301] APPLY: starting
1025 07:52:38.380520 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:38.380575 (+ 55us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:38.380608 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:38.380749 (+ 141us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:38.380792 (+ 43us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:38.380803 (+ 11us) write_op.cc:312] APPLY: finished
1025 07:52:38.380896 (+ 93us) log.cc:844] Serialized 29 byte log entry
1025 07:52:38.380969 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.380995 (+ 26us) write_op.cc:454] Released schema lock
1025 07:52:38.381004 (+ 9us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:38.381087 (+ 83us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.381093 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":201,"apply.run_cpu_time_us":580,"apply.run_wall_time_us":580,"num_ops":2,"prepare.queue_time_us":49,"prepare.run_cpu_time_us":798,"prepare.run_wall_time_us":797,"raft.queue_time_us":1608,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":228,"replication_time_us":5437}]]}
I20251025 07:52:38.392542 2106 raft_consensus.cc:697] T 8f2c8582fed34c848d9903b8ad8b4057 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:38.392699 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5033e5ff404a5ebc6c8443d7548327" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:38.393298 748 tablet_service.cc:1505] Processing CreateTablet for tablet 3c29d2fe91e54aa9a40e50f38b710c36 (DEFAULT_TABLE table=default.Table_6eff76f8460c40668372f850e2fc299a [id=c74693e799854673b6865d11b914477a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:38.393716 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c29d2fe91e54aa9a40e50f38b710c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:38.394781 447 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_f56958ab784443f39df5bef925069408"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:38.395051 710 leader_election.cc:343] T fe5033e5ff404a5ebc6c8443d7548327 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:38.395223 710 leader_election.cc:304] T fe5033e5ff404a5ebc6c8443d7548327 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:38.395617 1659 raft_consensus.cc:2749] T fe5033e5ff404a5ebc6c8443d7548327 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:38.398353 447 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f56958ab784443f39df5bef925069408 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:38.402976 1167 tablet_bootstrap.cc:492] T 34315f64b89148efbf1f20a43d5c5efb P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:38.403131 1167 ts_tablet_manager.cc:1403] T 34315f64b89148efbf1f20a43d5c5efb P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.053s user 0.006s sys 0.004s
I20251025 07:52:38.403874 1167 raft_consensus.cc:359] T 34315f64b89148efbf1f20a43d5c5efb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.404080 1167 raft_consensus.cc:385] T 34315f64b89148efbf1f20a43d5c5efb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:38.404143 1167 raft_consensus.cc:740] T 34315f64b89148efbf1f20a43d5c5efb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:38.388204 448 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 39) took 1385 ms. Trace:
I20251025 07:52:38.404361 1167 consensus_queue.cc:260] T 34315f64b89148efbf1f20a43d5c5efb P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.403317 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 5361f08f50b745eb98bf02e979d7c71e (table default.Table_a3f50508f4b047b6837c67248417b21c [id=572bc5595be947468e24f2a9144f12a4]) successfully deleted
I20251025 07:52:38.404942 1167 ts_tablet_manager.cc:1434] T 34315f64b89148efbf1f20a43d5c5efb P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:38.404364 448 rpcz_store.cc:276] 1025 07:52:37.002313 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.344226 (+1341913us) service_pool.cc:225] Handling call
1025 07:52:38.376978 (+ 32752us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:38.377496 (+ 518us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:38.387832 (+ 10336us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:38.387989 (+ 157us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:38.388107 (+ 118us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:38.378049 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:38.378192 (+ 143us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:38.378199 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:38.378201 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:38.378289 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:38.378301 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:38.378306 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:38.378387 (+ 81us) tablet.cc:699] Row locks acquired
1025 07:52:38.378390 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:38.378422 (+ 32us) write_op.cc:270] Start()
1025 07:52:38.378449 (+ 27us) write_op.cc:276] Timestamp: P: 1761378758378408 usec, L: 0
1025 07:52:38.378452 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:38.378603 (+ 151us) log.cc:844] Serialized 433 byte log entry
1025 07:52:38.386847 (+ 8244us) op_driver.cc:464] REPLICATION: finished
1025 07:52:38.387106 (+ 259us) write_op.cc:301] APPLY: starting
1025 07:52:38.387123 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:38.387189 (+ 66us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:38.387229 (+ 40us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:38.387375 (+ 146us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:38.387417 (+ 42us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:38.387428 (+ 11us) write_op.cc:312] APPLY: finished
1025 07:52:38.387526 (+ 98us) log.cc:844] Serialized 29 byte log entry
1025 07:52:38.387599 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.387626 (+ 27us) write_op.cc:454] Released schema lock
1025 07:52:38.387634 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:38.387716 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.387722 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":201,"apply.run_cpu_time_us":598,"apply.run_wall_time_us":598,"num_ops":2,"prepare.queue_time_us":46,"prepare.run_cpu_time_us":632,"prepare.run_wall_time_us":631,"replication_time_us":8357}]]}
I20251025 07:52:38.406052 448 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_1a037b3b43c74608a119c8f6c2007fcc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:38.406761 448 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a037b3b43c74608a119c8f6c2007fcc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:38.409731 1659 raft_consensus.cc:493] T ef9a2c9f2ccf432ab835396020872208 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:38.409857 1659 raft_consensus.cc:515] T ef9a2c9f2ccf432ab835396020872208 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.410451 1659 leader_election.cc:290] T ef9a2c9f2ccf432ab835396020872208 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:38.410851 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef9a2c9f2ccf432ab835396020872208" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:38.411064 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f2c8582fed34c848d9903b8ad8b4057" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:38.411341 712 leader_election.cc:343] T ef9a2c9f2ccf432ab835396020872208 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:38.412124 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f2c8582fed34c848d9903b8ad8b4057" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:38.414708 1095 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
I20251025 07:52:38.414745 1169 ts_tablet_manager.cc:1434] T d66ad049d5484228a268b0facb00ecf1 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.316s user 0.009s sys 0.004s
I20251025 07:52:38.415941 1167 tablet_bootstrap.cc:492] T b947ecc352e54f4caa73c6ca97372256 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:38.416203 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 5361f08f50b745eb98bf02e979d7c71e (table default.Table_a3f50508f4b047b6837c67248417b21c [id=572bc5595be947468e24f2a9144f12a4]) successfully deleted
W20251025 07:52:38.416467 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 92aa605fabbc4257a6a204c087e7cc7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 92aa605fabbc4257a6a204c087e7cc7f already in progress: creating tablet
I20251025 07:52:38.417366 2146 ts_tablet_manager.cc:1916] T db99a1e68daf4e948a3d66ad276cc5f9 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:38.417749 1167 tablet_bootstrap.cc:654] T b947ecc352e54f4caa73c6ca97372256 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:38.393821 452 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 075f0a151e024220b51ba7fcaecdcc3d consensus state has no leader
W20251025 07:52:38.420157 452 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet bc2d16b33ca8426abbd7bd38339c9f1d (table default.Table_4ea985df4c5b415c9570c64506a98cc4 [id=98d2b7797e33485e8872054aa2eb8ec9], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet bc2d16b33ca8426abbd7bd38339c9f1d consensus state has no leader
W20251025 07:52:38.420480 452 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet ef1854d35a3047e3a9fbe0166ab1295a (table default.Table_088aad2a46b041bfa29ed8e6929760b1 [id=834669c54dae449790818684fd8659bc], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet ef1854d35a3047e3a9fbe0166ab1295a consensus state has no leader
I20251025 07:52:38.420624 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef9a2c9f2ccf432ab835396020872208" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:38.421434 452 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet d0240d20f9904b4bbc8bf7f0f05e8850 consensus state has no leader
W20251025 07:52:38.421628 452 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet ef9a2c9f2ccf432ab835396020872208 (table default.Table_2b9b772d4c234ac286bef3658482e490 [id=409fab60f9d744798e6f92b868f0f7c2], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet ef9a2c9f2ccf432ab835396020872208 consensus state has no leader
I20251025 07:52:38.422923 447 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 43) took 1352 ms. Trace:
I20251025 07:52:38.423048 447 rpcz_store.cc:276] 1025 07:52:37.070453 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.394588 (+1324135us) service_pool.cc:225] Handling call
1025 07:52:38.404323 (+ 9735us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:38.404819 (+ 496us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:38.419425 (+ 14606us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:38.419535 (+ 110us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:38.422889 (+ 3354us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:38.410588 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:38.410747 (+ 159us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:38.410753 (+ 6us) write_op.cc:435] Acquired schema lock
1025 07:52:38.410756 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:38.410842 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:38.410853 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:38.410857 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:38.410922 (+ 65us) tablet.cc:699] Row locks acquired
1025 07:52:38.410924 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:38.410958 (+ 34us) write_op.cc:270] Start()
1025 07:52:38.410992 (+ 34us) write_op.cc:276] Timestamp: P: 1761378758410943 usec, L: 0
1025 07:52:38.410995 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:38.411216 (+ 221us) log.cc:844] Serialized 433 byte log entry
1025 07:52:38.416688 (+ 5472us) op_driver.cc:464] REPLICATION: finished
1025 07:52:38.416778 (+ 90us) write_op.cc:301] APPLY: starting
1025 07:52:38.416795 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:38.416844 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:38.416868 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:38.416979 (+ 111us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:38.417014 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:38.417022 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:38.417098 (+ 76us) log.cc:844] Serialized 29 byte log entry
1025 07:52:38.417185 (+ 87us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.417204 (+ 19us) write_op.cc:454] Released schema lock
1025 07:52:38.417211 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:38.417292 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.417296 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":49,"apply.run_cpu_time_us":503,"apply.run_wall_time_us":502,"num_ops":2,"prepare.queue_time_us":5184,"prepare.run_cpu_time_us":730,"prepare.run_wall_time_us":730,"raft.queue_time_us":194,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":233,"replication_time_us":5665}]]}
I20251025 07:52:38.423825 447 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_a88d62768a0043398282bc79b0d5c96f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:38.424218 710 leader_election.cc:343] T ef9a2c9f2ccf432ab835396020872208 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:38.424341 710 leader_election.cc:304] T ef9a2c9f2ccf432ab835396020872208 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:38.424440 447 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a88d62768a0043398282bc79b0d5c96f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:38.424577 2124 raft_consensus.cc:2749] T ef9a2c9f2ccf432ab835396020872208 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:38.425593 876 tablet_service.cc:1552] Processing DeleteTablet for tablet fa77085e81c7440d9e255c44d1e80c26 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.425772 1169 tablet_bootstrap.cc:492] T 8f2c8582fed34c848d9903b8ad8b4057 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:38.425865 891 tablet_service.cc:1552] Processing DeleteTablet for tablet a6ddefa1738f4ff98586900c344359d2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f0c4f0e8f014e59be9e46d2afe0be9d) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.426261 883 tablet_service.cc:1552] Processing DeleteTablet for tablet f0bdaa8cfd764f1b83f47881353d26b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.426249 2183 consensus_queue.cc:1048] T 53198283718d4b45b30df65b5bde96a6 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:38.426463 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 8956ad28b35d45f28c683ac055bba441 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.426095 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b21b7da1e2f47829c3f6cdaee689e8a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.427023 881 tablet_service.cc:1552] Processing DeleteTablet for tablet c11f674385184ae6b235f1db8bf1fb91 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.427496 883 tablet_service.cc:1552] Processing DeleteTablet for tablet fd265cdbf6a8440393a2b5d8240d1872 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.427698 2106 consensus_queue.cc:237] T 8f2c8582fed34c848d9903b8ad8b4057 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.430419 1169 tablet_bootstrap.cc:654] T 8f2c8582fed34c848d9903b8ad8b4057 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:38.430577 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 48ff4ce7846b4e56a37eee9bcb17bd5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.430866 881 tablet_service.cc:1552] Processing DeleteTablet for tablet d77704b4fe6c4456a2ba40ffec0c60fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.431499 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 887e7c7a5d0e4aaf88f26246306df19c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.431768 881 tablet_service.cc:1552] Processing DeleteTablet for tablet e4da89f863cc43f3bf6df908bb9dc6bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.432282 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f7c15529be54a11961f385a953b0b03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.432535 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 72dc084f8efc45478771c247e07a16f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
W20251025 07:52:38.432711 453 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet ab42909afc524f2fa3c1041495d13a59 (table default.Table_7eb8424d5192485da1a2d67bfdb2bbbf [id=40c29df99590408cbd5c583d4fc741eb], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet ab42909afc524f2fa3c1041495d13a59 consensus state has no leader
I20251025 07:52:38.432919 883 tablet_service.cc:1552] Processing DeleteTablet for tablet be5ab0db6ac34c2c9a370445e251eb46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.433419 894 tablet_service.cc:1552] Processing DeleteTablet for tablet b772025852794a49b072f37cccbd3291 with delete_type TABLET_DATA_DELETED (replaced by tablet f97acc9a638b4d5a968dcc3412a384bc) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.433631 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c76d1b509f04ab1a10b83cff675c11e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.437091 448 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 44) took 1339 ms. Trace:
I20251025 07:52:38.437320 448 rpcz_store.cc:276] 1025 07:52:37.097628 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.405795 (+1308167us) service_pool.cc:225] Handling call
1025 07:52:38.406910 (+ 1115us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:38.407323 (+ 413us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:38.436181 (+ 28858us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:38.436767 (+ 586us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:38.436975 (+ 208us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:38.411873 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:38.411966 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:38.411972 (+ 6us) write_op.cc:435] Acquired schema lock
1025 07:52:38.411974 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:38.412070 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:38.412080 (+ 10us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:38.412085 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:38.412191 (+ 106us) tablet.cc:699] Row locks acquired
1025 07:52:38.412194 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:38.412224 (+ 30us) write_op.cc:270] Start()
1025 07:52:38.412250 (+ 26us) write_op.cc:276] Timestamp: P: 1761378758412211 usec, L: 0
1025 07:52:38.412254 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:38.412386 (+ 132us) log.cc:844] Serialized 433 byte log entry
1025 07:52:38.435165 (+ 22779us) op_driver.cc:464] REPLICATION: finished
1025 07:52:38.435307 (+ 142us) write_op.cc:301] APPLY: starting
1025 07:52:38.435332 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:38.435393 (+ 61us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:38.435444 (+ 51us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:38.435608 (+ 164us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:38.435656 (+ 48us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:38.435668 (+ 12us) write_op.cc:312] APPLY: finished
1025 07:52:38.435774 (+ 106us) log.cc:844] Serialized 29 byte log entry
1025 07:52:38.435854 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.435885 (+ 31us) write_op.cc:454] Released schema lock
1025 07:52:38.435895 (+ 10us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:38.435984 (+ 89us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.435990 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":76,"apply.run_cpu_time_us":663,"apply.run_wall_time_us":663,"mutex_wait_us":3982,"num_ops":2,"prepare.queue_time_us":4022,"prepare.run_cpu_time_us":577,"prepare.run_wall_time_us":577,"replication_time_us":22858}]]}
I20251025 07:52:38.439597 1167 tablet_bootstrap.cc:492] T b947ecc352e54f4caa73c6ca97372256 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:38.439730 1167 ts_tablet_manager.cc:1403] T b947ecc352e54f4caa73c6ca97372256 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.003s
I20251025 07:52:38.440091 448 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_dd5cb130dd9441929fa1e3efe915743b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:38.440500 1167 raft_consensus.cc:359] T b947ecc352e54f4caa73c6ca97372256 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.440690 1167 raft_consensus.cc:385] T b947ecc352e54f4caa73c6ca97372256 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:38.440774 1167 raft_consensus.cc:740] T b947ecc352e54f4caa73c6ca97372256 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:38.441069 1167 consensus_queue.cc:260] T b947ecc352e54f4caa73c6ca97372256 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
W20251025 07:52:38.441615 448 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd5cb130dd9441929fa1e3efe915743b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:38.441644 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 4037dee385d04e85bb78c8e77cb00379 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.441704 1167 ts_tablet_manager.cc:1434] T b947ecc352e54f4caa73c6ca97372256 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:38.441838 1169 tablet_bootstrap.cc:492] T 8f2c8582fed34c848d9903b8ad8b4057 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:38.441982 1169 ts_tablet_manager.cc:1403] T 8f2c8582fed34c848d9903b8ad8b4057 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.001s
I20251025 07:52:38.441967 1167 tablet_bootstrap.cc:492] T 7c7a9d7a4fb3449bb57eafa83230a839 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:38.442272 884 tablet_service.cc:1552] Processing DeleteTablet for tablet e47c4f741f354a19b49777bd06e54764 with delete_type TABLET_DATA_DELETED (replaced by tablet 755f0d6c87044e20a8020f8410940123) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.443526 881 tablet_service.cc:1552] Processing DeleteTablet for tablet cc407c7eb12e4eadb41fb21385611c94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.444271 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 7f29b64704e147aa9b0b5f6cda860f09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.444634 1167 tablet_bootstrap.cc:654] T 7c7a9d7a4fb3449bb57eafa83230a839 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:38.444654 1169 raft_consensus.cc:359] T 8f2c8582fed34c848d9903b8ad8b4057 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.446111 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 716fb61312e94a4d9eb8ab8336ca140c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.446488 881 tablet_service.cc:1552] Processing DeleteTablet for tablet bef0ef23d90d480d9c99aef21d7271fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.446848 881 tablet_service.cc:1552] Processing DeleteTablet for tablet abf94c32ff2945119b710905c075d63c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.447213 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 1912e59548874e0e8e989050ec0064c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.448431 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e098c9c4613459295b6043048cba587 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.448978 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 6dcab53a020847a1aa27930265ba0205 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.449224 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f25c9caa05b4da4bd56a8a21fde423c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.450052 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 1b5ac7e845a24b2eb5b2429d47361232 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.451614 883 tablet_service.cc:1505] Processing CreateTablet for tablet 3c29d2fe91e54aa9a40e50f38b710c36 (DEFAULT_TABLE table=default.Table_6eff76f8460c40668372f850e2fc299a [id=c74693e799854673b6865d11b914477a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:38.452154 884 tablet_service.cc:1552] Processing DeleteTablet for tablet a7ab3150a35b4caca9119a1374d3f2f7 with delete_type TABLET_DATA_DELETED (replaced by tablet c19c97f881cb45218d4a6f6f231aa96e) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.453689 881 tablet_service.cc:1552] Processing DeleteTablet for tablet cb824bd780524832b2b60f5618a39da5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.454360 881 tablet_service.cc:1552] Processing DeleteTablet for tablet ee1c8ea694284d80a30f9c05be4cbf49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.444850 1169 raft_consensus.cc:385] T 8f2c8582fed34c848d9903b8ad8b4057 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:38.455170 881 tablet_service.cc:1552] Processing DeleteTablet for tablet fab67b56eaa44b54886c17181d8c52b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.455217 1169 raft_consensus.cc:740] T 8f2c8582fed34c848d9903b8ad8b4057 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:38.455849 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 27651312cc2e42a8bfcd245f5d301679 with delete_type TABLET_DATA_DELETED (replaced by tablet a4e4b99be0434baaa75d988281797846) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.456060 1169 consensus_queue.cc:260] T 8f2c8582fed34c848d9903b8ad8b4057 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.456594 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b6383c99e9e4c90a374da9ccd830a6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.456961 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 80fe3255ed614270bd8352a2853fbc43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:58412
W20251025 07:52:38.457587 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 5b6383c99e9e4c90a374da9ccd830a6f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b6383c99e9e4c90a374da9ccd830a6f already in progress: creating tablet
W20251025 07:52:38.457808 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 80fe3255ed614270bd8352a2853fbc43 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80fe3255ed614270bd8352a2853fbc43 already in progress: creating tablet
I20251025 07:52:38.458623 748 tablet_service.cc:1552] Processing DeleteTablet for tablet ebc824ec4c60411dab1a2ea4a385148e with delete_type TABLET_DATA_DELETED (replaced by tablet 29dbaf9149134f548a974a5779f7da67) from {username='slave'} at 127.0.0.1:58412
W20251025 07:52:38.459180 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet ebc824ec4c60411dab1a2ea4a385148e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ebc824ec4c60411dab1a2ea4a385148e already in progress: creating tablet
I20251025 07:52:38.459187 893 tablet_service.cc:1552] Processing DeleteTablet for tablet b947ecc352e54f4caa73c6ca97372256 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.459465 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 896bd7870d87408e8adfdb7a78998111 with delete_type TABLET_DATA_DELETED (replaced by tablet e8402fd33ba842e981c6c68787c7c548) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.459523 2182 consensus_queue.cc:1048] T 896bd7870d87408e8adfdb7a78998111 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:38.460186 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 95efd553493e4966bc48a01cac6e1e10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.460536 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 467324e2432f4caca148951329caa292 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.460857 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d261e432cf5403ea162129706bc4d4d with delete_type TABLET_DATA_DELETED (replaced by tablet e54fe7cbcffa40c4b04144b47bee2b48) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.462786 881 tablet_service.cc:1552] Processing DeleteTablet for tablet b94634ac5077439fadfbbce569f60e8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.466318 2146 ts_tablet_manager.cc:1929] T db99a1e68daf4e948a3d66ad276cc5f9 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:38.466534 2146 log.cc:1199] T db99a1e68daf4e948a3d66ad276cc5f9 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/db99a1e68daf4e948a3d66ad276cc5f9
I20251025 07:52:38.467283 2146 ts_tablet_manager.cc:1950] T db99a1e68daf4e948a3d66ad276cc5f9 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:38.467293 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5]). Expected version 1 got 0
I20251025 07:52:38.467993 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3]). Expected version 1 got 0
I20251025 07:52:38.468556 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_a6c869949b744bc98efb735b0e73e7aa [id=bf77b5ff6f1048dd996d948437f9ff5e]). Expected version 2 got 0
W20251025 07:52:38.470320 452 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:42387 (request call id 209) took 2045 ms (client timeout 1000 ms). Trace:
W20251025 07:52:38.470594 452 rpcz_store.cc:269] 1025 07:52:36.424843 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:36.690982 (+266139us) service_pool.cc:225] Handling call
1025 07:52:36.811810 (+120828us) spinlock_profiling.cc:243] Waited 80 ms on lock 0x616000041254. stack: 00007fb90f7779f6 00007fb90f7777b0 00007fb90e7533ad 0000000000983f04 0000000000983ecd 00007fb91ea756bd 00007fb91ea757b3 00007fb91b640441 00007fb9232f69dd 00007fb92347daa8 00007fb919f96ca6 00007fb919f969d5 00007fb91671327e 00007fb91671215e 00007fb916718395 00007fb916718cbf
1025 07:52:38.470254 (+1658444us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:37.672317 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:37.672449 (+ 132us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:37.672457 (+ 8us) write_op.cc:435] Acquired schema lock
1025 07:52:37.672459 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:37.672568 (+ 109us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:37.672583 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:37.672588 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:37.672654 (+ 66us) tablet.cc:699] Row locks acquired
1025 07:52:37.672656 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:37.672688 (+ 32us) write_op.cc:270] Start()
1025 07:52:37.672723 (+ 35us) write_op.cc:276] Timestamp: P: 1761378757672673 usec, L: 0
1025 07:52:37.672726 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:37.672948 (+ 222us) log.cc:844] Serialized 527 byte log entry
1025 07:52:37.677067 (+ 4119us) op_driver.cc:464] REPLICATION: finished
1025 07:52:37.677259 (+ 192us) write_op.cc:301] APPLY: starting
1025 07:52:37.677281 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:37.677339 (+ 58us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:37.677376 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:37.677462 (+ 86us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:37.677511 (+ 49us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:37.677523 (+ 12us) write_op.cc:312] APPLY: finished
1025 07:52:37.677626 (+ 103us) log.cc:844] Serialized 23 byte log entry
1025 07:52:37.677704 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:37.677733 (+ 29us) write_op.cc:454] Released schema lock
1025 07:52:37.677742 (+ 9us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:37.677829 (+ 87us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:37.677835 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3586,"reported_tablets":239,"spinlock_wait_cycles":183956224,"child_traces":[["op",{"apply.queue_time_us":130,"apply.run_cpu_time_us":559,"apply.run_wall_time_us":560,"num_ops":1,"prepare.queue_time_us":265,"prepare.run_cpu_time_us":747,"prepare.run_wall_time_us":748,"raft.queue_time_us":1117,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":227,"replication_time_us":4300}]]}
I20251025 07:52:38.468389 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 3775a2e08d4e40bd9f57e4cda759d535 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.468662 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 2784bbfb97d5466f9ec856add7d0b29a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.472208 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 91728987303b41fd840eb56f3a311c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.468771 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 943ec703e65f4fd3b0c690e08c0bd226 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.469107 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 627a10a0714f4be8bdbced503fd6388d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:33 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.469632 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 35ba9d23262248cda6fdf03c51be11a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 01f82980559b4662bdc09786ed429ddc) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.470744 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 2209089667dd4937abd96615f6929449 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.471625 876 tablet_service.cc:1552] Processing DeleteTablet for tablet 74e2d78c024849b0af5c0bde1f48c36f with delete_type TABLET_DATA_DELETED (replaced by tablet fd265cdbf6a8440393a2b5d8240d1872) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.472397 452 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_834c71dad0864775a527dda8c2247132"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:38.473353 452 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_834c71dad0864775a527dda8c2247132 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:38.456193 453 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 4b28215f61574cf48c447dd01955ac77 (table default.Table_a437a2623d704e3ea3c5b8c65c88b936 [id=51b12903e6024e1a9adc2f9fa7400937], current schema version=3): Not found: failed to reset TS proxy: No leader found: Tablet 4b28215f61574cf48c447dd01955ac77 consensus state has no leader
W20251025 07:52:38.474007 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 91728987303b41fd840eb56f3a311c5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91728987303b41fd840eb56f3a311c5f already in progress: creating tablet
I20251025 07:52:38.474782 1169 ts_tablet_manager.cc:1434] T 8f2c8582fed34c848d9903b8ad8b4057 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.028s user 0.001s sys 0.001s
I20251025 07:52:38.475597 748 tablet_service.cc:1552] Processing DeleteTablet for tablet ce1ebb1ec14b4801930a72765bca1a7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
W20251025 07:52:38.476176 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet ce1ebb1ec14b4801930a72765bca1a7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce1ebb1ec14b4801930a72765bca1a7d already in progress: creating tablet
I20251025 07:52:38.476913 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 0f5f6ca6c3cd4d369690e425387a4344 with delete_type TABLET_DATA_DELETED (replaced by tablet 22fc90c704f64d2f9f4b8c54dee45576) from {username='slave'} at 127.0.0.1:58412
W20251025 07:52:38.477159 453 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:38361 (request call id 207) took 2205 ms (client timeout 1000 ms). Trace:
W20251025 07:52:38.477474 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 0f5f6ca6c3cd4d369690e425387a4344 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f5f6ca6c3cd4d369690e425387a4344 already in progress: creating tablet
I20251025 07:52:38.478083 447 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 45) took 1380 ms. Trace:
I20251025 07:52:38.479127 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c29d2fe91e54aa9a40e50f38b710c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:38.478518 447 rpcz_store.cc:276] 1025 07:52:37.097801 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.423645 (+1325844us) service_pool.cc:225] Handling call
1025 07:52:38.424532 (+ 887us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:38.425012 (+ 480us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:38.477507 (+ 52495us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.477546 (+ 39us) write_op.cc:454] Released schema lock
1025 07:52:38.477693 (+ 147us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:38.477880 (+ 187us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:38.478029 (+ 149us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:38.456913 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:38.457050 (+ 137us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:38.457057 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:38.457059 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:38.457168 (+ 109us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:38.457182 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:38.457186 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:38.457269 (+ 83us) tablet.cc:699] Row locks acquired
1025 07:52:38.457272 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:38.457310 (+ 38us) write_op.cc:270] Start()
1025 07:52:38.457348 (+ 38us) write_op.cc:276] Timestamp: P: 1761378758457294 usec, L: 0
1025 07:52:38.457351 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:38.457540 (+ 189us) log.cc:844] Serialized 433 byte log entry
1025 07:52:38.463360 (+ 5820us) op_driver.cc:464] REPLICATION: finished
1025 07:52:38.463546 (+ 186us) write_op.cc:301] APPLY: starting
1025 07:52:38.463564 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:38.463622 (+ 58us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:38.463655 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:38.463773 (+ 118us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:38.463809 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:38.463818 (+ 9us) write_op.cc:312] APPLY: finished
1025 07:52:38.463895 (+ 77us) log.cc:844] Serialized 29 byte log entry
1025 07:52:38.463966 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.463989 (+ 23us) write_op.cc:454] Released schema lock
1025 07:52:38.463997 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":133,"apply.run_cpu_time_us":509,"apply.run_wall_time_us":509,"num_ops":2,"prepare.queue_time_us":31376,"prepare.run_cpu_time_us":746,"prepare.run_wall_time_us":746,"raft.queue_time_us":65,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":210,"replication_time_us":5965}]]}
I20251025 07:52:38.480298 447 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_ae530c9ed4814fb999a45b84c6c01634"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:38.481400 447 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae530c9ed4814fb999a45b84c6c01634 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:38.483978 822 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
I20251025 07:52:38.492161 448 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 46) took 1374 ms. Trace:
I20251025 07:52:38.492372 448 rpcz_store.cc:276] 1025 07:52:37.117473 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.439394 (+1321921us) service_pool.cc:225] Handling call
1025 07:52:38.441866 (+ 2472us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:38.451615 (+ 9749us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:38.491897 (+ 40282us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:38.491998 (+ 101us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:38.492088 (+ 90us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:38.486474 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:38.486595 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:38.486602 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:38.486605 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:38.486706 (+ 101us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:38.486722 (+ 16us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:38.486727 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:38.486810 (+ 83us) tablet.cc:699] Row locks acquired
1025 07:52:38.486813 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:38.486848 (+ 35us) write_op.cc:270] Start()
1025 07:52:38.486895 (+ 47us) write_op.cc:276] Timestamp: P: 1761378758486832 usec, L: 0
1025 07:52:38.486899 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:38.487086 (+ 187us) log.cc:844] Serialized 433 byte log entry
1025 07:52:38.491184 (+ 4098us) op_driver.cc:464] REPLICATION: finished
1025 07:52:38.491294 (+ 110us) write_op.cc:301] APPLY: starting
1025 07:52:38.491310 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:38.491354 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:38.491379 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:38.491489 (+ 110us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:38.491524 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:38.491533 (+ 9us) write_op.cc:312] APPLY: finished
1025 07:52:38.491604 (+ 71us) log.cc:844] Serialized 29 byte log entry
1025 07:52:38.491662 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.491686 (+ 24us) write_op.cc:454] Released schema lock
1025 07:52:38.491695 (+ 9us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:38.491763 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.491771 (+ 8us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":58,"apply.run_cpu_time_us":462,"apply.run_wall_time_us":462,"num_ops":2,"prepare.queue_time_us":53,"prepare.run_cpu_time_us":709,"prepare.run_wall_time_us":709,"replication_time_us":4252}]]}
I20251025 07:52:38.493746 448 master_service.cc:954] ReplaceTablet: received request to replace tablet 062c0f3e51a3487880cea373f8cf1f54 from {username='slave'} at 127.0.0.1:47610
I20251025 07:52:38.494976 1167 tablet_bootstrap.cc:492] T 7c7a9d7a4fb3449bb57eafa83230a839 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:38.495114 1167 ts_tablet_manager.cc:1403] T 7c7a9d7a4fb3449bb57eafa83230a839 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.053s user 0.006s sys 0.002s
I20251025 07:52:38.495877 1167 raft_consensus.cc:359] T 7c7a9d7a4fb3449bb57eafa83230a839 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.496047 1167 raft_consensus.cc:385] T 7c7a9d7a4fb3449bb57eafa83230a839 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:38.496115 1167 raft_consensus.cc:740] T 7c7a9d7a4fb3449bb57eafa83230a839 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:38.496351 1167 consensus_queue.cc:260] T 7c7a9d7a4fb3449bb57eafa83230a839 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.497001 1167 ts_tablet_manager.cc:1434] T 7c7a9d7a4fb3449bb57eafa83230a839 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:38.497273 1167 tablet_bootstrap.cc:492] T b94634ac5077439fadfbbce569f60e8b P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:38.497906 2146 tablet_replica.cc:333] T d37758f606344127b3db74c87babe69c P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:38.497638 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f]). Expected version 1 got 0
I20251025 07:52:38.498947 1167 tablet_bootstrap.cc:654] T b94634ac5077439fadfbbce569f60e8b P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:38.502101 1169 tablet_bootstrap.cc:492] T df1248f6dbd94af68c587da7d149e279 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:38.503887 748 tablet_service.cc:1552] Processing DeleteTablet for tablet aaef24ccdfba45fe8204327723df834e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:58412
W20251025 07:52:38.504690 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet aaef24ccdfba45fe8204327723df834e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aaef24ccdfba45fe8204327723df834e already in progress: creating tablet
I20251025 07:52:38.509773 1169 tablet_bootstrap.cc:654] T df1248f6dbd94af68c587da7d149e279 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:38.511758 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b940fa5ae26427dafbcfd26372754b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.512943 740 tablet_service.cc:1552] Processing DeleteTablet for tablet 04449cc97f404a19b6a1b20dea074f95 with delete_type TABLET_DATA_DELETED (replaced by tablet f70091024c034c118fb86578db19a31c) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.513023 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 26c849d376d143c6bef0c8e0e1d7d03a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.513151 744 tablet_service.cc:1552] Processing DeleteTablet for tablet f43ef51ee45d47a99ec732b49d50d3b4 with delete_type TABLET_DATA_DELETED (replaced by tablet e429b94326cc4827a03968fd92de38e0) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.513314 738 tablet_service.cc:1552] Processing DeleteTablet for tablet d37758f606344127b3db74c87babe69c with delete_type TABLET_DATA_DELETED (replaced by tablet 07898860aa594a05a211fc89768cb76d) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.513489 750 tablet_service.cc:1552] Processing DeleteTablet for tablet f97acc9a638b4d5a968dcc3412a384bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.513657 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 410ef150a7014c6c867a021b544a2f4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.513818 737 tablet_service.cc:1552] Processing DeleteTablet for tablet 88eb51447e0f4080baa49dd902deb24f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.513967 739 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f26667f053a4f16b5da330c0b5e40f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 34315f64b89148efbf1f20a43d5c5efb) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.514000 2166 tablet_replica.cc:333] T d37758f606344127b3db74c87babe69c P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:38.513314 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 050a5d3b5f934dd19bf2e221f2fe0780 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.514319 749 tablet_service.cc:1552] Processing DeleteTablet for tablet b947ecc352e54f4caa73c6ca97372256 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.514207 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 48ff4ce7846b4e56a37eee9bcb17bd5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.514478 751 tablet_service.cc:1552] Processing DeleteTablet for tablet d9bfd7c0c3ad4e5b98b6708793d61d41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.514634 753 tablet_service.cc:1552] Processing DeleteTablet for tablet a4e4b99be0434baaa75d988281797846 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.514817 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 80ee75935d7e461d9d4066fac9819bef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.514207 2166 raft_consensus.cc:2243] T d37758f606344127b3db74c87babe69c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:38.515007 755 tablet_service.cc:1552] Processing DeleteTablet for tablet b94634ac5077439fadfbbce569f60e8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.515069 2166 raft_consensus.cc:2272] T d37758f606344127b3db74c87babe69c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:38.515193 741 tablet_service.cc:1552] Processing DeleteTablet for tablet 92aa605fabbc4257a6a204c087e7cc7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.515386 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 50ae9bca6bed4f749d54191ac1543410 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.515555 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 3abfd52327a04ba88625f391a8658fbb with delete_type TABLET_DATA_DELETED (replaced by tablet 04d27d54fdb7483b869408af65ff85c9) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.515679 754 tablet_service.cc:1552] Processing DeleteTablet for tablet b3a0af893a8947848e12093d0a7f0229 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.515863 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 34315f64b89148efbf1f20a43d5c5efb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.515688 2166 ts_tablet_manager.cc:1916] T d37758f606344127b3db74c87babe69c P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:38.516376 745 tablet_service.cc:1552] Processing DeleteTablet for tablet af743ee344e543a0aa6da3cd7a30e467 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.516520 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 08da9d2687704725aacd11cd92c6a9f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.516800 754 tablet_service.cc:1552] Processing DeleteTablet for tablet d66ad049d5484228a268b0facb00ecf1 with delete_type TABLET_DATA_DELETED (replaced by tablet 2e098c9c4613459295b6043048cba587) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.517057 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b28215f61574cf48c447dd01955ac77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.517310 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 7fbae8be73bf4e18b5a5c4ec4cdebc12 with delete_type TABLET_DATA_DELETED (replaced by tablet 4c155aa7fa5b4d68bbab1918b8432372) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.517580 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 551d07a9242740dcb9366bb2d707a4a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.517930 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ec421eb820b4f40afcff5a5bf80c508 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.518095 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 467324e2432f4caca148951329caa292 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.518419 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d261e432cf5403ea162129706bc4d4d with delete_type TABLET_DATA_DELETED (replaced by tablet e54fe7cbcffa40c4b04144b47bee2b48) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.518695 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 716fb61312e94a4d9eb8ab8336ca140c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.519630 1167 tablet_bootstrap.cc:492] T b94634ac5077439fadfbbce569f60e8b P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:38.498271 886 tablet_service.cc:1552] Processing DeleteTablet for tablet 5fca357d7e24494fb6af4bd73d98833c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.519758 1167 ts_tablet_manager.cc:1403] T b94634ac5077439fadfbbce569f60e8b P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.004s
I20251025 07:52:38.498768 876 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b594d4718344aafbdaec34040695589 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.520020 2146 raft_consensus.cc:2243] T d37758f606344127b3db74c87babe69c P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:38.520236 2146 raft_consensus.cc:2272] T d37758f606344127b3db74c87babe69c P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:38.520426 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 3b940fa5ae26427dafbcfd26372754b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b940fa5ae26427dafbcfd26372754b3 already in progress: creating tablet
I20251025 07:52:38.520543 1167 raft_consensus.cc:359] T b94634ac5077439fadfbbce569f60e8b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
W20251025 07:52:38.520681 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 04449cc97f404a19b6a1b20dea074f95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04449cc97f404a19b6a1b20dea074f95 already in progress: creating tablet
I20251025 07:52:38.520718 1167 raft_consensus.cc:385] T b94634ac5077439fadfbbce569f60e8b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:38.520776 1167 raft_consensus.cc:740] T b94634ac5077439fadfbbce569f60e8b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
W20251025 07:52:38.520846 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 26c849d376d143c6bef0c8e0e1d7d03a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 26c849d376d143c6bef0c8e0e1d7d03a already in progress: creating tablet
W20251025 07:52:38.521029 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet f43ef51ee45d47a99ec732b49d50d3b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f43ef51ee45d47a99ec732b49d50d3b4 already in progress: creating tablet
I20251025 07:52:38.520996 1167 consensus_queue.cc:260] T b94634ac5077439fadfbbce569f60e8b P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.521597 1167 ts_tablet_manager.cc:1434] T b94634ac5077439fadfbbce569f60e8b P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:38.521908 1167 tablet_bootstrap.cc:492] T aa9d9ab8073945d28d52fb02213f201a P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:38.503407 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7]). Expected version 1 got 0
I20251025 07:52:38.499400 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 570201e7bf05406a97c81dde071d894b with delete_type TABLET_DATA_DELETED (replaced by tablet 3a0f50c8000144a4acb5219484b01800) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.499985 889 tablet_service.cc:1552] Processing DeleteTablet for tablet a13bf4d7eb1440f7a28b85cb1bf99667 with delete_type TABLET_DATA_DELETED (replaced by tablet 74e2d78c024849b0af5c0bde1f48c36f) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.500527 892 tablet_service.cc:1552] Processing DeleteTablet for tablet f3001314b56345f5a70314777cae0125 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.523272 2146 ts_tablet_manager.cc:1916] T d37758f606344127b3db74c87babe69c P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:38.501157 1166 raft_consensus.cc:359] T d66ad049d5484228a268b0facb00ecf1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.501631 881 tablet_service.cc:1552] Processing DeleteTablet for tablet f170424be2f643e7af499834955472b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.514398 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 02e4c4d6ffb34cecaf17beb5ba197e81 with delete_type TABLET_DATA_DELETED (replaced by tablet 3abfd52327a04ba88625f391a8658fbb) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.518934 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c7a9d7a4fb3449bb57eafa83230a839 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:33 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.519095 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 70fe9aa2f58d4abea3b3b71066a29653 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.519142 885 tablet_service.cc:1552] Processing DeleteTablet for tablet c59f6e6eb9594a8e83f109a09b034045 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.519238 878 tablet_service.cc:1552] Processing DeleteTablet for tablet f4578ce69a0d4322944e5ce7c8a59831 with delete_type TABLET_DATA_DELETED (replaced by tablet 48b73e0c0dce4be686d05f991a21bd19) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.519299 880 tablet_service.cc:1552] Processing DeleteTablet for tablet 71432a6848944a0da06ef9b81f160dc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.524767 1166 raft_consensus.cc:385] T d66ad049d5484228a268b0facb00ecf1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:38.524838 1166 raft_consensus.cc:740] T d66ad049d5484228a268b0facb00ecf1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:38.499083 893 tablet_service.cc:1552] Processing DeleteTablet for tablet c4610b0c62e741eca28c21f63cc3de81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.525010 1166 consensus_queue.cc:260] T d66ad049d5484228a268b0facb00ecf1 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.525244 1167 tablet_bootstrap.cc:654] T aa9d9ab8073945d28d52fb02213f201a P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:38.525667 1166 ts_tablet_manager.cc:1434] T d66ad049d5484228a268b0facb00ecf1 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.158s user 0.002s sys 0.000s
I20251025 07:52:38.525928 1166 tablet_bootstrap.cc:492] T 3abfd52327a04ba88625f391a8658fbb P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:38.527956 879 tablet_service.cc:1552] Processing DeleteTablet for tablet c88c46e022ff41a58b02538f2151e260 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.528352 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d6a9937d10a4cc2813671dd1bfa1664 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.528546 2166 ts_tablet_manager.cc:1929] T d37758f606344127b3db74c87babe69c P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:38.528671 879 tablet_service.cc:1552] Processing DeleteTablet for tablet d5e5e5bbcf9f43118eea502b69880af5 with delete_type TABLET_DATA_DELETED (replaced by tablet f4578ce69a0d4322944e5ce7c8a59831) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.529348 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 5361f08f50b745eb98bf02e979d7c71e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.529881 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 53198283718d4b45b30df65b5bde96a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.530040 2166 log.cc:1199] T d37758f606344127b3db74c87babe69c P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/d37758f606344127b3db74c87babe69c
I20251025 07:52:38.530592 2166 ts_tablet_manager.cc:1950] T d37758f606344127b3db74c87babe69c P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:38.530798 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ec421eb820b4f40afcff5a5bf80c508 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.531215 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet d37758f606344127b3db74c87babe69c (table default.Table_92cbcd8230f249a8b2821f69a81b41f1 [id=08e53121e86d405dbbd02a4605695fec]) successfully deleted
W20251025 07:52:38.532024 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet f97acc9a638b4d5a968dcc3412a384bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f97acc9a638b4d5a968dcc3412a384bc already in progress: creating tablet
I20251025 07:52:38.532598 2166 tablet_replica.cc:333] T 410ef150a7014c6c867a021b544a2f4e P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:38.532840 2166 raft_consensus.cc:2243] T 410ef150a7014c6c867a021b544a2f4e P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:38.533104 2166 pending_rounds.cc:70] T 410ef150a7014c6c867a021b544a2f4e P 23eeffd3de254ce18db5febfde971c6b: Trying to abort 1 pending ops.
W20251025 07:52:38.477340 453 rpcz_store.cc:269] 1025 07:52:36.271697 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:36.274751 (+ 3054us) service_pool.cc:225] Handling call
1025 07:52:37.052970 (+778219us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:37.052976 (+ 6us) write_op.cc:454] Released schema lock
1025 07:52:38.066250 (+1013274us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.066257 (+ 7us) write_op.cc:454] Released schema lock
1025 07:52:38.477075 (+410818us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:37.043795 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:37.043936 (+ 141us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:37.043946 (+ 10us) write_op.cc:435] Acquired schema lock
1025 07:52:37.043949 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:37.044174 (+ 225us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:37.044191 (+ 17us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:37.044196 (+ 5us) tablet.cc:683] Acquiring locks for 3 operations
1025 07:52:37.044312 (+ 116us) tablet.cc:699] Row locks acquired
1025 07:52:37.044315 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:37.044356 (+ 41us) write_op.cc:270] Start()
1025 07:52:37.044393 (+ 37us) write_op.cc:276] Timestamp: P: 1761378757044322 usec, L: 0
1025 07:52:37.044397 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:37.044639 (+ 242us) log.cc:844] Serialized 1275 byte log entry
1025 07:52:37.048469 (+ 3830us) op_driver.cc:464] REPLICATION: finished
1025 07:52:37.048616 (+ 147us) write_op.cc:301] APPLY: starting
1025 07:52:37.048631 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:37.048702 (+ 71us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:37.048733 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:37.048816 (+ 83us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:37.048851 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
1025 07:52:37.048858 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:37.048937 (+ 79us) log.cc:844] Serialized 35 byte log entry
1025 07:52:37.048996 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:37.049019 (+ 23us) write_op.cc:454] Released schema lock
1025 07:52:37.049026 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
1025 07:52:38.059347 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:38.059514 (+ 167us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:38.059523 (+ 9us) write_op.cc:435] Acquired schema lock
1025 07:52:38.059527 (+ 4us) tablet.cc:660] Decoding operations
1025 07:52:38.059652 (+ 125us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:38.059672 (+ 20us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:38.059679 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:38.059759 (+ 80us) tablet.cc:699] Row locks acquired
1025 07:52:38.059763 (+ 4us) write_op.cc:260] PREPARE: finished
1025 07:52:38.059808 (+ 45us) write_op.cc:270] Start()
1025 07:52:38.059847 (+ 39us) write_op.cc:276] Timestamp: P: 1761378758059786 usec, L: 0
1025 07:52:38.059852 (+ 5us) op_driver.cc:348] REPLICATION: starting
1025 07:52:38.060072 (+ 220us) log.cc:844] Serialized 356 byte log entry
1025 07:52:38.064246 (+ 4174us) op_driver.cc:464] REPLICATION: finished
1025 07:52:38.064456 (+ 210us) write_op.cc:301] APPLY: starting
1025 07:52:38.064475 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:38.064522 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:38.064560 (+ 38us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:38.064615 (+ 55us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:38.064648 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:38.064656 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:38.064733 (+ 77us) log.cc:844] Serialized 23 byte log entry
1025 07:52:38.064797 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.064819 (+ 22us) write_op.cc:454] Released schema lock
1025 07:52:38.064827 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":46628,"reported_tablets":244,"child_traces":[["op",{"apply.queue_time_us":88,"apply.run_cpu_time_us":468,"apply.run_wall_time_us":463,"num_ops":3,"prepare.queue_time_us":1556,"prepare.run_cpu_time_us":963,"prepare.run_wall_time_us":963,"raft.queue_time_us":1251,"raft.run_cpu_time_us":325,"raft.run_wall_time_us":316,"replication_time_us":4045}],["op",{"apply.queue_time_us":150,"apply.run_cpu_time_us":432,"apply.run_wall_time_us":431,"num_ops":1,"prepare.queue_time_us":94,"prepare.run_cpu_time_us":884,"prepare.run_wall_time_us":884,"raft.queue_time_us":1381,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":270,"replication_time_us":4362}]]}
I20251025 07:52:38.533172 2166 pending_rounds.cc:77] T 410ef150a7014c6c867a021b544a2f4e P 23eeffd3de254ce18db5febfde971c6b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214607393286721536 op_type: NO_OP noop_request { }
I20251025 07:52:38.533844 2166 raft_consensus.cc:2889] T 410ef150a7014c6c867a021b544a2f4e P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 07:52:38.533994 2166 raft_consensus.cc:2272] T 410ef150a7014c6c867a021b544a2f4e P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:38.535272 2205 tablet.cc:1722] T cd23b8094d7e472eba6bd170248858a9 P db42be7ad3c741a0aa427f27e245993b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251025 07:52:38.536586 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 551d07a9242740dcb9366bb2d707a4a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.536983 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 7fbae8be73bf4e18b5a5c4ec4cdebc12 with delete_type TABLET_DATA_DELETED (replaced by tablet 4c155aa7fa5b4d68bbab1918b8432372) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.536983 745 tablet_service.cc:1552] Processing DeleteTablet for tablet bef0ef23d90d480d9c99aef21d7271fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.537303 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 50ae9bca6bed4f749d54191ac1543410 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.533521 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet cb829649f73d4d3f8862d268c953bd86 (table default.Table_ba4b8cc7b930482690b7e192e0edff12 [id=7944225d3ce7439b881bfb9289d9f040]). Expected version 1 got 0
I20251025 07:52:38.538748 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 04d27d54fdb7483b869408af65ff85c9 (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]). Expected version 2 got 0
W20251025 07:52:38.538833 850 leader_election.cc:343] T 8f2c8582fed34c848d9903b8ad8b4057 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:38.539080 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 4c155aa7fa5b4d68bbab1918b8432372 (table default.Table_6c5a24747ba14352b0285a41ec2985a4 [id=28bbaad474164a8c8c97a76bb8081791]). Expected version 2 got 0
W20251025 07:52:38.540019 850 leader_election.cc:343] T 8f2c8582fed34c848d9903b8ad8b4057 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:38.540169 960 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
I20251025 07:52:38.540866 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet aa9d9ab8073945d28d52fb02213f201a (table default.Table_4791672658f043f6b97b9320a531ebb5 [id=dfd20cf9c3e541c4abe23d59e0aa7f2a]). Expected version 1 got 0
I20251025 07:52:38.541555 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 062c0f3e51a3487880cea373f8cf1f54 (table default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de]). Expected version 1 got 0
I20251025 07:52:38.541976 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 292fe04055954afd8c08ac2636b68191 (table default.Table_9fa69b10b5e842639fc0b0a58cd9b6a5 [id=a253ce0253bd4da4814741c482193d8f]). Expected version 1 got 0
I20251025 07:52:38.542727 452 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 47) took 1414 ms. Trace:
I20251025 07:52:38.542946 452 rpcz_store.cc:276] 1025 07:52:37.128603 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.472039 (+1343436us) service_pool.cc:225] Handling call
1025 07:52:38.473490 (+ 1451us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:38.500822 (+ 27332us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:38.532404 (+ 31582us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.532411 (+ 7us) write_op.cc:454] Released schema lock
1025 07:52:38.533063 (+ 652us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:38.542479 (+ 9416us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:38.542656 (+ 177us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:38.523550 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:38.523682 (+ 132us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:38.523688 (+ 6us) write_op.cc:435] Acquired schema lock
1025 07:52:38.523690 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:38.523783 (+ 93us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:38.523814 (+ 31us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:38.523819 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:38.523895 (+ 76us) tablet.cc:699] Row locks acquired
1025 07:52:38.523899 (+ 4us) write_op.cc:260] PREPARE: finished
1025 07:52:38.523941 (+ 42us) write_op.cc:270] Start()
1025 07:52:38.523979 (+ 38us) write_op.cc:276] Timestamp: P: 1761378758523923 usec, L: 0
1025 07:52:38.523984 (+ 5us) op_driver.cc:348] REPLICATION: starting
1025 07:52:38.524208 (+ 224us) log.cc:844] Serialized 433 byte log entry
1025 07:52:38.529647 (+ 5439us) op_driver.cc:464] REPLICATION: finished
1025 07:52:38.529739 (+ 92us) write_op.cc:301] APPLY: starting
1025 07:52:38.529754 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:38.529801 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:38.529826 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:38.529943 (+ 117us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:38.529975 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:38.529987 (+ 12us) write_op.cc:312] APPLY: finished
1025 07:52:38.530070 (+ 83us) log.cc:844] Serialized 29 byte log entry
1025 07:52:38.530168 (+ 98us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.530190 (+ 22us) write_op.cc:454] Released schema lock
1025 07:52:38.530198 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":50,"apply.run_cpu_time_us":510,"apply.run_wall_time_us":509,"num_ops":2,"prepare.queue_time_us":321,"prepare.run_cpu_time_us":758,"prepare.run_wall_time_us":759,"raft.queue_time_us":225,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":265,"replication_time_us":5634,"spinlock_wait_cycles":118016}]]}
I20251025 07:52:38.544384 452 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_bcd5afa1072f47898d67504fc72b84c4" } modify_external_catalogs: true
I20251025 07:52:38.544742 452 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_bcd5afa1072f47898d67504fc72b84c4" } modify_external_catalogs: true
I20251025 07:52:38.545994 447 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 48) took 1405 ms. Trace:
I20251025 07:52:38.546823 447 rpcz_store.cc:276] 1025 07:52:37.140386 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.479932 (+1339546us) service_pool.cc:225] Handling call
1025 07:52:38.481538 (+ 1606us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:38.498179 (+ 16641us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:38.509139 (+ 10960us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.509144 (+ 5us) write_op.cc:454] Released schema lock
1025 07:52:38.523053 (+ 13909us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:38.532946 (+ 9893us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:38.545956 (+ 13010us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:38.503835 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:38.503947 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:38.503954 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:38.503958 (+ 4us) tablet.cc:660] Decoding operations
1025 07:52:38.504078 (+ 120us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:38.504093 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:38.504098 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:38.504191 (+ 93us) tablet.cc:699] Row locks acquired
1025 07:52:38.504194 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:38.504229 (+ 35us) write_op.cc:270] Start()
1025 07:52:38.504265 (+ 36us) write_op.cc:276] Timestamp: P: 1761378758504214 usec, L: 0
1025 07:52:38.504269 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:38.504442 (+ 173us) log.cc:844] Serialized 433 byte log entry
1025 07:52:38.507109 (+ 2667us) op_driver.cc:464] REPLICATION: finished
1025 07:52:38.507222 (+ 113us) write_op.cc:301] APPLY: starting
1025 07:52:38.507236 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:38.507279 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:38.507309 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:38.507411 (+ 102us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:38.507444 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:38.507452 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:38.507523 (+ 71us) log.cc:844] Serialized 29 byte log entry
1025 07:52:38.507592 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.507613 (+ 21us) write_op.cc:454] Released schema lock
1025 07:52:38.507626 (+ 13us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":56,"apply.run_cpu_time_us":456,"apply.run_wall_time_us":455,"num_ops":2,"prepare.queue_time_us":454,"prepare.run_cpu_time_us":708,"prepare.run_wall_time_us":707,"raft.queue_time_us":57,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":208,"replication_time_us":2808}]]}
I20251025 07:52:38.533977 453 master_service.cc:954] ReplaceTablet: received request to replace tablet 36c8fcafd5464b4e999276a8062a812a from {username='slave'} at 127.0.0.1:47610
I20251025 07:52:38.550982 447 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_92c0120dd29348618040150dedb14c43" } modify_external_catalogs: true
I20251025 07:52:38.551376 447 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_92c0120dd29348618040150dedb14c43" } modify_external_catalogs: true
I20251025 07:52:38.551605 2166 ts_tablet_manager.cc:1916] T 410ef150a7014c6c867a021b544a2f4e P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:38.542395 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525]). Expected version 1 got 0
I20251025 07:52:38.554508 745 tablet_service.cc:1552] Processing DeleteTablet for tablet f661d5909657455581dcb5e6eadd3c0b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.554934 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f0c4f0e8f014e59be9e46d2afe0be9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.555347 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 2784bbfb97d5466f9ec856add7d0b29a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.555807 1659 raft_consensus.cc:493] T 80ee75935d7e461d9d4066fac9819bef P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:38.555940 1659 raft_consensus.cc:515] T 80ee75935d7e461d9d4066fac9819bef P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.556268 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 22f7db660616457b83b24e947a0b0bae (table default.Table_4a86191163f2408f9bcc795cd8d1bbf6 [id=bb316a7e000748ca902a9bfb494802d7]). Expected version 1 got 0
I20251025 07:52:38.556597 1659 leader_election.cc:290] T 80ee75935d7e461d9d4066fac9819bef P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:38.557139 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84]). Expected version 1 got 0
I20251025 07:52:38.557226 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "80ee75935d7e461d9d4066fac9819bef" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:38.557751 710 leader_election.cc:343] T 80ee75935d7e461d9d4066fac9819bef P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:38.558585 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "80ee75935d7e461d9d4066fac9819bef" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:38.558934 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 5dab439bc50d4e869548154e1d222ad6 (table default.Table_bd82d871f84943f4baa967f5c0ec74a3 [id=45325c188eac4d34bb016d01de091f87]). Expected version 2 got 0
I20251025 07:52:38.559720 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 87268e27540c4ec58ce1e62df819f19d (table default.Table_350383f0e98947eba77186e74ad88781 [id=96b0e47c666c4d8f982741aea7b17ae7]). Expected version 2 got 0
I20251025 07:52:38.560830 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_87d38832f85e4965bfab1a077990114f [id=ad812862698d421184ebf501ce79b7be]). Expected version 2 got 0
I20251025 07:52:38.562747 745 tablet_service.cc:1552] Processing DeleteTablet for tablet fd265cdbf6a8440393a2b5d8240d1872 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
W20251025 07:52:38.563068 712 leader_election.cc:343] T 80ee75935d7e461d9d4066fac9819bef P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:38.563236 712 leader_election.cc:304] T 80ee75935d7e461d9d4066fac9819bef P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:38.563515 1659 raft_consensus.cc:2749] T 80ee75935d7e461d9d4066fac9819bef P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:38.564576 2206 tablet.cc:1722] T cd23b8094d7e472eba6bd170248858a9 P 23eeffd3de254ce18db5febfde971c6b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251025 07:52:38.564500 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c]). Expected version 1 got 0
I20251025 07:52:38.566620 2183 raft_consensus.cc:493] T d66ad049d5484228a268b0facb00ecf1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:38.566332 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 0325d462441349ef81841f1b0cab2a20 (table default.Table_750ecb905839434fa9a1aa72f87394be [id=9ce3ba77c56a4ef09cb3b40ea0e488f6]). Expected version 1 got 0
I20251025 07:52:38.566681 745 tablet_service.cc:1552] Processing DeleteTablet for tablet d77704b4fe6c4456a2ba40ffec0c60fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.566771 2183 raft_consensus.cc:515] T d66ad049d5484228a268b0facb00ecf1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.567381 2183 leader_election.cc:290] T d66ad049d5484228a268b0facb00ecf1 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:38.567382 745 tablet_service.cc:1552] Processing DeleteTablet for tablet e4da89f863cc43f3bf6df908bb9dc6bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.568900 745 tablet_service.cc:1552] Processing DeleteTablet for tablet e47c4f741f354a19b49777bd06e54764 with delete_type TABLET_DATA_DELETED (replaced by tablet 755f0d6c87044e20a8020f8410940123) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.569649 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 72dc084f8efc45478771c247e07a16f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.570520 745 tablet_service.cc:1552] Processing DeleteTablet for tablet be5ab0db6ac34c2c9a370445e251eb46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.570372 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 6897405bbc1c47a4ab0aa3af80e60e3d (table default.Table_92c0120dd29348618040150dedb14c43 [id=ea892976886644d8aa025ae2edef718d]). Expected version 2 got 0
W20251025 07:52:38.571681 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 7e6544202d4f489d82f22d6073ef240d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e6544202d4f489d82f22d6073ef240d already in progress: creating tablet
I20251025 07:52:38.572317 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 7f29b64704e147aa9b0b5f6cda860f09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:58412
W20251025 07:52:38.572328 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet dda0b26e4f1a44928bc4b84f333901c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dda0b26e4f1a44928bc4b84f333901c0 already in progress: creating tablet
W20251025 07:52:38.572670 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet ffe7c8e1d6c44872b7b65e512764a56a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ffe7c8e1d6c44872b7b65e512764a56a already in progress: creating tablet
I20251025 07:52:38.573932 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet db99a1e68daf4e948a3d66ad276cc5f9 (table default.Table_e1d698cd782e4d58b29e9cc7d1f56703 [id=e8dcb9f9f6554f5ebcdfee63c6259f03]) successfully deleted
I20251025 07:52:38.571569 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 830fdbc6812e416690d0db42a73af503 (table default.Table_92824b3f2a774564b3aea04aaa25d204 [id=fb21a45b86ab446e8c3dae4512a9598f]). Expected version 1 got 0
I20251025 07:52:38.574790 1659 raft_consensus.cc:493] T b947ecc352e54f4caa73c6ca97372256 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:38.574851 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 8f2c8582fed34c848d9903b8ad8b4057 (table default.Table_438f67f3dc62465999a917986922b3ed [id=7c7fcd265e624d7fab74661db505d9c3]). Expected version 1 got 0
I20251025 07:52:38.574957 1659 raft_consensus.cc:515] T b947ecc352e54f4caa73c6ca97372256 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.575139 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_e90d7f5ff13b40878e5b625c97792c7d [id=fc8180c09d7b4e74b2e161f7acb8fe0e]). Expected version 1 got 0
I20251025 07:52:38.575533 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]). Expected version 1 got 0
I20251025 07:52:38.575616 1659 leader_election.cc:290] T b947ecc352e54f4caa73c6ca97372256 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:38.575834 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]). Expected version 3 got 0
I20251025 07:52:38.574038 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d66ad049d5484228a268b0facb00ecf1" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:38.576247 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236]). Expected version 1 got 0
I20251025 07:52:38.576385 774 raft_consensus.cc:2468] T d66ad049d5484228a268b0facb00ecf1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:38.576339 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b947ecc352e54f4caa73c6ca97372256" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:38.576756 982 leader_election.cc:304] T d66ad049d5484228a268b0facb00ecf1 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
W20251025 07:52:38.576960 712 leader_election.cc:343] T b947ecc352e54f4caa73c6ca97372256 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:38.577008 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 36262ab8c6964dd684c01b4db6219a4f (table default.Table_76158f8cf82e4cdb9a90580e46aaea48 [id=f249f30497bc450c887d659721469de8]). Expected version 1 got 0
I20251025 07:52:38.577157 2183 raft_consensus.cc:2804] T d66ad049d5484228a268b0facb00ecf1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:38.577230 2183 raft_consensus.cc:493] T d66ad049d5484228a268b0facb00ecf1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:38.577296 2183 raft_consensus.cc:3060] T d66ad049d5484228a268b0facb00ecf1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:38.577410 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 2c50ec95e48e49b59a76fac4fc862d3d (table default.Table_32b54e82cbc84d39a0020826f8737d62 [id=c28b0b6acd61490caecfdaab803a9d92]). Expected version 2 got 0
I20251025 07:52:38.580508 2183 raft_consensus.cc:515] T d66ad049d5484228a268b0facb00ecf1 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.580925 1018 tablet_service.cc:1505] Processing CreateTablet for tablet 8e049e460f50482aa920e0c76740fd79 (DEFAULT_TABLE table=default.Table_2dd1c1390f234ba8ac968b6842dd28be [id=1a843288ea7c49bfa3f7758fc1fcf9a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:38.580963 1169 tablet_bootstrap.cc:492] T df1248f6dbd94af68c587da7d149e279 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:38.582297 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d66ad049d5484228a268b0facb00ecf1" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:38.582618 774 raft_consensus.cc:3060] T d66ad049d5484228a268b0facb00ecf1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:38.584563 1169 ts_tablet_manager.cc:1403] T df1248f6dbd94af68c587da7d149e279 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.083s user 0.002s sys 0.002s
I20251025 07:52:38.581537 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e049e460f50482aa920e0c76740fd79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:38.585484 1169 raft_consensus.cc:359] T df1248f6dbd94af68c587da7d149e279 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.588523 774 raft_consensus.cc:2468] T d66ad049d5484228a268b0facb00ecf1 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:38.588753 1169 raft_consensus.cc:385] T df1248f6dbd94af68c587da7d149e279 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:38.588908 1169 raft_consensus.cc:740] T df1248f6dbd94af68c587da7d149e279 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:38.586580 2183 leader_election.cc:290] T d66ad049d5484228a268b0facb00ecf1 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:38.589466 1169 consensus_queue.cc:260] T df1248f6dbd94af68c587da7d149e279 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.595417 1169 ts_tablet_manager.cc:1434] T df1248f6dbd94af68c587da7d149e279 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20251025 07:52:38.596041 1169 tablet_bootstrap.cc:492] T 410ef150a7014c6c867a021b544a2f4e P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:38.602849 1169 tablet_bootstrap.cc:654] T 410ef150a7014c6c867a021b544a2f4e P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:38.596889 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 95efd553493e4966bc48a01cac6e1e10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.598670 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 896bd7870d87408e8adfdb7a78998111 with delete_type TABLET_DATA_DELETED (replaced by tablet e8402fd33ba842e981c6c68787c7c548) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.599283 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f25c9caa05b4da4bd56a8a21fde423c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.603721 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 6dcab53a020847a1aa27930265ba0205 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.604315 1167 tablet_bootstrap.cc:492] T aa9d9ab8073945d28d52fb02213f201a P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:38.604799 1167 ts_tablet_manager.cc:1403] T aa9d9ab8073945d28d52fb02213f201a P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.083s user 0.006s sys 0.002s
I20251025 07:52:38.605696 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 1b5ac7e845a24b2eb5b2429d47361232 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.606506 1167 raft_consensus.cc:359] T aa9d9ab8073945d28d52fb02213f201a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.606899 1167 raft_consensus.cc:385] T aa9d9ab8073945d28d52fb02213f201a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:38.607131 1167 raft_consensus.cc:740] T aa9d9ab8073945d28d52fb02213f201a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:38.607583 1167 consensus_queue.cc:260] T aa9d9ab8073945d28d52fb02213f201a P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.609686 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 21e52928580c4762b0ae7152490ef955 with delete_type TABLET_DATA_DELETED (replaced by tablet 9d6a9937d10a4cc2813671dd1bfa1664) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.610709 1167 ts_tablet_manager.cc:1434] T aa9d9ab8073945d28d52fb02213f201a P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251025 07:52:38.611416 2212 tablet.cc:1722] T 87268e27540c4ec58ce1e62df819f19d P 23eeffd3de254ce18db5febfde971c6b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251025 07:52:38.611588 754 tablet_service.cc:1552] Processing DeleteTablet for tablet a7ab3150a35b4caca9119a1374d3f2f7 with delete_type TABLET_DATA_DELETED (replaced by tablet c19c97f881cb45218d4a6f6f231aa96e) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.612926 754 tablet_service.cc:1552] Processing DeleteTablet for tablet cb824bd780524832b2b60f5618a39da5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.613394 754 tablet_service.cc:1552] Processing DeleteTablet for tablet cc407c7eb12e4eadb41fb21385611c94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.613705 1167 tablet_bootstrap.cc:492] T b3a0af893a8947848e12093d0a7f0229 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:38.615440 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e098c9c4613459295b6043048cba587 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.616909 754 tablet_service.cc:1552] Processing DeleteTablet for tablet abf94c32ff2945119b710905c075d63c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.604982 982 leader_election.cc:304] T d66ad049d5484228a268b0facb00ecf1 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:38.617421 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 1912e59548874e0e8e989050ec0064c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.617624 2183 raft_consensus.cc:2804] T d66ad049d5484228a268b0facb00ecf1 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:38.617848 2183 raft_consensus.cc:697] T d66ad049d5484228a268b0facb00ecf1 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:38.618115 2183 consensus_queue.cc:237] T d66ad049d5484228a268b0facb00ecf1 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.620510 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 887e7c7a5d0e4aaf88f26246306df19c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.620559 1011 tablet_service.cc:1505] Processing CreateTablet for tablet ff3daed6c9af47e3976b82b4b9a41086 (DEFAULT_TABLE table=default.Table_6caa55a17907446d81c7f0656b329cdd [id=5b6faf06c72745f68f7c478e50a304d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:38.620890 754 tablet_service.cc:1552] Processing DeleteTablet for tablet ee1c8ea694284d80a30f9c05be4cbf49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
W20251025 07:52:38.621268 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 80fe3255ed614270bd8352a2853fbc43 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80fe3255ed614270bd8352a2853fbc43 already in progress: creating tablet
I20251025 07:52:38.621656 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 570201e7bf05406a97c81dde071d894b with delete_type TABLET_DATA_DELETED (replaced by tablet 3a0f50c8000144a4acb5219484b01800) from {username='slave'} at 127.0.0.1:58412
W20251025 07:52:38.621728 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet ce1ebb1ec14b4801930a72765bca1a7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce1ebb1ec14b4801930a72765bca1a7d already in progress: creating tablet
W20251025 07:52:38.622035 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 5b6383c99e9e4c90a374da9ccd830a6f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b6383c99e9e4c90a374da9ccd830a6f already in progress: creating tablet
W20251025 07:52:38.622485 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 91728987303b41fd840eb56f3a311c5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91728987303b41fd840eb56f3a311c5f already in progress: creating tablet
W20251025 07:52:38.622833 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 050a5d3b5f934dd19bf2e221f2fe0780 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 050a5d3b5f934dd19bf2e221f2fe0780 already in progress: creating tablet
W20251025 07:52:38.623245 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 04449cc97f404a19b6a1b20dea074f95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04449cc97f404a19b6a1b20dea074f95 already in progress: creating tablet
I20251025 07:52:38.623499 1017 tablet_service.cc:1505] Processing CreateTablet for tablet 3c41741ea55842f289bd44762a59c595 (DEFAULT_TABLE table=default.Table_be8af62c5969471b9e7b869c6820db0c [id=5ebbedc74c894c32940ac4492e95909b]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:38.623595 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet ebc824ec4c60411dab1a2ea4a385148e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ebc824ec4c60411dab1a2ea4a385148e already in progress: creating tablet
I20251025 07:52:38.623868 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c41741ea55842f289bd44762a59c595. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:38.624264 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet aaef24ccdfba45fe8204327723df834e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aaef24ccdfba45fe8204327723df834e already in progress: creating tablet
W20251025 07:52:38.624653 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet f97acc9a638b4d5a968dcc3412a384bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f97acc9a638b4d5a968dcc3412a384bc already in progress: creating tablet
I20251025 07:52:38.624699 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff3daed6c9af47e3976b82b4b9a41086. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:38.625727 1029 tablet_service.cc:1505] Processing CreateTablet for tablet a0ac349532534848a9016ad5660e85ae (DEFAULT_TABLE table=default.Table_e72e2534c5c44fd299ed3ec3a367a34c [id=c0903d54e7114c7d83233474e7345a3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:38.626040 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0ac349532534848a9016ad5660e85ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:38.626940 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet 716fb61312e94a4d9eb8ab8336ca140c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
W20251025 07:52:38.629457 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet d9bfd7c0c3ad4e5b98b6708793d61d41 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9bfd7c0c3ad4e5b98b6708793d61d41 already in progress: creating tablet
I20251025 07:52:38.637375 1167 tablet_bootstrap.cc:654] T b3a0af893a8947848e12093d0a7f0229 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:38.642220 2212 tablet.cc:1722] T 6897405bbc1c47a4ab0aa3af80e60e3d P 23eeffd3de254ce18db5febfde971c6b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251025 07:52:38.643177 1018 tablet_service.cc:1505] Processing CreateTablet for tablet 8390711f8ab84170be461b9eeb4c361c (DEFAULT_TABLE table=default.Table_b3ca6007437c4040be247798121808d8 [id=88d28fc51ba94fb4aef4eca8ee82f809]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:38.644124 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 5cba9c05dd424cc788febc614a34819a with delete_type TABLET_DATA_DELETED (replaced by tablet 4b28215f61574cf48c447dd01955ac77) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.645861 754 tablet_service.cc:1552] Processing DeleteTablet for tablet dda0b26e4f1a44928bc4b84f333901c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.646576 754 tablet_service.cc:1552] Processing DeleteTablet for tablet fab67b56eaa44b54886c17181d8c52b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.648918 754 tablet_service.cc:1552] Processing DeleteTablet for tablet c11f674385184ae6b235f1db8bf1fb91 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.578001 449 catalog_manager.cc:5649] T b97e39aaac3048ed93dbfe873b757254 P db42be7ad3c741a0aa427f27e245993b reported cstate change: term changed from 0 to 1, leader changed from <none> to db42be7ad3c741a0aa427f27e245993b (127.0.104.3). New cstate: current_term: 1 leader_uuid: "db42be7ad3c741a0aa427f27e245993b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: UNKNOWN } } }
I20251025 07:52:38.650885 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 27651312cc2e42a8bfcd245f5d301679 with delete_type TABLET_DATA_DELETED (replaced by tablet a4e4b99be0434baaa75d988281797846) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.650913 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet b97e39aaac3048ed93dbfe873b757254 (table default.Table_99fd36369f9a4e0fb14f234c98635203 [id=c8059ab6cec146efa631d06799d969a9]). Expected version 1 got 0
I20251025 07:52:38.651429 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 8e6d76b1a97945cfb6c3055e1b622f8a (table default.Table_10e57b42f97544208675708b1e028846 [id=f5c1be203e2b4831a025871cf8a2ef17]). Expected version 2 got 0
I20251025 07:52:38.651701 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f7c15529be54a11961f385a953b0b03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.651723 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet b17d2a9ad6be49d29f2413666fba7e69 (table default.Table_fcddb31989ee4236a4981c12779b3f5d [id=f3794fa081364666b944c412415bdba0]). Expected version 3 got 0
I20251025 07:52:38.651961 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3]). Expected version 2 got 0
I20251025 07:52:38.652465 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 509c94f2a422493ea4390ecee576db72 (table default.Table_3f081b7bb4e1420c8121ea4115439f3d [id=696a1285fbfe4038a8b1b38708c8ea49]). Expected version 1 got 0
I20251025 07:52:38.652551 754 tablet_service.cc:1552] Processing DeleteTablet for tablet f0bdaa8cfd764f1b83f47881353d26b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.652664 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet ab42909afc524f2fa3c1041495d13a59 (table default.Table_7eb8424d5192485da1a2d67bfdb2bbbf [id=40c29df99590408cbd5c583d4fc741eb]). Expected version 2 got 0
I20251025 07:52:38.652906 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet cd23b8094d7e472eba6bd170248858a9 (table default.Table_ca479cb5c1184c5391b7e1c9d8a08937 [id=d82b3e1317434cf08a0d51e8a9427224]). Expected version 3 got 2
I20251025 07:52:38.653501 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b21b7da1e2f47829c3f6cdaee689e8a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.654789 754 tablet_service.cc:1552] Processing DeleteTablet for tablet f3001314b56345f5a70314777cae0125 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.655624 754 tablet_service.cc:1552] Processing DeleteTablet for tablet f170424be2f643e7af499834955472b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.656911 754 tablet_service.cc:1552] Processing DeleteTablet for tablet a13bf4d7eb1440f7a28b85cb1bf99667 with delete_type TABLET_DATA_DELETED (replaced by tablet 74e2d78c024849b0af5c0bde1f48c36f) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.658386 754 tablet_service.cc:1552] Processing DeleteTablet for tablet a6ddefa1738f4ff98586900c344359d2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f0c4f0e8f014e59be9e46d2afe0be9d) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.659075 754 tablet_service.cc:1552] Processing DeleteTablet for tablet fa77085e81c7440d9e255c44d1e80c26 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.659835 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c76d1b509f04ab1a10b83cff675c11e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.667227 2166 ts_tablet_manager.cc:1929] T 410ef150a7014c6c867a021b544a2f4e P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:38.644029 1569 consensus_queue.cc:1048] T ef1854d35a3047e3a9fbe0166ab1295a P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:38.654549 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet bef0ef23d90d480d9c99aef21d7271fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.654933 1010 tablet_service.cc:1552] Processing DeleteTablet for tablet dda0b26e4f1a44928bc4b84f333901c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.671281 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b940fa5ae26427dafbcfd26372754b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.671383 2169 tablet_replica.cc:333] T bef0ef23d90d480d9c99aef21d7271fd P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:38.655195 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 80ee75935d7e461d9d4066fac9819bef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.671435 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet f661d5909657455581dcb5e6eadd3c0b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.671543 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 88eb51447e0f4080baa49dd902deb24f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.671692 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet f97acc9a638b4d5a968dcc3412a384bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.671547 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f26667f053a4f16b5da330c0b5e40f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 34315f64b89148efbf1f20a43d5c5efb) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.671833 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 410ef150a7014c6c867a021b544a2f4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.671865 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet 26c849d376d143c6bef0c8e0e1d7d03a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.671960 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 48ff4ce7846b4e56a37eee9bcb17bd5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.671989 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet b947ecc352e54f4caa73c6ca97372256 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.672086 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet f656945454fe4921ab819c772a089eb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.672142 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet e54fe7cbcffa40c4b04144b47bee2b48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.672238 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet b94634ac5077439fadfbbce569f60e8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.672322 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet 2784bbfb97d5466f9ec856add7d0b29a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.672363 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d261e432cf5403ea162129706bc4d4d with delete_type TABLET_DATA_DELETED (replaced by tablet e54fe7cbcffa40c4b04144b47bee2b48) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.672453 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet 467324e2432f4caca148951329caa292 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.672484 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c76d1b509f04ab1a10b83cff675c11e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.672600 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet b772025852794a49b072f37cccbd3291 with delete_type TABLET_DATA_DELETED (replaced by tablet f97acc9a638b4d5a968dcc3412a384bc) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.667842 754 tablet_service.cc:1552] Processing DeleteTablet for tablet b772025852794a49b072f37cccbd3291 with delete_type TABLET_DATA_DELETED (replaced by tablet f97acc9a638b4d5a968dcc3412a384bc) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.669085 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 4037dee385d04e85bb78c8e77cb00379 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.672842 752 tablet_service.cc:1552] Processing DeleteTablet for tablet ac0ea298bc114c56a8093e12def21ce1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.655864 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet db99a1e68daf4e948a3d66ad276cc5f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.656072 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet ffe7c8e1d6c44872b7b65e512764a56a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.673156 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8390711f8ab84170be461b9eeb4c361c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:38.656288 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f0c4f0e8f014e59be9e46d2afe0be9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.656514 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 80fe3255ed614270bd8352a2853fbc43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.673244 2166 log.cc:1199] T 410ef150a7014c6c867a021b544a2f4e P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/410ef150a7014c6c867a021b544a2f4e
I20251025 07:52:38.656772 1015 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b6383c99e9e4c90a374da9ccd830a6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.657075 1014 tablet_service.cc:1552] Processing DeleteTablet for tablet ebc824ec4c60411dab1a2ea4a385148e with delete_type TABLET_DATA_DELETED (replaced by tablet 29dbaf9149134f548a974a5779f7da67) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.657464 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet ce1ebb1ec14b4801930a72765bca1a7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.657760 1012 tablet_service.cc:1552] Processing DeleteTablet for tablet 91728987303b41fd840eb56f3a311c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.658097 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 050a5d3b5f934dd19bf2e221f2fe0780 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.658517 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet aaef24ccdfba45fe8204327723df834e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.658897 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet 0f5f6ca6c3cd4d369690e425387a4344 with delete_type TABLET_DATA_DELETED (replaced by tablet 22fc90c704f64d2f9f4b8c54dee45576) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.659274 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet d9bfd7c0c3ad4e5b98b6708793d61d41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.673686 2166 ts_tablet_manager.cc:1950] T 410ef150a7014c6c867a021b544a2f4e P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:38.671296 1010 tablet_service.cc:1552] Processing DeleteTablet for tablet b3a0af893a8947848e12093d0a7f0229 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.661980 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 04449cc97f404a19b6a1b20dea074f95 with delete_type TABLET_DATA_DELETED (replaced by tablet f70091024c034c118fb86578db19a31c) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.664070 1011 tablet_service.cc:1552] Processing DeleteTablet for tablet f43ef51ee45d47a99ec732b49d50d3b4 with delete_type TABLET_DATA_DELETED (replaced by tablet e429b94326cc4827a03968fd92de38e0) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.665685 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet d37758f606344127b3db74c87babe69c with delete_type TABLET_DATA_DELETED (replaced by tablet 07898860aa594a05a211fc89768cb76d) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.655613 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e6544202d4f489d82f22d6073ef240d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.674429 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 87a93778620f4c84b3b3f7a3d7d7e87f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.675020 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 2aa522477e934c89834026dc480fe7c0 with delete_type TABLET_DATA_DELETED (replaced by tablet 1068f566477e4c6db7dde566fff4c2fb) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.675637 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 3775a2e08d4e40bd9f57e4cda759d535 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.676323 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 35ba9d23262248cda6fdf03c51be11a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 01f82980559b4662bdc09786ed429ddc) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.676661 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet a6ddefa1738f4ff98586900c344359d2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f0c4f0e8f014e59be9e46d2afe0be9d) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.677186 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 410ef150a7014c6c867a021b544a2f4e (table default.Table_39bd22f18d6540d3be94520671683032 [id=baf017a8539143ad9c8a443a9e1b6339]) successfully deleted
I20251025 07:52:38.677897 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 2209089667dd4937abd96615f6929449 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:58412
W20251025 07:52:38.678751 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 88eb51447e0f4080baa49dd902deb24f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88eb51447e0f4080baa49dd902deb24f already in progress: creating tablet
I20251025 07:52:38.679329 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 74e2d78c024849b0af5c0bde1f48c36f with delete_type TABLET_DATA_DELETED (replaced by tablet fd265cdbf6a8440393a2b5d8240d1872) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.679466 1659 raft_consensus.cc:493] T 36262ab8c6964dd684c01b4db6219a4f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:38.679630 1659 raft_consensus.cc:515] T 36262ab8c6964dd684c01b4db6219a4f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.680055 2166 tablet_replica.cc:333] T 2f26667f053a4f16b5da330c0b5e40f9 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:38.680274 1659 leader_election.cc:290] T 36262ab8c6964dd684c01b4db6219a4f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:38.680936 712 consensus_peers.cc:597] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:38.680974 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 5fca357d7e24494fb6af4bd73d98833c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.681224 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36262ab8c6964dd684c01b4db6219a4f" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:38.681761 712 leader_election.cc:343] T 36262ab8c6964dd684c01b4db6219a4f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:38.682401 1569 consensus_queue.cc:1048] T 896bd7870d87408e8adfdb7a78998111 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:38.682564 745 tablet_service.cc:1552] Processing DeleteTablet for tablet c4610b0c62e741eca28c21f63cc3de81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.683224 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b594d4718344aafbdaec34040695589 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.094511 1612 consensus_queue.cc:1048] T b97e39aaac3048ed93dbfe873b757254 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:38.683823 2183 consensus_queue.cc:1048] T 53198283718d4b45b30df65b5bde96a6 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:38.691715 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet fa77085e81c7440d9e255c44d1e80c26 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.693267 2169 raft_consensus.cc:2243] T bef0ef23d90d480d9c99aef21d7271fd P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:38.693475 2169 raft_consensus.cc:2272] T bef0ef23d90d480d9c99aef21d7271fd P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:38.694062 2169 ts_tablet_manager.cc:1916] T bef0ef23d90d480d9c99aef21d7271fd P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:38.697748 1659 raft_consensus.cc:493] T 34315f64b89148efbf1f20a43d5c5efb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:38.697911 1659 raft_consensus.cc:515] T 34315f64b89148efbf1f20a43d5c5efb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.698592 1659 leader_election.cc:290] T 34315f64b89148efbf1f20a43d5c5efb P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:38.699885 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34315f64b89148efbf1f20a43d5c5efb" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:38.700692 2169 ts_tablet_manager.cc:1929] T bef0ef23d90d480d9c99aef21d7271fd P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:38.684321 2166 raft_consensus.cc:2243] T 2f26667f053a4f16b5da330c0b5e40f9 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:38.700930 2166 raft_consensus.cc:2272] T 2f26667f053a4f16b5da330c0b5e40f9 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:38.701016 2169 log.cc:1199] T bef0ef23d90d480d9c99aef21d7271fd P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/bef0ef23d90d480d9c99aef21d7271fd
I20251025 07:52:38.701404 2169 ts_tablet_manager.cc:1950] T bef0ef23d90d480d9c99aef21d7271fd P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:38.693105 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b21b7da1e2f47829c3f6cdaee689e8a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.704499 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet fd265cdbf6a8440393a2b5d8240d1872 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.705876 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 887e7c7a5d0e4aaf88f26246306df19c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.708052 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet c11f674385184ae6b235f1db8bf1fb91 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.708884 2169 tablet_replica.cc:333] T 2f26667f053a4f16b5da330c0b5e40f9 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:38.709496 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet d77704b4fe6c4456a2ba40ffec0c60fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.711012 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet e4da89f863cc43f3bf6df908bb9dc6bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.711455 2169 raft_consensus.cc:2243] T 2f26667f053a4f16b5da330c0b5e40f9 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:38.712824 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 4037dee385d04e85bb78c8e77cb00379 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.713624 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet e47c4f741f354a19b49777bd06e54764 with delete_type TABLET_DATA_DELETED (replaced by tablet 755f0d6c87044e20a8020f8410940123) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.714175 2169 raft_consensus.cc:2272] T 2f26667f053a4f16b5da330c0b5e40f9 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:38.715204 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 72dc084f8efc45478771c247e07a16f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.715626 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet be5ab0db6ac34c2c9a370445e251eb46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.716396 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f7c15529be54a11961f385a953b0b03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.717553 2169 ts_tablet_manager.cc:1916] T 2f26667f053a4f16b5da330c0b5e40f9 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:38.719727 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet abf94c32ff2945119b710905c075d63c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.720090 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 1912e59548874e0e8e989050ec0064c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.720536 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b594d4718344aafbdaec34040695589 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.721096 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet ee1c8ea694284d80a30f9c05be4cbf49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
W20251025 07:52:38.701581 712 leader_election.cc:343] T 34315f64b89148efbf1f20a43d5c5efb P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:38.701612 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 02e4c4d6ffb34cecaf17beb5ba197e81 with delete_type TABLET_DATA_DELETED (replaced by tablet 3abfd52327a04ba88625f391a8658fbb) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.722393 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 7f29b64704e147aa9b0b5f6cda860f09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.722856 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c7a9d7a4fb3449bb57eafa83230a839 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:33 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.723313 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 6dcab53a020847a1aa27930265ba0205 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.724304 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f25c9caa05b4da4bd56a8a21fde423c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.724891 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 943ec703e65f4fd3b0c690e08c0bd226 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.727320 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 70fe9aa2f58d4abea3b3b71066a29653 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.727841 1167 tablet_bootstrap.cc:492] T b3a0af893a8947848e12093d0a7f0229 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:38.728096 1167 ts_tablet_manager.cc:1403] T b3a0af893a8947848e12093d0a7f0229 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.115s user 0.002s sys 0.014s
I20251025 07:52:38.728945 1167 raft_consensus.cc:359] T b3a0af893a8947848e12093d0a7f0229 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.729182 745 tablet_service.cc:1552] Processing DeleteTablet for tablet f4578ce69a0d4322944e5ce7c8a59831 with delete_type TABLET_DATA_DELETED (replaced by tablet 48b73e0c0dce4be686d05f991a21bd19) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.729269 1167 raft_consensus.cc:385] T b3a0af893a8947848e12093d0a7f0229 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:38.729367 1167 raft_consensus.cc:740] T b3a0af893a8947848e12093d0a7f0229 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:38.729668 1167 consensus_queue.cc:260] T b3a0af893a8947848e12093d0a7f0229 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.730175 745 tablet_service.cc:1552] Processing DeleteTablet for tablet e54fe7cbcffa40c4b04144b47bee2b48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.730322 1167 ts_tablet_manager.cc:1434] T b3a0af893a8947848e12093d0a7f0229 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:38.731716 1167 tablet_bootstrap.cc:492] T ebf3a2a926754305b5bd95fcebb25d29 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:38.733858 1169 tablet_bootstrap.cc:492] T 410ef150a7014c6c867a021b544a2f4e P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:38.734171 1169 ts_tablet_manager.cc:1403] T 410ef150a7014c6c867a021b544a2f4e P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.138s user 0.004s sys 0.001s
I20251025 07:52:38.735074 745 tablet_service.cc:1552] Processing DeleteTablet for tablet f656945454fe4921ab819c772a089eb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.735692 1169 raft_consensus.cc:359] T 410ef150a7014c6c867a021b544a2f4e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.727854 2166 ts_tablet_manager.cc:1916] T 2f26667f053a4f16b5da330c0b5e40f9 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:38.736012 1169 raft_consensus.cc:385] T 410ef150a7014c6c867a021b544a2f4e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:38.736106 1169 raft_consensus.cc:740] T 410ef150a7014c6c867a021b544a2f4e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:38.736488 1169 consensus_queue.cc:260] T 410ef150a7014c6c867a021b544a2f4e P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.737107 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 71432a6848944a0da06ef9b81f160dc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.740267 745 tablet_service.cc:1552] Processing DeleteTablet for tablet c59f6e6eb9594a8e83f109a09b034045 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.741175 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 8956ad28b35d45f28c683ac055bba441 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.742435 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet cc407c7eb12e4eadb41fb21385611c94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.742799 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 1b5ac7e845a24b2eb5b2429d47361232 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.743175 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 21e52928580c4762b0ae7152490ef955 with delete_type TABLET_DATA_DELETED (replaced by tablet 9d6a9937d10a4cc2813671dd1bfa1664) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.743153 745 tablet_service.cc:1552] Processing DeleteTablet for tablet ffe7c8e1d6c44872b7b65e512764a56a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.743791 1021 tablet_service.cc:1505] Processing CreateTablet for tablet 3c29d2fe91e54aa9a40e50f38b710c36 (DEFAULT_TABLE table=default.Table_6eff76f8460c40668372f850e2fc299a [id=c74693e799854673b6865d11b914477a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:38.744246 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c29d2fe91e54aa9a40e50f38b710c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:38.744277 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet ef1854d35a3047e3a9fbe0166ab1295a (table default.Table_088aad2a46b041bfa29ed8e6929760b1 [id=834669c54dae449790818684fd8659bc]). Expected version 2 got 0
I20251025 07:52:38.744549 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet bc2d16b33ca8426abbd7bd38339c9f1d (table default.Table_4ea985df4c5b415c9570c64506a98cc4 [id=98d2b7797e33485e8872054aa2eb8ec9]). Expected version 1 got 0
I20251025 07:52:38.744727 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e]). Expected version 1 got 0
I20251025 07:52:38.745072 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]). Expected version 1 got 0
I20251025 07:52:38.745276 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet ef9a2c9f2ccf432ab835396020872208 (table default.Table_2b9b772d4c234ac286bef3658482e490 [id=409fab60f9d744798e6f92b868f0f7c2]). Expected version 1 got 0
I20251025 07:52:38.745417 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]). Expected version 1 got 0
I20251025 07:52:38.752918 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e098c9c4613459295b6043048cba587 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.753331 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet ac0ea298bc114c56a8093e12def21ce1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.754047 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 27651312cc2e42a8bfcd245f5d301679 with delete_type TABLET_DATA_DELETED (replaced by tablet a4e4b99be0434baaa75d988281797846) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.754349 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet fab67b56eaa44b54886c17181d8c52b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.755687 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 95efd553493e4966bc48a01cac6e1e10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.755951 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 896bd7870d87408e8adfdb7a78998111 with delete_type TABLET_DATA_DELETED (replaced by tablet e8402fd33ba842e981c6c68787c7c548) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.757433 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet a13bf4d7eb1440f7a28b85cb1bf99667 with delete_type TABLET_DATA_DELETED (replaced by tablet 74e2d78c024849b0af5c0bde1f48c36f) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.757733 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 570201e7bf05406a97c81dde071d894b with delete_type TABLET_DATA_DELETED (replaced by tablet 3a0f50c8000144a4acb5219484b01800) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.759145 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet f3001314b56345f5a70314777cae0125 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:44572
W20251025 07:52:38.759346 982 consensus_peers.cc:597] T d37758f606344127b3db74c87babe69c P db42be7ad3c741a0aa427f27e245993b -> Peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Couldn't send request to peer 23eeffd3de254ce18db5febfde971c6b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d37758f606344127b3db74c87babe69c. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:38.760802 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet f170424be2f643e7af499834955472b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.762152 1612 consensus_queue.cc:1048] T 509c94f2a422493ea4390ecee576db72 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:38.763155 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 943ec703e65f4fd3b0c690e08c0bd226 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.762630 1169 ts_tablet_manager.cc:1434] T 410ef150a7014c6c867a021b544a2f4e P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.028s user 0.001s sys 0.001s
I20251025 07:52:38.764935 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 3775a2e08d4e40bd9f57e4cda759d535 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.570258 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b28215f61574cf48c447dd01955ac77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.570778 893 tablet_service.cc:1552] Processing DeleteTablet for tablet af743ee344e543a0aa6da3cd7a30e467 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.570996 880 tablet_service.cc:1552] Processing DeleteTablet for tablet 34315f64b89148efbf1f20a43d5c5efb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.571245 878 tablet_service.cc:1552] Processing DeleteTablet for tablet a4e4b99be0434baaa75d988281797846 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.571434 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 92aa605fabbc4257a6a204c087e7cc7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.574270 875 tablet_service.cc:1552] Processing DeleteTablet for tablet 3abfd52327a04ba88625f391a8658fbb with delete_type TABLET_DATA_DELETED (replaced by tablet 04d27d54fdb7483b869408af65ff85c9) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.575915 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b947ecc352e54f4caa73c6ca97372256" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:38.574337 894 tablet_service.cc:1552] Processing DeleteTablet for tablet d66ad049d5484228a268b0facb00ecf1 with delete_type TABLET_DATA_DELETED (replaced by tablet 2e098c9c4613459295b6043048cba587) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.582666 1166 tablet_bootstrap.cc:654] T 3abfd52327a04ba88625f391a8658fbb P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:38.768018 710 leader_election.cc:343] T b947ecc352e54f4caa73c6ca97372256 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:38.768206 710 leader_election.cc:304] T b947ecc352e54f4caa73c6ca97372256 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:38.613497 2211 tablet.cc:1722] T 87268e27540c4ec58ce1e62df819f19d P 3aea2869ff814cc7b142012faa274382: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251025 07:52:38.680876 907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36262ab8c6964dd684c01b4db6219a4f" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:38.699235 1838 raft_consensus.cc:493] T ef1854d35a3047e3a9fbe0166ab1295a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:38.769378 1838 raft_consensus.cc:515] T ef1854d35a3047e3a9fbe0166ab1295a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.769495 1659 raft_consensus.cc:2749] T b947ecc352e54f4caa73c6ca97372256 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:38.769773 710 leader_election.cc:343] T 36262ab8c6964dd684c01b4db6219a4f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:38.769910 710 leader_election.cc:304] T 36262ab8c6964dd684c01b4db6219a4f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:38.770011 1838 leader_election.cc:290] T ef1854d35a3047e3a9fbe0166ab1295a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:38.770097 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 627a10a0714f4be8bdbced503fd6388d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:33 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.699254 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34315f64b89148efbf1f20a43d5c5efb" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:38.770242 1612 consensus_queue.cc:1048] T 551d07a9242740dcb9366bb2d707a4a8 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:38.770357 1659 raft_consensus.cc:2749] T 36262ab8c6964dd684c01b4db6219a4f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:38.770357 1169 tablet_bootstrap.cc:492] T 50ae9bca6bed4f749d54191ac1543410 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:38.770903 745 tablet_service.cc:1552] Processing DeleteTablet for tablet db99a1e68daf4e948a3d66ad276cc5f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:58412
W20251025 07:52:38.771205 710 leader_election.cc:343] T 34315f64b89148efbf1f20a43d5c5efb P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:38.670418 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d66ad049d5484228a268b0facb00ecf1" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:38.771857 902 raft_consensus.cc:3060] T d66ad049d5484228a268b0facb00ecf1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:38.772255 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 35ba9d23262248cda6fdf03c51be11a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 01f82980559b4662bdc09786ed429ddc) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.773497 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 2209089667dd4937abd96615f6929449 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.774681 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 74e2d78c024849b0af5c0bde1f48c36f with delete_type TABLET_DATA_DELETED (replaced by tablet fd265cdbf6a8440393a2b5d8240d1872) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.775477 902 raft_consensus.cc:2468] T d66ad049d5484228a268b0facb00ecf1 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:38.775748 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 5fca357d7e24494fb6af4bd73d98833c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.771502 710 leader_election.cc:304] T 34315f64b89148efbf1f20a43d5c5efb P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:38.776273 754 tablet_service.cc:1552] Processing DeleteTablet for tablet d5e5e5bbcf9f43118eea502b69880af5 with delete_type TABLET_DATA_DELETED (replaced by tablet f4578ce69a0d4322944e5ce7c8a59831) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.776679 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef1854d35a3047e3a9fbe0166ab1295a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:38.776896 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e6544202d4f489d82f22d6073ef240d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.777141 1659 raft_consensus.cc:2749] T 34315f64b89148efbf1f20a43d5c5efb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:38.777613 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 5361f08f50b745eb98bf02e979d7c71e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.777817 2211 tablet.cc:1722] T cd23b8094d7e472eba6bd170248858a9 P 3aea2869ff814cc7b142012faa274382: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251025 07:52:38.778067 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 53198283718d4b45b30df65b5bde96a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.778472 745 tablet_service.cc:1552] Processing DeleteTablet for tablet c88c46e022ff41a58b02538f2151e260 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.778472 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet c4610b0c62e741eca28c21f63cc3de81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.778679 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 02e4c4d6ffb34cecaf17beb5ba197e81 with delete_type TABLET_DATA_DELETED (replaced by tablet 3abfd52327a04ba88625f391a8658fbb) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.778952 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d6a9937d10a4cc2813671dd1bfa1664 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.779383 745 tablet_service.cc:1552] Processing DeleteTablet for tablet f3c87a83082b421cb4be546ac4ce7ad8 with delete_type TABLET_DATA_DELETED (replaced by tablet b17d2a9ad6be49d29f2413666fba7e69) from {username='slave'} at 127.0.0.1:58412
W20251025 07:52:38.780293 451 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:35777 (request call id 189) took 2419 ms (client timeout 1000 ms). Trace:
I20251025 07:52:38.780855 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c7a9d7a4fb3449bb57eafa83230a839 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:33 UTC) from {username='slave'} at 127.0.0.1:44572
W20251025 07:52:38.780460 451 rpcz_store.cc:269] 1025 07:52:36.361216 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:36.686435 (+325219us) service_pool.cc:225] Handling call
1025 07:52:36.802459 (+116024us) spinlock_profiling.cc:243] Waited 75.4 ms on lock 0x616000041254. stack: 00007fb90f7779f6 00007fb90f7777b0 00007fb90e7533ad 0000000000983f04 0000000000983ecd 00007fb91ea756bd 00007fb91ea757b3 00007fb91b640441 00007fb9232f69dd 00007fb92347daa8 00007fb919f96ca6 00007fb919f969d5 00007fb91671327e 00007fb91671215e 00007fb916718395 00007fb916718cbf
1025 07:52:38.780230 (+1977771us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:38.072946 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:38.073097 (+ 151us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:38.073118 (+ 21us) write_op.cc:435] Acquired schema lock
1025 07:52:38.073123 (+ 5us) tablet.cc:660] Decoding operations
1025 07:52:38.073233 (+ 110us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:38.073252 (+ 19us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:38.073260 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:38.073336 (+ 76us) tablet.cc:699] Row locks acquired
1025 07:52:38.073340 (+ 4us) write_op.cc:260] PREPARE: finished
1025 07:52:38.073380 (+ 40us) write_op.cc:270] Start()
1025 07:52:38.073417 (+ 37us) write_op.cc:276] Timestamp: P: 1761378758073360 usec, L: 0
1025 07:52:38.073421 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:38.073628 (+ 207us) log.cc:844] Serialized 527 byte log entry
1025 07:52:38.077329 (+ 3701us) op_driver.cc:464] REPLICATION: finished
1025 07:52:38.077499 (+ 170us) write_op.cc:301] APPLY: starting
1025 07:52:38.077513 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:38.077560 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:38.077601 (+ 41us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:38.077665 (+ 64us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:38.077698 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:38.077709 (+ 11us) write_op.cc:312] APPLY: finished
1025 07:52:38.077810 (+ 101us) log.cc:844] Serialized 23 byte log entry
1025 07:52:38.077873 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.077894 (+ 21us) write_op.cc:454] Released schema lock
1025 07:52:38.077903 (+ 9us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:38.077990 (+ 87us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.077995 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":55796,"reported_tablets":246,"spinlock_wait_cycles":173450112,"child_traces":[["op",{"apply.queue_time_us":117,"apply.run_cpu_time_us":482,"apply.run_wall_time_us":481,"num_ops":1,"prepare.queue_time_us":56,"prepare.run_cpu_time_us":811,"prepare.run_wall_time_us":811,"raft.queue_time_us":435,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":262,"replication_time_us":3876}]]}
I20251025 07:52:38.644711 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d66ad049d5484228a268b0facb00ecf1" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:38.787644 910 raft_consensus.cc:2376] T d66ad049d5484228a268b0facb00ecf1 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1. Re-sending same reply.
I20251025 07:52:38.748589 2215 tablet.cc:1722] T 6897405bbc1c47a4ab0aa3af80e60e3d P 3aea2869ff814cc7b142012faa274382: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251025 07:52:38.570119 2146 ts_tablet_manager.cc:1929] T d37758f606344127b3db74c87babe69c P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:38.792109 2146 log.cc:1199] T d37758f606344127b3db74c87babe69c P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/d37758f606344127b3db74c87babe69c
I20251025 07:52:38.792488 2146 ts_tablet_manager.cc:1950] T d37758f606344127b3db74c87babe69c P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:38.794582 1167 tablet_bootstrap.cc:654] T ebf3a2a926754305b5bd95fcebb25d29 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:38.797796 910 raft_consensus.cc:3060] T ef1854d35a3047e3a9fbe0166ab1295a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:38.798946 1166 tablet_bootstrap.cc:492] T 3abfd52327a04ba88625f391a8658fbb P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:38.799077 1166 ts_tablet_manager.cc:1403] T 3abfd52327a04ba88625f391a8658fbb P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.273s user 0.006s sys 0.000s
I20251025 07:52:38.800040 1166 raft_consensus.cc:359] T 3abfd52327a04ba88625f391a8658fbb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.800200 1166 raft_consensus.cc:385] T 3abfd52327a04ba88625f391a8658fbb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:38.800246 1166 raft_consensus.cc:740] T 3abfd52327a04ba88625f391a8658fbb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:38.800411 1166 consensus_queue.cc:260] T 3abfd52327a04ba88625f391a8658fbb P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.800958 1166 ts_tablet_manager.cc:1434] T 3abfd52327a04ba88625f391a8658fbb P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:38.802487 2183 consensus_queue.cc:1048] T ef1854d35a3047e3a9fbe0166ab1295a P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:38.808838 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 70fe9aa2f58d4abea3b3b71066a29653 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.809595 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet f4578ce69a0d4322944e5ce7c8a59831 with delete_type TABLET_DATA_DELETED (replaced by tablet 48b73e0c0dce4be686d05f991a21bd19) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.810257 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet f0bdaa8cfd764f1b83f47881353d26b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.811230 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet a7ab3150a35b4caca9119a1374d3f2f7 with delete_type TABLET_DATA_DELETED (replaced by tablet c19c97f881cb45218d4a6f6f231aa96e) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.811882 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet cb824bd780524832b2b60f5618a39da5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.812552 2146 tablet_replica.cc:333] T 410ef150a7014c6c867a021b544a2f4e P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:38.812944 1169 tablet_bootstrap.cc:654] T 50ae9bca6bed4f749d54191ac1543410 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:38.814690 2146 raft_consensus.cc:2243] T 410ef150a7014c6c867a021b544a2f4e P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:38.813328 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 8956ad28b35d45f28c683ac055bba441 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.813859 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 71432a6848944a0da06ef9b81f160dc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.814357 2169 ts_tablet_manager.cc:1929] T 2f26667f053a4f16b5da330c0b5e40f9 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:38.814627 1011 tablet_service.cc:1552] Processing DeleteTablet for tablet c59f6e6eb9594a8e83f109a09b034045 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.815133 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet c88c46e022ff41a58b02538f2151e260 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.815599 1010 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d6a9937d10a4cc2813671dd1bfa1664 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.816296 2169 log.cc:1199] T 2f26667f053a4f16b5da330c0b5e40f9 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/2f26667f053a4f16b5da330c0b5e40f9
I20251025 07:52:38.817344 1010 tablet_service.cc:1552] Processing DeleteTablet for tablet d5e5e5bbcf9f43118eea502b69880af5 with delete_type TABLET_DATA_DELETED (replaced by tablet f4578ce69a0d4322944e5ce7c8a59831) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.818549 1010 tablet_service.cc:1552] Processing DeleteTablet for tablet 53198283718d4b45b30df65b5bde96a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.819726 2146 raft_consensus.cc:2272] T 410ef150a7014c6c867a021b544a2f4e P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:38.820084 1010 tablet_service.cc:1552] Processing DeleteTablet for tablet f3c87a83082b421cb4be546ac4ce7ad8 with delete_type TABLET_DATA_DELETED (replaced by tablet b17d2a9ad6be49d29f2413666fba7e69) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.820472 2146 ts_tablet_manager.cc:1916] T 410ef150a7014c6c867a021b544a2f4e P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:38.820401 2169 ts_tablet_manager.cc:1950] T 2f26667f053a4f16b5da330c0b5e40f9 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:38.821604 1010 tablet_service.cc:1552] Processing DeleteTablet for tablet 08da9d2687704725aacd11cd92c6a9f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.812883 1166 tablet_bootstrap.cc:492] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:38.822774 1010 tablet_service.cc:1552] Processing DeleteTablet for tablet 551d07a9242740dcb9366bb2d707a4a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.823629 1166 tablet_bootstrap.cc:654] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:38.824493 1010 tablet_service.cc:1552] Processing DeleteTablet for tablet 7fbae8be73bf4e18b5a5c4ec4cdebc12 with delete_type TABLET_DATA_DELETED (replaced by tablet 4c155aa7fa5b4d68bbab1918b8432372) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.825392 1169 tablet_bootstrap.cc:492] T 50ae9bca6bed4f749d54191ac1543410 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:38.833863 1167 tablet_bootstrap.cc:492] T ebf3a2a926754305b5bd95fcebb25d29 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:38.834228 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet d37758f606344127b3db74c87babe69c (table default.Table_92cbcd8230f249a8b2821f69a81b41f1 [id=08e53121e86d405dbbd02a4605695fec]) successfully deleted
I20251025 07:52:38.835944 1169 ts_tablet_manager.cc:1403] T 50ae9bca6bed4f749d54191ac1543410 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.067s user 0.005s sys 0.000s
I20251025 07:52:38.831398 1612 consensus_queue.cc:1048] T 509c94f2a422493ea4390ecee576db72 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:38.836099 2205 tablet.cc:1722] T 8e6d76b1a97945cfb6c3055e1b622f8a P db42be7ad3c741a0aa427f27e245993b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251025 07:52:38.834498 1010 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ec421eb820b4f40afcff5a5bf80c508 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.835004 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 3abfd52327a04ba88625f391a8658fbb with delete_type TABLET_DATA_DELETED (replaced by tablet 04d27d54fdb7483b869408af65ff85c9) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.830765 2169 tablet_replica.cc:333] T 410ef150a7014c6c867a021b544a2f4e P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:38.837843 2169 raft_consensus.cc:2243] T 410ef150a7014c6c867a021b544a2f4e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:38.838860 2169 raft_consensus.cc:2272] T 410ef150a7014c6c867a021b544a2f4e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:38.839609 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet d66ad049d5484228a268b0facb00ecf1 with delete_type TABLET_DATA_DELETED (replaced by tablet 2e098c9c4613459295b6043048cba587) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.841787 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet af743ee344e543a0aa6da3cd7a30e467 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
W20251025 07:52:38.842286 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 0f5f6ca6c3cd4d369690e425387a4344 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f5f6ca6c3cd4d369690e425387a4344 already in progress: creating tablet
W20251025 07:52:38.842682 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet f43ef51ee45d47a99ec732b49d50d3b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f43ef51ee45d47a99ec732b49d50d3b4 already in progress: creating tablet
I20251025 07:52:38.843288 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 34315f64b89148efbf1f20a43d5c5efb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.836915 2216 tablet.cc:1722] T 8e6d76b1a97945cfb6c3055e1b622f8a P 23eeffd3de254ce18db5febfde971c6b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251025 07:52:38.843909 1167 ts_tablet_manager.cc:1403] T ebf3a2a926754305b5bd95fcebb25d29 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.112s user 0.004s sys 0.000s
I20251025 07:52:38.844859 1167 raft_consensus.cc:359] T ebf3a2a926754305b5bd95fcebb25d29 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.846644 2226 tablet.cc:1722] T 8e6d76b1a97945cfb6c3055e1b622f8a P 3aea2869ff814cc7b142012faa274382: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251025 07:52:38.847556 1671 consensus_queue.cc:1048] T 551d07a9242740dcb9366bb2d707a4a8 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:38.849664 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 50ae9bca6bed4f749d54191ac1543410 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.844419 1169 raft_consensus.cc:359] T 50ae9bca6bed4f749d54191ac1543410 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.850479 2105 consensus_queue.cc:1048] T 2c50ec95e48e49b59a76fac4fc862d3d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:38.850560 2169 ts_tablet_manager.cc:1916] T 410ef150a7014c6c867a021b544a2f4e P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:38.850713 1169 raft_consensus.cc:385] T 50ae9bca6bed4f749d54191ac1543410 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:38.851440 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet a4e4b99be0434baaa75d988281797846 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.851295 1169 raft_consensus.cc:740] T 50ae9bca6bed4f749d54191ac1543410 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
W20251025 07:52:38.853436 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 716fb61312e94a4d9eb8ab8336ca140c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 716fb61312e94a4d9eb8ab8336ca140c already in progress: creating tablet
I20251025 07:52:38.852968 1169 consensus_queue.cc:260] T 50ae9bca6bed4f749d54191ac1543410 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.854970 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet bef0ef23d90d480d9c99aef21d7271fd (table default.Table_e3b368ac698c46f4a8c947d02f9a2680 [id=ce5f22039d104e518ef6f40468b404cc]) successfully deleted
W20251025 07:52:38.855572 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet dda0b26e4f1a44928bc4b84f333901c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dda0b26e4f1a44928bc4b84f333901c0 already in progress: creating tablet
W20251025 07:52:38.856171 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 3b940fa5ae26427dafbcfd26372754b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b940fa5ae26427dafbcfd26372754b3 already in progress: creating tablet
W20251025 07:52:38.856844 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 80ee75935d7e461d9d4066fac9819bef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80ee75935d7e461d9d4066fac9819bef already in progress: creating tablet
W20251025 07:52:38.857429 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet f661d5909657455581dcb5e6eadd3c0b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f661d5909657455581dcb5e6eadd3c0b already in progress: creating tablet
W20251025 07:52:38.857982 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 88eb51447e0f4080baa49dd902deb24f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88eb51447e0f4080baa49dd902deb24f already in progress: creating tablet
I20251025 07:52:38.858404 1166 tablet_bootstrap.cc:492] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:38.858368 1169 ts_tablet_manager.cc:1434] T 50ae9bca6bed4f749d54191ac1543410 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
W20251025 07:52:38.858759 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet f97acc9a638b4d5a968dcc3412a384bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f97acc9a638b4d5a968dcc3412a384bc already in progress: creating tablet
I20251025 07:52:38.859726 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 2f26667f053a4f16b5da330c0b5e40f9 (table default.Table_4b3d7b3a731942189cdc214938a45820 [id=88f1e8b1c5fa4913a5f7b5b2c6cbf6ec]) successfully deleted
I20251025 07:52:38.860657 1169 tablet_bootstrap.cc:492] T 80ee75935d7e461d9d4066fac9819bef P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:38.860878 1167 raft_consensus.cc:385] T ebf3a2a926754305b5bd95fcebb25d29 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:38.862682 1167 raft_consensus.cc:740] T ebf3a2a926754305b5bd95fcebb25d29 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:38.864117 1166 ts_tablet_manager.cc:1403] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.000s
I20251025 07:52:38.863538 1167 consensus_queue.cc:260] T ebf3a2a926754305b5bd95fcebb25d29 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.866080 1167 ts_tablet_manager.cc:1434] T ebf3a2a926754305b5bd95fcebb25d29 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20251025 07:52:38.873835 1169 tablet_bootstrap.cc:654] T 80ee75935d7e461d9d4066fac9819bef P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:38.879627 2050 raft_consensus.cc:493] T fa998b9ca6724c4cb1eb68a3eecb0b11 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:38.879776 2050 raft_consensus.cc:515] T fa998b9ca6724c4cb1eb68a3eecb0b11 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.880234 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef1854d35a3047e3a9fbe0166ab1295a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:38.880379 2050 leader_election.cc:290] T fa998b9ca6724c4cb1eb68a3eecb0b11 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:38.881232 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa998b9ca6724c4cb1eb68a3eecb0b11" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:38.881506 773 raft_consensus.cc:2468] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:38.882020 982 leader_election.cc:304] T fa998b9ca6724c4cb1eb68a3eecb0b11 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:38.883296 1612 raft_consensus.cc:2804] T fa998b9ca6724c4cb1eb68a3eecb0b11 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:38.883364 1612 raft_consensus.cc:493] T fa998b9ca6724c4cb1eb68a3eecb0b11 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:38.883428 1612 raft_consensus.cc:3060] T fa998b9ca6724c4cb1eb68a3eecb0b11 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:38.883574 850 leader_election.cc:304] T ef1854d35a3047e3a9fbe0166ab1295a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:38.884305 2106 raft_consensus.cc:2749] T ef1854d35a3047e3a9fbe0166ab1295a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:38.888208 1612 raft_consensus.cc:515] T fa998b9ca6724c4cb1eb68a3eecb0b11 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.888926 1612 leader_election.cc:290] T fa998b9ca6724c4cb1eb68a3eecb0b11 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:38.889420 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa998b9ca6724c4cb1eb68a3eecb0b11" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:38.889622 773 raft_consensus.cc:3060] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:38.889849 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa998b9ca6724c4cb1eb68a3eecb0b11" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:38.890044 448 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet 062c0f3e51a3487880cea373f8cf1f54
I20251025 07:52:38.890581 448 catalog_manager.cc:6719] ReplaceTablet: tablet 062c0f3e51a3487880cea373f8cf1f54 deleted and replaced by tablet b56209e4934e4407a8b4518ca4af0d46
I20251025 07:52:38.890574 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa998b9ca6724c4cb1eb68a3eecb0b11" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:38.890688 448 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:47610 (request call id 49) took 1737 ms. Trace:
I20251025 07:52:38.890779 448 rpcz_store.cc:276] 1025 07:52:37.152966 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.493583 (+1340617us) service_pool.cc:225] Handling call
1025 07:52:38.890656 (+397073us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:38.884747 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:38.884890 (+ 143us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:38.884897 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:38.884900 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:38.885024 (+ 124us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:38.885051 (+ 27us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:38.885057 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:38.885144 (+ 87us) tablet.cc:699] Row locks acquired
1025 07:52:38.885147 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:38.885185 (+ 38us) write_op.cc:270] Start()
1025 07:52:38.885224 (+ 39us) write_op.cc:276] Timestamp: P: 1761378758885166 usec, L: 0
1025 07:52:38.885228 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:38.885455 (+ 227us) log.cc:844] Serialized 622 byte log entry
1025 07:52:38.888685 (+ 3230us) op_driver.cc:464] REPLICATION: finished
1025 07:52:38.888931 (+ 246us) write_op.cc:301] APPLY: starting
1025 07:52:38.888948 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:38.888995 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:38.889021 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:38.889151 (+ 130us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:38.889221 (+ 70us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:38.889233 (+ 12us) write_op.cc:312] APPLY: finished
1025 07:52:38.889317 (+ 84us) log.cc:844] Serialized 29 byte log entry
1025 07:52:38.889380 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.889404 (+ 24us) write_op.cc:454] Released schema lock
1025 07:52:38.889411 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:38.889486 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.889489 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":127,"apply.run_cpu_time_us":550,"apply.run_wall_time_us":549,"num_ops":2,"prepare.queue_time_us":1367,"prepare.run_cpu_time_us":816,"prepare.run_wall_time_us":816,"raft.queue_time_us":414,"raft.run_cpu_time_us":295,"raft.run_wall_time_us":295,"replication_time_us":3425}]]}
I20251025 07:52:38.892510 773 raft_consensus.cc:2468] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:38.892976 982 leader_election.cc:304] T fa998b9ca6724c4cb1eb68a3eecb0b11 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters: 3aea2869ff814cc7b142012faa274382
I20251025 07:52:38.893496 2050 raft_consensus.cc:2804] T fa998b9ca6724c4cb1eb68a3eecb0b11 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:38.893599 2050 raft_consensus.cc:697] T fa998b9ca6724c4cb1eb68a3eecb0b11 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:38.893812 2050 consensus_queue.cc:237] T fa998b9ca6724c4cb1eb68a3eecb0b11 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.895290 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 062c0f3e51a3487880cea373f8cf1f54 with delete_type TABLET_DATA_DELETED (replaced by tablet b56209e4934e4407a8b4518ca4af0d46) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:38.878598 1166 raft_consensus.cc:359] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.898885 1167 tablet_bootstrap.cc:492] T 88eb51447e0f4080baa49dd902deb24f P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:38.899042 1166 raft_consensus.cc:385] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:38.900921 1166 raft_consensus.cc:740] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:38.900933 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 062c0f3e51a3487880cea373f8cf1f54 with delete_type TABLET_DATA_DELETED (replaced by tablet b56209e4934e4407a8b4518ca4af0d46) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:38.901369 1166 consensus_queue.cc:260] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.902526 1166 ts_tablet_manager.cc:1434] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.038s user 0.002s sys 0.000s
I20251025 07:52:38.903349 1166 tablet_bootstrap.cc:492] T df1248f6dbd94af68c587da7d149e279 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:38.904232 910 raft_consensus.cc:2468] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:38.905965 1166 tablet_bootstrap.cc:654] T df1248f6dbd94af68c587da7d149e279 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:38.906365 1167 tablet_bootstrap.cc:654] T 88eb51447e0f4080baa49dd902deb24f P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:38.909322 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 062c0f3e51a3487880cea373f8cf1f54 with delete_type TABLET_DATA_DELETED (replaced by tablet b56209e4934e4407a8b4518ca4af0d46) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:38.916374 1167 tablet_bootstrap.cc:492] T 88eb51447e0f4080baa49dd902deb24f P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:38.916584 1167 ts_tablet_manager.cc:1403] T 88eb51447e0f4080baa49dd902deb24f P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.000s
I20251025 07:52:38.917510 1167 raft_consensus.cc:359] T 88eb51447e0f4080baa49dd902deb24f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.917733 1167 raft_consensus.cc:385] T 88eb51447e0f4080baa49dd902deb24f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:38.917833 1167 raft_consensus.cc:740] T 88eb51447e0f4080baa49dd902deb24f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:38.918748 1167 consensus_queue.cc:260] T 88eb51447e0f4080baa49dd902deb24f P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:38.920531 1167 ts_tablet_manager.cc:1434] T 88eb51447e0f4080baa49dd902deb24f P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251025 07:52:38.925812 2169 ts_tablet_manager.cc:1929] T 410ef150a7014c6c867a021b544a2f4e P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:38.928736 2169 log.cc:1199] T 410ef150a7014c6c867a021b544a2f4e P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/410ef150a7014c6c867a021b544a2f4e
I20251025 07:52:38.935840 1659 raft_consensus.cc:493] T e54fe7cbcffa40c4b04144b47bee2b48 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:38.935997 1659 raft_consensus.cc:515] T e54fe7cbcffa40c4b04144b47bee2b48 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.936645 1659 leader_election.cc:290] T e54fe7cbcffa40c4b04144b47bee2b48 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:38.936862 1167 tablet_bootstrap.cc:492] T e9f7cd765d614d59a7c190b744b0844d P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:38.938426 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e54fe7cbcffa40c4b04144b47bee2b48" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:38.939153 1167 tablet_bootstrap.cc:654] T e9f7cd765d614d59a7c190b744b0844d P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:38.939209 710 leader_election.cc:343] T e54fe7cbcffa40c4b04144b47bee2b48 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:38.939709 2169 ts_tablet_manager.cc:1950] T 410ef150a7014c6c867a021b544a2f4e P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:38.942449 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 410ef150a7014c6c867a021b544a2f4e (table default.Table_39bd22f18d6540d3be94520671683032 [id=baf017a8539143ad9c8a443a9e1b6339]) successfully deleted
I20251025 07:52:38.947420 2050 consensus_queue.cc:1048] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:38.948621 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e54fe7cbcffa40c4b04144b47bee2b48" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:38.952782 712 leader_election.cc:343] T e54fe7cbcffa40c4b04144b47bee2b48 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:38.952963 712 leader_election.cc:304] T e54fe7cbcffa40c4b04144b47bee2b48 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:38.953423 1659 raft_consensus.cc:2749] T e54fe7cbcffa40c4b04144b47bee2b48 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:38.953730 446 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:47610 (request call id 36) took 2224 ms. Trace:
I20251025 07:52:38.953863 446 rpcz_store.cc:276] 1025 07:52:36.729433 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.306055 (+1576622us) service_pool.cc:225] Handling call
1025 07:52:38.953679 (+647624us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251025 07:52:38.955559 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 26c849d376d143c6bef0c8e0e1d7d03a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 26c849d376d143c6bef0c8e0e1d7d03a already in progress: creating tablet
W20251025 07:52:38.959776 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 48ff4ce7846b4e56a37eee9bcb17bd5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48ff4ce7846b4e56a37eee9bcb17bd5e already in progress: creating tablet
I20251025 07:52:38.960108 1659 raft_consensus.cc:493] T b94634ac5077439fadfbbce569f60e8b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:38.960263 1659 raft_consensus.cc:515] T b94634ac5077439fadfbbce569f60e8b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:38.961465 1659 leader_election.cc:290] T b94634ac5077439fadfbbce569f60e8b P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:38.962083 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b94634ac5077439fadfbbce569f60e8b" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:38.962675 710 leader_election.cc:343] T b94634ac5077439fadfbbce569f60e8b P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:38.962242 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet b947ecc352e54f4caa73c6ca97372256 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b947ecc352e54f4caa73c6ca97372256 already in progress: creating tablet
W20251025 07:52:38.965904 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet f656945454fe4921ab819c772a089eb3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f656945454fe4921ab819c772a089eb3 already in progress: creating tablet
I20251025 07:52:38.965699 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b94634ac5077439fadfbbce569f60e8b" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:38.970698 712 leader_election.cc:343] T b94634ac5077439fadfbbce569f60e8b P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:38.970922 712 leader_election.cc:304] T b94634ac5077439fadfbbce569f60e8b P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:38.971859 1659 raft_consensus.cc:2749] T b94634ac5077439fadfbbce569f60e8b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:38.973537 850 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:45417, user_credentials={real_user=slave}} blocked reactor thread for 89597.9us
I20251025 07:52:38.979388 2205 tablet.cc:1722] T 87268e27540c4ec58ce1e62df819f19d P db42be7ad3c741a0aa427f27e245993b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251025 07:52:38.980616 2166 ts_tablet_manager.cc:1929] T 2f26667f053a4f16b5da330c0b5e40f9 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:38.981057 2166 log.cc:1199] T 2f26667f053a4f16b5da330c0b5e40f9 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/2f26667f053a4f16b5da330c0b5e40f9
I20251025 07:52:38.984370 2205 tablet.cc:1722] T 6897405bbc1c47a4ab0aa3af80e60e3d P db42be7ad3c741a0aa427f27e245993b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251025 07:52:38.988070 2105 consensus_queue.cc:1048] T 2c50ec95e48e49b59a76fac4fc862d3d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251025 07:52:38.999722 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet e54fe7cbcffa40c4b04144b47bee2b48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e54fe7cbcffa40c4b04144b47bee2b48 already in progress: creating tablet
W20251025 07:52:39.001296 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet b94634ac5077439fadfbbce569f60e8b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b94634ac5077439fadfbbce569f60e8b already in progress: creating tablet
I20251025 07:52:39.001418 2169 tablet_replica.cc:333] T 2784bbfb97d5466f9ec856add7d0b29a P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:39.002526 2169 raft_consensus.cc:2243] T 2784bbfb97d5466f9ec856add7d0b29a P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:39.005614 2169 raft_consensus.cc:2272] T 2784bbfb97d5466f9ec856add7d0b29a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:39.010071 1167 tablet_bootstrap.cc:492] T e9f7cd765d614d59a7c190b744b0844d P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:39.010651 1167 ts_tablet_manager.cc:1403] T e9f7cd765d614d59a7c190b744b0844d P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.074s user 0.007s sys 0.003s
I20251025 07:52:39.011250 2216 tablet.cc:1722] T b97e39aaac3048ed93dbfe873b757254 P 23eeffd3de254ce18db5febfde971c6b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 07:52:39.011827 2169 ts_tablet_manager.cc:1916] T 2784bbfb97d5466f9ec856add7d0b29a P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:39.013067 1167 raft_consensus.cc:359] T e9f7cd765d614d59a7c190b744b0844d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.014781 1167 raft_consensus.cc:385] T e9f7cd765d614d59a7c190b744b0844d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.015218 1167 raft_consensus.cc:740] T e9f7cd765d614d59a7c190b744b0844d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.015822 1167 consensus_queue.cc:260] T e9f7cd765d614d59a7c190b744b0844d P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.016239 2252 tablet.cc:1722] T b97e39aaac3048ed93dbfe873b757254 P db42be7ad3c741a0aa427f27e245993b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 07:52:39.017411 1167 ts_tablet_manager.cc:1434] T e9f7cd765d614d59a7c190b744b0844d P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20251025 07:52:39.018187 1167 tablet_bootstrap.cc:492] T 80fe3255ed614270bd8352a2853fbc43 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:39.029004 1169 tablet_bootstrap.cc:492] T 80ee75935d7e461d9d4066fac9819bef P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:39.029341 1169 ts_tablet_manager.cc:1403] T 80ee75935d7e461d9d4066fac9819bef P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.169s user 0.006s sys 0.003s
I20251025 07:52:39.031039 1169 raft_consensus.cc:359] T 80ee75935d7e461d9d4066fac9819bef P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.031260 1169 raft_consensus.cc:385] T 80ee75935d7e461d9d4066fac9819bef P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.031365 1169 raft_consensus.cc:740] T 80ee75935d7e461d9d4066fac9819bef P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.031430 2215 tablet.cc:1722] T b97e39aaac3048ed93dbfe873b757254 P 3aea2869ff814cc7b142012faa274382: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 07:52:39.032590 1169 consensus_queue.cc:260] T 80ee75935d7e461d9d4066fac9819bef P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.033138 1169 ts_tablet_manager.cc:1434] T 80ee75935d7e461d9d4066fac9819bef P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251025 07:52:39.033421 1169 tablet_bootstrap.cc:492] T 22c97b3ee92140c68a7e1b93033cb299 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:39.036154 1169 tablet_bootstrap.cc:654] T 22c97b3ee92140c68a7e1b93033cb299 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.042234 1671 consensus_queue.cc:1048] T 6ec421eb820b4f40afcff5a5bf80c508 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:39.042532 1612 raft_consensus.cc:493] T df1248f6dbd94af68c587da7d149e279 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:39.042647 1612 raft_consensus.cc:515] T df1248f6dbd94af68c587da7d149e279 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:39.043223 1612 leader_election.cc:290] T df1248f6dbd94af68c587da7d149e279 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:39.043769 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df1248f6dbd94af68c587da7d149e279" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:39.044303 983 leader_election.cc:343] T df1248f6dbd94af68c587da7d149e279 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:39.044276 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df1248f6dbd94af68c587da7d149e279" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:39.044472 775 raft_consensus.cc:2468] T df1248f6dbd94af68c587da7d149e279 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:39.044898 982 leader_election.cc:304] T df1248f6dbd94af68c587da7d149e279 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters: 3aea2869ff814cc7b142012faa274382
I20251025 07:52:39.045367 1612 raft_consensus.cc:2804] T df1248f6dbd94af68c587da7d149e279 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:39.045493 1612 raft_consensus.cc:493] T df1248f6dbd94af68c587da7d149e279 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:39.045603 1612 raft_consensus.cc:3060] T df1248f6dbd94af68c587da7d149e279 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:39.046605 983 consensus_peers.cc:597] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:39.049050 1612 raft_consensus.cc:515] T df1248f6dbd94af68c587da7d149e279 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:39.050123 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df1248f6dbd94af68c587da7d149e279" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
W20251025 07:52:39.050792 983 leader_election.cc:343] T df1248f6dbd94af68c587da7d149e279 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:39.051769 1166 tablet_bootstrap.cc:492] T df1248f6dbd94af68c587da7d149e279 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:39.051898 1166 ts_tablet_manager.cc:1403] T df1248f6dbd94af68c587da7d149e279 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.149s user 0.026s sys 0.045s
I20251025 07:52:39.052651 1166 raft_consensus.cc:359] T df1248f6dbd94af68c587da7d149e279 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:39.052814 1166 raft_consensus.cc:385] T df1248f6dbd94af68c587da7d149e279 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.052866 1166 raft_consensus.cc:740] T df1248f6dbd94af68c587da7d149e279 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.053066 1166 consensus_queue.cc:260] T df1248f6dbd94af68c587da7d149e279 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:39.053670 1166 ts_tablet_manager.cc:1434] T df1248f6dbd94af68c587da7d149e279 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:39.053941 1166 tablet_bootstrap.cc:492] T 5fca357d7e24494fb6af4bd73d98833c P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:39.054348 1612 leader_election.cc:290] T df1248f6dbd94af68c587da7d149e279 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:39.055418 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df1248f6dbd94af68c587da7d149e279" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:39.056063 774 raft_consensus.cc:3060] T df1248f6dbd94af68c587da7d149e279 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:39.057587 1166 tablet_bootstrap.cc:654] T 5fca357d7e24494fb6af4bd73d98833c P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.065603 1166 tablet_bootstrap.cc:492] T 5fca357d7e24494fb6af4bd73d98833c P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:39.065795 1166 ts_tablet_manager.cc:1403] T 5fca357d7e24494fb6af4bd73d98833c P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20251025 07:52:39.066570 1166 raft_consensus.cc:359] T 5fca357d7e24494fb6af4bd73d98833c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:39.066764 1166 raft_consensus.cc:385] T 5fca357d7e24494fb6af4bd73d98833c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.066844 1166 raft_consensus.cc:740] T 5fca357d7e24494fb6af4bd73d98833c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.067095 1166 consensus_queue.cc:260] T 5fca357d7e24494fb6af4bd73d98833c P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:39.067770 1166 ts_tablet_manager.cc:1434] T 5fca357d7e24494fb6af4bd73d98833c P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:39.068092 1166 tablet_bootstrap.cc:492] T 5361f08f50b745eb98bf02e979d7c71e P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:39.069108 2169 ts_tablet_manager.cc:1929] T 2784bbfb97d5466f9ec856add7d0b29a P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:39.070329 2169 log.cc:1199] T 2784bbfb97d5466f9ec856add7d0b29a P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/2784bbfb97d5466f9ec856add7d0b29a
I20251025 07:52:39.072474 2105 raft_consensus.cc:493] T 50ae9bca6bed4f749d54191ac1543410 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:39.072710 2105 raft_consensus.cc:515] T 50ae9bca6bed4f749d54191ac1543410 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.073494 2105 leader_election.cc:290] T 50ae9bca6bed4f749d54191ac1543410 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:39.073905 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50ae9bca6bed4f749d54191ac1543410" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:39.074163 773 raft_consensus.cc:2468] T 50ae9bca6bed4f749d54191ac1543410 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:39.074282 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50ae9bca6bed4f749d54191ac1543410" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:39.074482 1044 raft_consensus.cc:2468] T 50ae9bca6bed4f749d54191ac1543410 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:39.074572 847 leader_election.cc:304] T 50ae9bca6bed4f749d54191ac1543410 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:39.074906 2105 raft_consensus.cc:2804] T 50ae9bca6bed4f749d54191ac1543410 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:39.074994 2105 raft_consensus.cc:493] T 50ae9bca6bed4f749d54191ac1543410 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:39.075071 2105 raft_consensus.cc:3060] T 50ae9bca6bed4f749d54191ac1543410 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:39.075480 2169 ts_tablet_manager.cc:1950] T 2784bbfb97d5466f9ec856add7d0b29a P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:39.077906 2105 raft_consensus.cc:515] T 50ae9bca6bed4f749d54191ac1543410 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.078817 2183 consensus_queue.cc:1048] T 6ec421eb820b4f40afcff5a5bf80c508 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:39.078917 2105 leader_election.cc:290] T 50ae9bca6bed4f749d54191ac1543410 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:39.079681 1166 tablet_bootstrap.cc:654] T 5361f08f50b745eb98bf02e979d7c71e P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.079736 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 2784bbfb97d5466f9ec856add7d0b29a (table default.Table_5aff13b5fcbc4899800760718fb3cc90 [id=ab3900039f5c491bbc1b96925deb1f1b]) successfully deleted
I20251025 07:52:39.079826 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50ae9bca6bed4f749d54191ac1543410" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:39.080045 773 raft_consensus.cc:3060] T 50ae9bca6bed4f749d54191ac1543410 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:39.080077 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50ae9bca6bed4f749d54191ac1543410" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:39.080282 1044 raft_consensus.cc:3060] T 50ae9bca6bed4f749d54191ac1543410 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:39.083380 773 raft_consensus.cc:2468] T 50ae9bca6bed4f749d54191ac1543410 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:39.084499 1166 tablet_bootstrap.cc:492] T 5361f08f50b745eb98bf02e979d7c71e P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:39.084669 1166 ts_tablet_manager.cc:1403] T 5361f08f50b745eb98bf02e979d7c71e P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.004s
I20251025 07:52:39.084966 1044 raft_consensus.cc:2468] T 50ae9bca6bed4f749d54191ac1543410 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:39.085429 850 leader_election.cc:304] T 50ae9bca6bed4f749d54191ac1543410 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:39.085498 1166 raft_consensus.cc:359] T 5361f08f50b745eb98bf02e979d7c71e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:39.085698 1166 raft_consensus.cc:385] T 5361f08f50b745eb98bf02e979d7c71e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.085752 2105 raft_consensus.cc:2804] T 50ae9bca6bed4f749d54191ac1543410 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:39.085783 1166 raft_consensus.cc:740] T 5361f08f50b745eb98bf02e979d7c71e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.085888 2105 raft_consensus.cc:697] T 50ae9bca6bed4f749d54191ac1543410 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:39.086199 2105 consensus_queue.cc:237] T 50ae9bca6bed4f749d54191ac1543410 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.086877 2169 tablet_replica.cc:333] T 1d261e432cf5403ea162129706bc4d4d P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:39.089138 1166 consensus_queue.cc:260] T 5361f08f50b745eb98bf02e979d7c71e P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:39.090173 1166 ts_tablet_manager.cc:1434] T 5361f08f50b745eb98bf02e979d7c71e P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251025 07:52:39.089681 2169 raft_consensus.cc:2243] T 1d261e432cf5403ea162129706bc4d4d P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:39.090878 1166 tablet_bootstrap.cc:492] T 80ee75935d7e461d9d4066fac9819bef P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:39.091152 2169 raft_consensus.cc:2272] T 1d261e432cf5403ea162129706bc4d4d P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:39.098735 1166 tablet_bootstrap.cc:654] T 80ee75935d7e461d9d4066fac9819bef P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:39.099912 883 consensus_meta.cc:287] T 3c29d2fe91e54aa9a40e50f38b710c36 P 3aea2869ff814cc7b142012faa274382: Time spent flushing consensus metadata: real 0.616s user 0.020s sys 0.000s
I20251025 07:52:39.100665 2169 ts_tablet_manager.cc:1916] T 1d261e432cf5403ea162129706bc4d4d P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:39.103529 1169 tablet_bootstrap.cc:492] T 22c97b3ee92140c68a7e1b93033cb299 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:39.105898 1169 ts_tablet_manager.cc:1403] T 22c97b3ee92140c68a7e1b93033cb299 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.072s user 0.004s sys 0.003s
I20251025 07:52:39.107540 2146 ts_tablet_manager.cc:1929] T 410ef150a7014c6c867a021b544a2f4e P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:39.107755 2146 log.cc:1199] T 410ef150a7014c6c867a021b544a2f4e P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/410ef150a7014c6c867a021b544a2f4e
I20251025 07:52:39.108466 2146 ts_tablet_manager.cc:1950] T 410ef150a7014c6c867a021b544a2f4e P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:39.109871 1044 raft_consensus.cc:1275] T b17d2a9ad6be49d29f2413666fba7e69 P db42be7ad3c741a0aa427f27e245993b [term 2 FOLLOWER]: Refusing update from remote peer 3aea2869ff814cc7b142012faa274382: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251025 07:52:39.110759 2245 consensus_queue.cc:1048] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:39.117465 1671 raft_consensus.cc:493] T 8f2c8582fed34c848d9903b8ad8b4057 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:39.117640 1671 raft_consensus.cc:515] T 8f2c8582fed34c848d9903b8ad8b4057 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:39.118552 1671 leader_election.cc:290] T 8f2c8582fed34c848d9903b8ad8b4057 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:39.122995 1166 tablet_bootstrap.cc:492] T 80ee75935d7e461d9d4066fac9819bef P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:39.123188 1166 ts_tablet_manager.cc:1403] T 80ee75935d7e461d9d4066fac9819bef P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.001s
I20251025 07:52:39.123936 1166 raft_consensus.cc:359] T 80ee75935d7e461d9d4066fac9819bef P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.124163 1166 raft_consensus.cc:385] T 80ee75935d7e461d9d4066fac9819bef P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.124253 1166 raft_consensus.cc:740] T 80ee75935d7e461d9d4066fac9819bef P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.124526 1166 consensus_queue.cc:260] T 80ee75935d7e461d9d4066fac9819bef P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.125162 1166 ts_tablet_manager.cc:1434] T 80ee75935d7e461d9d4066fac9819bef P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:39.130995 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f2c8582fed34c848d9903b8ad8b4057" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:39.132704 2146 tablet_replica.cc:333] T 2f26667f053a4f16b5da330c0b5e40f9 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:39.132633 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 410ef150a7014c6c867a021b544a2f4e (table default.Table_39bd22f18d6540d3be94520671683032 [id=baf017a8539143ad9c8a443a9e1b6339]) successfully deleted
W20251025 07:52:39.132942 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 3b940fa5ae26427dafbcfd26372754b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b940fa5ae26427dafbcfd26372754b3 already in progress: creating tablet
I20251025 07:52:39.133121 2146 raft_consensus.cc:2243] T 2f26667f053a4f16b5da330c0b5e40f9 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:39.133906 2146 raft_consensus.cc:2272] T 2f26667f053a4f16b5da330c0b5e40f9 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:39.134760 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 26c849d376d143c6bef0c8e0e1d7d03a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 26c849d376d143c6bef0c8e0e1d7d03a already in progress: creating tablet
W20251025 07:52:39.135013 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 88eb51447e0f4080baa49dd902deb24f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88eb51447e0f4080baa49dd902deb24f already in progress: creating tablet
W20251025 07:52:39.135223 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet b3a0af893a8947848e12093d0a7f0229 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3a0af893a8947848e12093d0a7f0229 already in progress: creating tablet
I20251025 07:52:39.135335 2146 ts_tablet_manager.cc:1916] T 2f26667f053a4f16b5da330c0b5e40f9 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:39.135430 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet f656945454fe4921ab819c772a089eb3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f656945454fe4921ab819c772a089eb3 already in progress: creating tablet
W20251025 07:52:39.135639 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet e54fe7cbcffa40c4b04144b47bee2b48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e54fe7cbcffa40c4b04144b47bee2b48 already in progress: creating tablet
I20251025 07:52:39.128728 1169 raft_consensus.cc:359] T 22c97b3ee92140c68a7e1b93033cb299 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.086696 2166 ts_tablet_manager.cc:1950] T 2f26667f053a4f16b5da330c0b5e40f9 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:39.111490 1659 raft_consensus.cc:493] T 292fe04055954afd8c08ac2636b68191 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:39.142416 1659 raft_consensus.cc:515] T 292fe04055954afd8c08ac2636b68191 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:39.143198 1659 leader_election.cc:290] T 292fe04055954afd8c08ac2636b68191 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:39.119074 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f2c8582fed34c848d9903b8ad8b4057" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:39.124301 1167 tablet_bootstrap.cc:654] T 80fe3255ed614270bd8352a2853fbc43 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.136569 776 raft_consensus.cc:1275] T b17d2a9ad6be49d29f2413666fba7e69 P 23eeffd3de254ce18db5febfde971c6b [term 2 FOLLOWER]: Refusing update from remote peer 3aea2869ff814cc7b142012faa274382: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251025 07:52:39.143460 775 raft_consensus.cc:2393] T 8f2c8582fed34c848d9903b8ad8b4057 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:39.144272 982 leader_election.cc:304] T 8f2c8582fed34c848d9903b8ad8b4057 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:39.144759 1671 raft_consensus.cc:3060] T 8f2c8582fed34c848d9903b8ad8b4057 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:39.146584 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 2f26667f053a4f16b5da330c0b5e40f9 (table default.Table_4b3d7b3a731942189cdc214938a45820 [id=88f1e8b1c5fa4913a5f7b5b2c6cbf6ec]) successfully deleted
I20251025 07:52:39.147264 2166 tablet_replica.cc:333] T b947ecc352e54f4caa73c6ca97372256 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:39.147527 2166 raft_consensus.cc:2243] T b947ecc352e54f4caa73c6ca97372256 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:39.147749 2166 raft_consensus.cc:2272] T b947ecc352e54f4caa73c6ca97372256 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:39.148510 2166 ts_tablet_manager.cc:1916] T b947ecc352e54f4caa73c6ca97372256 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:39.150552 774 raft_consensus.cc:2468] T df1248f6dbd94af68c587da7d149e279 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
W20251025 07:52:39.151062 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 050a5d3b5f934dd19bf2e221f2fe0780 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 050a5d3b5f934dd19bf2e221f2fe0780 already in progress: creating tablet
I20251025 07:52:39.153609 454 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet 3c4970c8a491460e8b4b1c29fde83e4c
I20251025 07:52:39.154465 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c4970c8a491460e8b4b1c29fde83e4c with delete_type TABLET_DATA_DELETED (replaced by tablet 783487b60c144f719ed2eb3c67b5ccab) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:39.154574 454 catalog_manager.cc:6719] ReplaceTablet: tablet 3c4970c8a491460e8b4b1c29fde83e4c deleted and replaced by tablet 783487b60c144f719ed2eb3c67b5ccab
I20251025 07:52:39.154698 454 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:47610 (request call id 42) took 2093 ms. Trace:
I20251025 07:52:39.155006 1167 tablet_bootstrap.cc:492] T 80fe3255ed614270bd8352a2853fbc43 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:39.155157 1167 ts_tablet_manager.cc:1403] T 80fe3255ed614270bd8352a2853fbc43 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.137s user 0.003s sys 0.003s
I20251025 07:52:39.154811 454 rpcz_store.cc:276] 1025 07:52:37.061231 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.380368 (+1319137us) service_pool.cc:225] Handling call
1025 07:52:39.154661 (+774293us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:38.922113 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:38.922960 (+ 847us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:38.922966 (+ 6us) write_op.cc:435] Acquired schema lock
1025 07:52:38.922968 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:38.923072 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:38.923082 (+ 10us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:38.923086 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:38.923146 (+ 60us) tablet.cc:699] Row locks acquired
1025 07:52:38.923148 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:38.923198 (+ 50us) write_op.cc:270] Start()
1025 07:52:38.923226 (+ 28us) write_op.cc:276] Timestamp: P: 1761378758923162 usec, L: 0
1025 07:52:38.923229 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:38.923366 (+ 137us) log.cc:844] Serialized 622 byte log entry
1025 07:52:38.933592 (+ 10226us) op_driver.cc:464] REPLICATION: finished
1025 07:52:38.933703 (+ 111us) write_op.cc:301] APPLY: starting
1025 07:52:38.933717 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:38.933756 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:38.933789 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:38.933877 (+ 88us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:38.933921 (+ 44us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:38.933928 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:38.933989 (+ 61us) log.cc:844] Serialized 29 byte log entry
1025 07:52:38.934039 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.934057 (+ 18us) write_op.cc:454] Released schema lock
1025 07:52:38.934063 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:38.934122 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.934126 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":64,"apply.run_cpu_time_us":411,"apply.run_wall_time_us":411,"num_ops":2,"prepare.queue_time_us":1178,"prepare.run_cpu_time_us":592,"prepare.run_wall_time_us":1306,"replication_time_us":10333}]]}
I20251025 07:52:39.157523 1167 raft_consensus.cc:359] T 80fe3255ed614270bd8352a2853fbc43 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.157734 1167 raft_consensus.cc:385] T 80fe3255ed614270bd8352a2853fbc43 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.157799 1167 raft_consensus.cc:740] T 80fe3255ed614270bd8352a2853fbc43 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.158020 1167 consensus_queue.cc:260] T 80fe3255ed614270bd8352a2853fbc43 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.158677 1167 ts_tablet_manager.cc:1434] T 80fe3255ed614270bd8352a2853fbc43 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251025 07:52:39.158958 1167 tablet_bootstrap.cc:492] T 724d0b7e1b65445a9a1f056852983bdb P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:39.158926 454 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_29aeaf089ca24270b3f21c2e08d61634" } modify_external_catalogs: true
I20251025 07:52:39.159168 454 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_29aeaf089ca24270b3f21c2e08d61634" } modify_external_catalogs: true
I20251025 07:52:39.160981 2166 ts_tablet_manager.cc:1929] T b947ecc352e54f4caa73c6ca97372256 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:39.161140 2166 log.cc:1199] T b947ecc352e54f4caa73c6ca97372256 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/b947ecc352e54f4caa73c6ca97372256
I20251025 07:52:39.161701 2166 ts_tablet_manager.cc:1950] T b947ecc352e54f4caa73c6ca97372256 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:39.163113 1167 tablet_bootstrap.cc:654] T 724d0b7e1b65445a9a1f056852983bdb P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.164086 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet b947ecc352e54f4caa73c6ca97372256 (table default.Table_503b86ee2ab545afa9829bebcc9ea11d [id=395c91146476423dad0455a12eb5a8af]) successfully deleted
I20251025 07:52:39.164888 2166 tablet_replica.cc:333] T 80ee75935d7e461d9d4066fac9819bef P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:39.165135 2166 raft_consensus.cc:2243] T 80ee75935d7e461d9d4066fac9819bef P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:39.165344 2166 raft_consensus.cc:2272] T 80ee75935d7e461d9d4066fac9819bef P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:39.166579 2166 ts_tablet_manager.cc:1916] T 80ee75935d7e461d9d4066fac9819bef P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:39.166801 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 48ff4ce7846b4e56a37eee9bcb17bd5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48ff4ce7846b4e56a37eee9bcb17bd5e already in progress: creating tablet
W20251025 07:52:39.167068 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet d9bfd7c0c3ad4e5b98b6708793d61d41 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9bfd7c0c3ad4e5b98b6708793d61d41 already in progress: creating tablet
W20251025 07:52:39.167449 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet a4e4b99be0434baaa75d988281797846 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a4e4b99be0434baaa75d988281797846 already in progress: creating tablet
I20251025 07:52:39.168229 1659 raft_consensus.cc:493] T e9f7cd765d614d59a7c190b744b0844d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:39.168396 1659 raft_consensus.cc:515] T e9f7cd765d614d59a7c190b744b0844d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.169118 1659 leader_election.cc:290] T e9f7cd765d614d59a7c190b744b0844d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:39.169183 1167 tablet_bootstrap.cc:492] T 724d0b7e1b65445a9a1f056852983bdb P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:39.169391 1167 ts_tablet_manager.cc:1403] T 724d0b7e1b65445a9a1f056852983bdb P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.001s
I20251025 07:52:39.170214 1167 raft_consensus.cc:359] T 724d0b7e1b65445a9a1f056852983bdb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:39.170404 1167 raft_consensus.cc:385] T 724d0b7e1b65445a9a1f056852983bdb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.170502 1167 raft_consensus.cc:740] T 724d0b7e1b65445a9a1f056852983bdb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.170732 1167 consensus_queue.cc:260] T 724d0b7e1b65445a9a1f056852983bdb P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:39.170948 453 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet 36c8fcafd5464b4e999276a8062a812a
I20251025 07:52:39.171329 1167 ts_tablet_manager.cc:1434] T 724d0b7e1b65445a9a1f056852983bdb P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251025 07:52:39.171564 453 catalog_manager.cc:6719] ReplaceTablet: tablet 36c8fcafd5464b4e999276a8062a812a deleted and replaced by tablet 181059cc79d54999bf58504c85b739e9
I20251025 07:52:39.171671 1167 tablet_bootstrap.cc:492] T 943ec703e65f4fd3b0c690e08c0bd226 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:39.171687 453 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:47610 (request call id 50) took 1985 ms. Trace:
I20251025 07:52:39.171792 453 rpcz_store.cc:276] 1025 07:52:37.186576 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.533842 (+1347266us) service_pool.cc:225] Handling call
1025 07:52:39.171647 (+637805us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:38.921346 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:38.921486 (+ 140us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:38.921493 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:38.921496 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:38.921615 (+ 119us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:38.921632 (+ 17us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:38.921639 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:38.921724 (+ 85us) tablet.cc:699] Row locks acquired
1025 07:52:38.921727 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:38.921779 (+ 52us) write_op.cc:270] Start()
1025 07:52:38.921812 (+ 33us) write_op.cc:276] Timestamp: P: 1761378758921762 usec, L: 0
1025 07:52:38.921816 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:38.921991 (+ 175us) log.cc:844] Serialized 622 byte log entry
1025 07:52:38.929038 (+ 7047us) op_driver.cc:464] REPLICATION: finished
1025 07:52:38.929141 (+ 103us) write_op.cc:301] APPLY: starting
1025 07:52:38.929156 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:38.929200 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:38.929226 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:38.929333 (+ 107us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:38.929405 (+ 72us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:38.929414 (+ 9us) write_op.cc:312] APPLY: finished
1025 07:52:38.929485 (+ 71us) log.cc:844] Serialized 29 byte log entry
1025 07:52:38.929542 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.929562 (+ 20us) write_op.cc:454] Released schema lock
1025 07:52:38.929569 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:38.929637 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.929641 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":56,"apply.run_cpu_time_us":487,"apply.run_wall_time_us":487,"num_ops":2,"prepare.queue_time_us":1930,"prepare.run_cpu_time_us":744,"prepare.run_wall_time_us":745,"raft.queue_time_us":574,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":218,"replication_time_us":7191}]]}
I20251025 07:52:39.172600 453 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_6caa55a17907446d81c7f0656b329cdd" } modify_external_catalogs: true
I20251025 07:52:39.176347 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 36c8fcafd5464b4e999276a8062a812a with delete_type TABLET_DATA_DELETED (replaced by tablet 181059cc79d54999bf58504c85b739e9) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:39.179345 1167 tablet_bootstrap.cc:654] T 943ec703e65f4fd3b0c690e08c0bd226 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.180963 2166 ts_tablet_manager.cc:1929] T 80ee75935d7e461d9d4066fac9819bef P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:39.181393 2166 log.cc:1199] T 80ee75935d7e461d9d4066fac9819bef P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/80ee75935d7e461d9d4066fac9819bef
I20251025 07:52:39.182006 2166 ts_tablet_manager.cc:1950] T 80ee75935d7e461d9d4066fac9819bef P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:39.182283 1169 raft_consensus.cc:385] T 22c97b3ee92140c68a7e1b93033cb299 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.182459 1169 raft_consensus.cc:740] T 22c97b3ee92140c68a7e1b93033cb299 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.182729 1169 consensus_queue.cc:260] T 22c97b3ee92140c68a7e1b93033cb299 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.183017 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 80ee75935d7e461d9d4066fac9819bef (table default.Table_bb258691f8844d85ab37eb73e9a4c314 [id=c80bab0ed06a459eb78b1d94df13f337]) successfully deleted
I20251025 07:52:39.183315 1169 ts_tablet_manager.cc:1434] T 22c97b3ee92140c68a7e1b93033cb299 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.076s user 0.000s sys 0.003s
I20251025 07:52:39.183615 2166 tablet_replica.cc:333] T b94634ac5077439fadfbbce569f60e8b P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:39.183547 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "292fe04055954afd8c08ac2636b68191" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:39.183771 2166 raft_consensus.cc:2243] T b94634ac5077439fadfbbce569f60e8b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:39.183907 2166 raft_consensus.cc:2272] T b94634ac5077439fadfbbce569f60e8b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:39.184618 712 leader_election.cc:343] T 292fe04055954afd8c08ac2636b68191 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:39.185611 2166 ts_tablet_manager.cc:1916] T b94634ac5077439fadfbbce569f60e8b P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:39.186211 982 leader_election.cc:304] T df1248f6dbd94af68c587da7d149e279 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters: 3aea2869ff814cc7b142012faa274382
I20251025 07:52:39.186869 2050 raft_consensus.cc:2804] T df1248f6dbd94af68c587da7d149e279 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:39.187770 2169 ts_tablet_manager.cc:1929] T 1d261e432cf5403ea162129706bc4d4d P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:39.188051 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c4970c8a491460e8b4b1c29fde83e4c with delete_type TABLET_DATA_DELETED (replaced by tablet 783487b60c144f719ed2eb3c67b5ccab) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:39.190212 1671 raft_consensus.cc:2749] T 8f2c8582fed34c848d9903b8ad8b4057 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:39.190752 2050 raft_consensus.cc:697] T df1248f6dbd94af68c587da7d149e279 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:39.191088 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9f7cd765d614d59a7c190b744b0844d" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:39.191521 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 36c8fcafd5464b4e999276a8062a812a with delete_type TABLET_DATA_DELETED (replaced by tablet 181059cc79d54999bf58504c85b739e9) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:39.191895 1169 tablet_bootstrap.cc:492] T 6a58931e7cd7454fb583f1c70cd2ad15 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:39.192767 2169 log.cc:1199] T 1d261e432cf5403ea162129706bc4d4d P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/1d261e432cf5403ea162129706bc4d4d
I20251025 07:52:39.194492 1169 tablet_bootstrap.cc:654] T 6a58931e7cd7454fb583f1c70cd2ad15 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:39.194526 712 leader_election.cc:343] T e9f7cd765d614d59a7c190b744b0844d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:39.193254 2050 consensus_queue.cc:237] T df1248f6dbd94af68c587da7d149e279 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:39.195672 2169 ts_tablet_manager.cc:1950] T 1d261e432cf5403ea162129706bc4d4d P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:39.196568 2169 tablet_replica.cc:333] T 467324e2432f4caca148951329caa292 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:39.196763 2169 raft_consensus.cc:2243] T 467324e2432f4caca148951329caa292 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:39.196910 2169 raft_consensus.cc:2272] T 467324e2432f4caca148951329caa292 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:39.197404 2169 ts_tablet_manager.cc:1916] T 467324e2432f4caca148951329caa292 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:39.142937 1166 tablet_bootstrap.cc:492] T 22c97b3ee92140c68a7e1b93033cb299 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:39.201356 1166 tablet_bootstrap.cc:654] T 22c97b3ee92140c68a7e1b93033cb299 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.201414 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 1d261e432cf5403ea162129706bc4d4d (table default.Table_590cc281d8b042b19ef12c7f6436752a [id=b1eee5c0243a48c4bbdecc0b4f38ab2d]) successfully deleted
I20251025 07:52:39.143945 2106 consensus_queue.cc:1048] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:39.144208 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "292fe04055954afd8c08ac2636b68191" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:39.155181 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c4970c8a491460e8b4b1c29fde83e4c with delete_type TABLET_DATA_DELETED (replaced by tablet 783487b60c144f719ed2eb3c67b5ccab) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:39.169975 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9f7cd765d614d59a7c190b744b0844d" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:39.175468 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 36c8fcafd5464b4e999276a8062a812a with delete_type TABLET_DATA_DELETED (replaced by tablet 181059cc79d54999bf58504c85b739e9) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:39.175772 2245 raft_consensus.cc:493] T 5fca357d7e24494fb6af4bd73d98833c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:39.202679 2245 raft_consensus.cc:515] T 5fca357d7e24494fb6af4bd73d98833c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:39.203298 2245 leader_election.cc:290] T 5fca357d7e24494fb6af4bd73d98833c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:39.203473 1167 tablet_bootstrap.cc:492] T 943ec703e65f4fd3b0c690e08c0bd226 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:39.203596 1167 ts_tablet_manager.cc:1403] T 943ec703e65f4fd3b0c690e08c0bd226 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.004s
I20251025 07:52:39.204475 1167 raft_consensus.cc:359] T 943ec703e65f4fd3b0c690e08c0bd226 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.204632 1167 raft_consensus.cc:385] T 943ec703e65f4fd3b0c690e08c0bd226 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.204685 1167 raft_consensus.cc:740] T 943ec703e65f4fd3b0c690e08c0bd226 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.204635 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5fca357d7e24494fb6af4bd73d98833c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:39.205142 710 leader_election.cc:343] T 292fe04055954afd8c08ac2636b68191 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:39.205250 710 leader_election.cc:304] T 292fe04055954afd8c08ac2636b68191 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:39.205657 710 leader_election.cc:343] T e9f7cd765d614d59a7c190b744b0844d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:39.205731 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5fca357d7e24494fb6af4bd73d98833c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:39.205796 1659 raft_consensus.cc:2749] T 292fe04055954afd8c08ac2636b68191 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:39.204900 1167 consensus_queue.cc:260] T 943ec703e65f4fd3b0c690e08c0bd226 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.206643 1167 ts_tablet_manager.cc:1434] T 943ec703e65f4fd3b0c690e08c0bd226 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251025 07:52:39.206931 1167 tablet_bootstrap.cc:492] T f97acc9a638b4d5a968dcc3412a384bc P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:39.207356 847 leader_election.cc:304] T 5fca357d7e24494fb6af4bd73d98833c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:39.205741 710 leader_election.cc:304] T e9f7cd765d614d59a7c190b744b0844d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:39.208623 1659 raft_consensus.cc:2749] T e9f7cd765d614d59a7c190b744b0844d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:39.208834 2245 raft_consensus.cc:2749] T 5fca357d7e24494fb6af4bd73d98833c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:39.210064 1166 tablet_bootstrap.cc:492] T 22c97b3ee92140c68a7e1b93033cb299 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:39.210291 1166 ts_tablet_manager.cc:1403] T 22c97b3ee92140c68a7e1b93033cb299 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.067s user 0.001s sys 0.005s
I20251025 07:52:39.211163 1166 raft_consensus.cc:359] T 22c97b3ee92140c68a7e1b93033cb299 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.211421 1166 raft_consensus.cc:385] T 22c97b3ee92140c68a7e1b93033cb299 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.211519 1166 raft_consensus.cc:740] T 22c97b3ee92140c68a7e1b93033cb299 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.211743 1167 tablet_bootstrap.cc:654] T f97acc9a638b4d5a968dcc3412a384bc P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.211891 1166 consensus_queue.cc:260] T 22c97b3ee92140c68a7e1b93033cb299 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.212632 1166 ts_tablet_manager.cc:1434] T 22c97b3ee92140c68a7e1b93033cb299 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:39.215019 1166 tablet_bootstrap.cc:492] T 92aa605fabbc4257a6a204c087e7cc7f P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:39.217176 1166 tablet_bootstrap.cc:654] T 92aa605fabbc4257a6a204c087e7cc7f P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.233386 1659 raft_consensus.cc:493] T aa9d9ab8073945d28d52fb02213f201a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:39.233605 1659 raft_consensus.cc:515] T aa9d9ab8073945d28d52fb02213f201a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.234370 1659 leader_election.cc:290] T aa9d9ab8073945d28d52fb02213f201a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:39.236212 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa9d9ab8073945d28d52fb02213f201a" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:39.236392 1167 tablet_bootstrap.cc:492] T f97acc9a638b4d5a968dcc3412a384bc P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:39.236513 1167 ts_tablet_manager.cc:1403] T f97acc9a638b4d5a968dcc3412a384bc P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.001s
I20251025 07:52:39.237352 1166 tablet_bootstrap.cc:492] T 92aa605fabbc4257a6a204c087e7cc7f P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
W20251025 07:52:39.237331 710 leader_election.cc:343] T aa9d9ab8073945d28d52fb02213f201a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:39.237360 1167 raft_consensus.cc:359] T f97acc9a638b4d5a968dcc3412a384bc P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.237520 1167 raft_consensus.cc:385] T f97acc9a638b4d5a968dcc3412a384bc P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.237584 1167 raft_consensus.cc:740] T f97acc9a638b4d5a968dcc3412a384bc P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.237782 1166 ts_tablet_manager.cc:1403] T 92aa605fabbc4257a6a204c087e7cc7f P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.004s
I20251025 07:52:39.237761 1167 consensus_queue.cc:260] T f97acc9a638b4d5a968dcc3412a384bc P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.238245 453 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_6caa55a17907446d81c7f0656b329cdd" } modify_external_catalogs: true
I20251025 07:52:39.238382 1167 ts_tablet_manager.cc:1434] T f97acc9a638b4d5a968dcc3412a384bc P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:39.238611 1166 raft_consensus.cc:359] T 92aa605fabbc4257a6a204c087e7cc7f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.238727 1167 tablet_bootstrap.cc:492] T c2713942584e4734acbc0775ead772fa P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:39.238811 1166 raft_consensus.cc:385] T 92aa605fabbc4257a6a204c087e7cc7f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.238909 1166 raft_consensus.cc:740] T 92aa605fabbc4257a6a204c087e7cc7f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.239128 1166 consensus_queue.cc:260] T 92aa605fabbc4257a6a204c087e7cc7f P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.240540 1167 tablet_bootstrap.cc:654] T c2713942584e4734acbc0775ead772fa P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.242180 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa9d9ab8073945d28d52fb02213f201a" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:39.242796 983 consensus_peers.cc:597] T d37758f606344127b3db74c87babe69c P db42be7ad3c741a0aa427f27e245993b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d37758f606344127b3db74c87babe69c. This is attempt 1: this message will repeat every 5th retry.
W20251025 07:52:39.243083 712 leader_election.cc:343] T aa9d9ab8073945d28d52fb02213f201a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:39.243199 712 leader_election.cc:304] T aa9d9ab8073945d28d52fb02213f201a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:39.243666 1166 ts_tablet_manager.cc:1434] T 92aa605fabbc4257a6a204c087e7cc7f P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20251025 07:52:39.243975 1166 tablet_bootstrap.cc:492] T 74e2d78c024849b0af5c0bde1f48c36f P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:39.244168 1659 raft_consensus.cc:2749] T aa9d9ab8073945d28d52fb02213f201a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:39.248271 902 raft_consensus.cc:3060] T 3abfd52327a04ba88625f391a8658fbb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:39.249354 1659 raft_consensus.cc:493] T 7c7a9d7a4fb3449bb57eafa83230a839 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:39.249475 1659 raft_consensus.cc:515] T 7c7a9d7a4fb3449bb57eafa83230a839 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.250046 1659 leader_election.cc:290] T 7c7a9d7a4fb3449bb57eafa83230a839 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:39.250700 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c7a9d7a4fb3449bb57eafa83230a839" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:39.251520 712 leader_election.cc:343] T 7c7a9d7a4fb3449bb57eafa83230a839 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:39.252038 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c7a9d7a4fb3449bb57eafa83230a839" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:39.252524 710 leader_election.cc:343] T 7c7a9d7a4fb3449bb57eafa83230a839 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:39.252624 710 leader_election.cc:304] T 7c7a9d7a4fb3449bb57eafa83230a839 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:39.252799 2050 consensus_queue.cc:1048] T 3abfd52327a04ba88625f391a8658fbb P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:39.253073 1659 raft_consensus.cc:2749] T 7c7a9d7a4fb3449bb57eafa83230a839 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:39.259104 1166 tablet_bootstrap.cc:654] T 74e2d78c024849b0af5c0bde1f48c36f P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.265805 2166 ts_tablet_manager.cc:1929] T b94634ac5077439fadfbbce569f60e8b P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:39.266083 2166 log.cc:1199] T b94634ac5077439fadfbbce569f60e8b P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/b94634ac5077439fadfbbce569f60e8b
I20251025 07:52:39.266497 2166 ts_tablet_manager.cc:1950] T b94634ac5077439fadfbbce569f60e8b P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:39.267134 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet b94634ac5077439fadfbbce569f60e8b (table default.Table_1ff54f4df7a2444489ae4e4060084156 [id=b88c37a678344c6b8e8017553e486cb2]) successfully deleted
W20251025 07:52:39.267937 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 92aa605fabbc4257a6a204c087e7cc7f because the tablet was not found. No further retry: Not found: Tablet not found: 92aa605fabbc4257a6a204c087e7cc7f
I20251025 07:52:39.268289 2166 tablet_replica.cc:333] T 50ae9bca6bed4f749d54191ac1543410 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:39.268460 2159 log.cc:826] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Log is configured to *not* fsync() on all Append() calls
I20251025 07:52:39.269932 2146 ts_tablet_manager.cc:1929] T 2f26667f053a4f16b5da330c0b5e40f9 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:39.270203 2146 log.cc:1199] T 2f26667f053a4f16b5da330c0b5e40f9 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/2f26667f053a4f16b5da330c0b5e40f9
I20251025 07:52:39.270881 2146 ts_tablet_manager.cc:1950] T 2f26667f053a4f16b5da330c0b5e40f9 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:39.271898 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 2f26667f053a4f16b5da330c0b5e40f9 (table default.Table_4b3d7b3a731942189cdc214938a45820 [id=88f1e8b1c5fa4913a5f7b5b2c6cbf6ec]) successfully deleted
W20251025 07:52:39.272943 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet f661d5909657455581dcb5e6eadd3c0b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f661d5909657455581dcb5e6eadd3c0b already in progress: creating tablet
I20251025 07:52:39.274147 1167 tablet_bootstrap.cc:492] T c2713942584e4734acbc0775ead772fa P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:39.274296 1167 ts_tablet_manager.cc:1403] T c2713942584e4734acbc0775ead772fa P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.000s
I20251025 07:52:39.276326 2245 consensus_queue.cc:1048] T 8f2c8582fed34c848d9903b8ad8b4057 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:39.277881 1167 raft_consensus.cc:359] T c2713942584e4734acbc0775ead772fa P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:39.278203 2146 tablet_replica.cc:333] T fa77085e81c7440d9e255c44d1e80c26 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:39.278192 1167 raft_consensus.cc:385] T c2713942584e4734acbc0775ead772fa P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.278443 1167 raft_consensus.cc:740] T c2713942584e4734acbc0775ead772fa P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.278659 2146 raft_consensus.cc:2243] T fa77085e81c7440d9e255c44d1e80c26 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:39.253666 2169 ts_tablet_manager.cc:1929] T 467324e2432f4caca148951329caa292 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:39.260984 1169 tablet_bootstrap.cc:492] T 6a58931e7cd7454fb583f1c70cd2ad15 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:39.278836 1167 consensus_queue.cc:260] T c2713942584e4734acbc0775ead772fa P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:39.279085 1169 ts_tablet_manager.cc:1403] T 6a58931e7cd7454fb583f1c70cd2ad15 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.088s user 0.009s sys 0.001s
I20251025 07:52:39.279594 1167 ts_tablet_manager.cc:1434] T c2713942584e4734acbc0775ead772fa P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20251025 07:52:39.279871 1167 tablet_bootstrap.cc:492] T f43ef51ee45d47a99ec732b49d50d3b4 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:39.280048 1169 raft_consensus.cc:359] T 6a58931e7cd7454fb583f1c70cd2ad15 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.280282 1169 raft_consensus.cc:385] T 6a58931e7cd7454fb583f1c70cd2ad15 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.280385 1169 raft_consensus.cc:740] T 6a58931e7cd7454fb583f1c70cd2ad15 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.280507 2050 consensus_queue.cc:1048] T 3abfd52327a04ba88625f391a8658fbb P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:39.280632 1169 consensus_queue.cc:260] T 6a58931e7cd7454fb583f1c70cd2ad15 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.281240 1169 ts_tablet_manager.cc:1434] T 6a58931e7cd7454fb583f1c70cd2ad15 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:39.281553 1169 tablet_bootstrap.cc:492] T 92aa605fabbc4257a6a204c087e7cc7f P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:39.281579 1167 tablet_bootstrap.cc:654] T f43ef51ee45d47a99ec732b49d50d3b4 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.283883 1169 tablet_bootstrap.cc:654] T 92aa605fabbc4257a6a204c087e7cc7f P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.284991 2169 log.cc:1199] T 467324e2432f4caca148951329caa292 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/467324e2432f4caca148951329caa292
I20251025 07:52:39.285739 2169 ts_tablet_manager.cc:1950] T 467324e2432f4caca148951329caa292 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:39.296183 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_a6c869949b744bc98efb735b0e73e7aa [id=bf77b5ff6f1048dd996d948437f9ff5e]). Expected version 2 got 0
I20251025 07:52:39.296507 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3]). Expected version 1 got 0
I20251025 07:52:39.296762 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5]). Expected version 1 got 0
I20251025 07:52:39.298197 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f]). Expected version 1 got 0
I20251025 07:52:39.299026 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7]). Expected version 1 got 0
I20251025 07:52:39.299737 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 04d27d54fdb7483b869408af65ff85c9 (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]). Expected version 2 got 0
I20251025 07:52:39.300354 1169 tablet_bootstrap.cc:492] T 92aa605fabbc4257a6a204c087e7cc7f P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:39.300556 1169 ts_tablet_manager.cc:1403] T 92aa605fabbc4257a6a204c087e7cc7f P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20251025 07:52:39.300855 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 4c155aa7fa5b4d68bbab1918b8432372 (table default.Table_6c5a24747ba14352b0285a41ec2985a4 [id=28bbaad474164a8c8c97a76bb8081791]). Expected version 2 got 0
I20251025 07:52:39.301263 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet cb829649f73d4d3f8862d268c953bd86 (table default.Table_ba4b8cc7b930482690b7e192e0edff12 [id=7944225d3ce7439b881bfb9289d9f040]). Expected version 1 got 0
I20251025 07:52:39.301568 1169 raft_consensus.cc:359] T 92aa605fabbc4257a6a204c087e7cc7f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.301829 1169 raft_consensus.cc:385] T 92aa605fabbc4257a6a204c087e7cc7f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.301929 1169 raft_consensus.cc:740] T 92aa605fabbc4257a6a204c087e7cc7f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.302273 1169 consensus_queue.cc:260] T 92aa605fabbc4257a6a204c087e7cc7f P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.303005 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 292fe04055954afd8c08ac2636b68191 (table default.Table_9fa69b10b5e842639fc0b0a58cd9b6a5 [id=a253ce0253bd4da4814741c482193d8f]). Expected version 1 got 0
I20251025 07:52:39.303452 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 467324e2432f4caca148951329caa292 (table default.Table_4cf617a8064c4be99402936ea4cf514b [id=11ce21ee2d434ce38aadc6152c02e2bb]) successfully deleted
I20251025 07:52:39.303498 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet aa9d9ab8073945d28d52fb02213f201a (table default.Table_4791672658f043f6b97b9320a531ebb5 [id=dfd20cf9c3e541c4abe23d59e0aa7f2a]). Expected version 1 got 0
I20251025 07:52:39.303524 1169 ts_tablet_manager.cc:1434] T 92aa605fabbc4257a6a204c087e7cc7f P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251025 07:52:39.303748 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet b17d2a9ad6be49d29f2413666fba7e69 (table default.Table_fcddb31989ee4236a4981c12779b3f5d [id=f3794fa081364666b944c412415bdba0]). Expected version 3 got 0
I20251025 07:52:39.303907 2169 tablet_replica.cc:333] T 8c76d1b509f04ab1a10b83cff675c11e P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:39.304162 2169 raft_consensus.cc:2243] T 8c76d1b509f04ab1a10b83cff675c11e P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:39.304569 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 22f7db660616457b83b24e947a0b0bae (table default.Table_4a86191163f2408f9bcc795cd8d1bbf6 [id=bb316a7e000748ca902a9bfb494802d7]). Expected version 1 got 0
I20251025 07:52:39.304808 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84]). Expected version 1 got 0
I20251025 07:52:39.304288 1169 tablet_bootstrap.cc:492] T 74e2d78c024849b0af5c0bde1f48c36f P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:39.305090 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 5dab439bc50d4e869548154e1d222ad6 (table default.Table_bd82d871f84943f4baa967f5c0ec74a3 [id=45325c188eac4d34bb016d01de091f87]). Expected version 2 got 0
I20251025 07:52:39.305387 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_87d38832f85e4965bfab1a077990114f [id=ad812862698d421184ebf501ce79b7be]). Expected version 2 got 0
I20251025 07:52:39.304373 2169 raft_consensus.cc:2272] T 8c76d1b509f04ab1a10b83cff675c11e P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:39.307060 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]). Expected version 1 got 0
I20251025 07:52:39.307505 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 0325d462441349ef81841f1b0cab2a20 (table default.Table_750ecb905839434fa9a1aa72f87394be [id=9ce3ba77c56a4ef09cb3b40ea0e488f6]). Expected version 1 got 0
I20251025 07:52:39.307505 2169 ts_tablet_manager.cc:1916] T 8c76d1b509f04ab1a10b83cff675c11e P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:39.308116 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet f70091024c034c118fb86578db19a31c (table default.Table_332b308220f64c6dbb5f29711d8463b3 [id=e041946c093b4c789ce175c736cc230b]). Expected version 1 got 0
I20251025 07:52:39.308526 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3]). Expected version 2 got 0
I20251025 07:52:39.308837 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 830fdbc6812e416690d0db42a73af503 (table default.Table_92824b3f2a774564b3aea04aaa25d204 [id=fb21a45b86ab446e8c3dae4512a9598f]). Expected version 1 got 0
I20251025 07:52:39.309068 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 755f0d6c87044e20a8020f8410940123 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 4 got 0
I20251025 07:52:39.309587 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_e90d7f5ff13b40878e5b625c97792c7d [id=fc8180c09d7b4e74b2e161f7acb8fe0e]). Expected version 1 got 0
I20251025 07:52:39.310523 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236]). Expected version 1 got 0
I20251025 07:52:39.310791 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c]). Expected version 1 got 0
I20251025 07:52:39.311260 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]). Expected version 3 got 0
I20251025 07:52:39.311633 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 87268e27540c4ec58ce1e62df819f19d (table default.Table_350383f0e98947eba77186e74ad88781 [id=96b0e47c666c4d8f982741aea7b17ae7]). Expected version 2 got 0
I20251025 07:52:39.312088 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 8f2c8582fed34c848d9903b8ad8b4057 (table default.Table_438f67f3dc62465999a917986922b3ed [id=7c7fcd265e624d7fab74661db505d9c3]). Expected version 1 got 0
I20251025 07:52:39.312381 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525]). Expected version 1 got 0
I20251025 07:52:39.312656 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 36262ab8c6964dd684c01b4db6219a4f (table default.Table_76158f8cf82e4cdb9a90580e46aaea48 [id=f249f30497bc450c887d659721469de8]). Expected version 1 got 0
I20251025 07:52:39.313261 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 2c50ec95e48e49b59a76fac4fc862d3d (table default.Table_32b54e82cbc84d39a0020826f8737d62 [id=c28b0b6acd61490caecfdaab803a9d92]). Expected version 2 got 0
I20251025 07:52:39.313587 450 catalog_manager.cc:5649] T 07898860aa594a05a211fc89768cb76d P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } }
I20251025 07:52:39.314438 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet b97e39aaac3048ed93dbfe873b757254 (table default.Table_99fd36369f9a4e0fb14f234c98635203 [id=c8059ab6cec146efa631d06799d969a9]). Expected version 1 got 0
I20251025 07:52:39.314723 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 8e6d76b1a97945cfb6c3055e1b622f8a (table default.Table_10e57b42f97544208675708b1e028846 [id=f5c1be203e2b4831a025871cf8a2ef17]). Expected version 2 got 0
I20251025 07:52:39.315325 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 509c94f2a422493ea4390ecee576db72 (table default.Table_3f081b7bb4e1420c8121ea4115439f3d [id=696a1285fbfe4038a8b1b38708c8ea49]). Expected version 1 got 0
I20251025 07:52:39.315667 450 catalog_manager.cc:5649] T 9979cae195e849908677d78c6c33be71 P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } }
I20251025 07:52:39.316195 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet ab42909afc524f2fa3c1041495d13a59 (table default.Table_7eb8424d5192485da1a2d67bfdb2bbbf [id=40c29df99590408cbd5c583d4fc741eb]). Expected version 2 got 0
I20251025 07:52:39.317401 1167 tablet_bootstrap.cc:492] T f43ef51ee45d47a99ec732b49d50d3b4 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:39.317538 1167 ts_tablet_manager.cc:1403] T f43ef51ee45d47a99ec732b49d50d3b4 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.002s
I20251025 07:52:39.318360 1167 raft_consensus.cc:359] T f43ef51ee45d47a99ec732b49d50d3b4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.318506 1167 raft_consensus.cc:385] T f43ef51ee45d47a99ec732b49d50d3b4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.318544 1167 raft_consensus.cc:740] T f43ef51ee45d47a99ec732b49d50d3b4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.318710 1167 consensus_queue.cc:260] T f43ef51ee45d47a99ec732b49d50d3b4 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.319125 1167 ts_tablet_manager.cc:1434] T f43ef51ee45d47a99ec732b49d50d3b4 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:39.319384 1167 tablet_bootstrap.cc:492] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:39.320962 1167 tablet_bootstrap.cc:654] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.323450 2146 raft_consensus.cc:2272] T fa77085e81c7440d9e255c44d1e80c26 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:39.324081 2146 ts_tablet_manager.cc:1916] T fa77085e81c7440d9e255c44d1e80c26 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:39.327612 1166 tablet_bootstrap.cc:492] T 74e2d78c024849b0af5c0bde1f48c36f P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:39.327729 1166 ts_tablet_manager.cc:1403] T 74e2d78c024849b0af5c0bde1f48c36f P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.084s user 0.002s sys 0.016s
I20251025 07:52:39.329411 1166 raft_consensus.cc:359] T 74e2d78c024849b0af5c0bde1f48c36f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.329723 1166 raft_consensus.cc:385] T 74e2d78c024849b0af5c0bde1f48c36f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.329838 1166 raft_consensus.cc:740] T 74e2d78c024849b0af5c0bde1f48c36f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.334820 1169 tablet_bootstrap.cc:654] T 74e2d78c024849b0af5c0bde1f48c36f P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.335179 961 heartbeater.cc:344] Connected to a master server at 127.0.104.61:42203
I20251025 07:52:39.323240 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e]). Expected version 1 got 0
I20251025 07:52:39.337020 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet bc2d16b33ca8426abbd7bd38339c9f1d (table default.Table_4ea985df4c5b415c9570c64506a98cc4 [id=98d2b7797e33485e8872054aa2eb8ec9]). Expected version 1 got 0
I20251025 07:52:39.337635 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet cd23b8094d7e472eba6bd170248858a9 (table default.Table_ca479cb5c1184c5391b7e1c9d8a08937 [id=d82b3e1317434cf08a0d51e8a9427224]). Expected version 3 got 2
I20251025 07:52:39.339902 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet ef1854d35a3047e3a9fbe0166ab1295a (table default.Table_088aad2a46b041bfa29ed8e6929760b1 [id=834669c54dae449790818684fd8659bc]). Expected version 2 got 0
I20251025 07:52:39.340446 2169 ts_tablet_manager.cc:1929] T 8c76d1b509f04ab1a10b83cff675c11e P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:39.340834 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]). Expected version 1 got 0
I20251025 07:52:39.341013 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet ef9a2c9f2ccf432ab835396020872208 (table default.Table_2b9b772d4c234ac286bef3658482e490 [id=409fab60f9d744798e6f92b868f0f7c2]). Expected version 1 got 0
I20251025 07:52:39.341177 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]). Expected version 1 got 0
I20251025 07:52:39.349676 2166 raft_consensus.cc:2243] T 50ae9bca6bed4f749d54191ac1543410 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:39.350126 2166 raft_consensus.cc:2272] T 50ae9bca6bed4f749d54191ac1543410 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:39.350705 2166 ts_tablet_manager.cc:1916] T 50ae9bca6bed4f749d54191ac1543410 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:39.352312 2146 ts_tablet_manager.cc:1929] T fa77085e81c7440d9e255c44d1e80c26 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:39.352439 2146 log.cc:1199] T fa77085e81c7440d9e255c44d1e80c26 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/fa77085e81c7440d9e255c44d1e80c26
I20251025 07:52:39.352846 2146 ts_tablet_manager.cc:1950] T fa77085e81c7440d9e255c44d1e80c26 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:39.353520 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet fa77085e81c7440d9e255c44d1e80c26 (table default.Table_936e8f1bd9344704940225361305cf5b [id=4998c2abb8fb48db83483bc0aa0170eb]) successfully deleted
I20251025 07:52:39.354220 2169 log.cc:1199] T 8c76d1b509f04ab1a10b83cff675c11e P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/8c76d1b509f04ab1a10b83cff675c11e
I20251025 07:52:39.352039 1169 tablet_bootstrap.cc:492] T 74e2d78c024849b0af5c0bde1f48c36f P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:39.354759 1169 ts_tablet_manager.cc:1403] T 74e2d78c024849b0af5c0bde1f48c36f P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.000s
I20251025 07:52:39.356288 1166 consensus_queue.cc:260] T 74e2d78c024849b0af5c0bde1f48c36f P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.357092 1166 ts_tablet_manager.cc:1434] T 74e2d78c024849b0af5c0bde1f48c36f P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.029s user 0.000s sys 0.002s
I20251025 07:52:39.357841 2169 ts_tablet_manager.cc:1950] T 8c76d1b509f04ab1a10b83cff675c11e P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:39.358052 1169 raft_consensus.cc:359] T 74e2d78c024849b0af5c0bde1f48c36f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
W20251025 07:52:39.359169 850 consensus_peers.cc:597] T 1d261e432cf5403ea162129706bc4d4d P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d261e432cf5403ea162129706bc4d4d. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:39.359503 2146 tablet_replica.cc:333] T a6ddefa1738f4ff98586900c344359d2 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:39.359737 2146 raft_consensus.cc:2243] T a6ddefa1738f4ff98586900c344359d2 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:39.359974 2146 raft_consensus.cc:2272] T a6ddefa1738f4ff98586900c344359d2 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:39.360030 1166 tablet_bootstrap.cc:492] T 6a58931e7cd7454fb583f1c70cd2ad15 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:39.361763 1166 tablet_bootstrap.cc:654] T 6a58931e7cd7454fb583f1c70cd2ad15 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.372195 2166 ts_tablet_manager.cc:1929] T 50ae9bca6bed4f749d54191ac1543410 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:39.372315 2166 log.cc:1199] T 50ae9bca6bed4f749d54191ac1543410 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/50ae9bca6bed4f749d54191ac1543410
I20251025 07:52:39.372588 2166 ts_tablet_manager.cc:1950] T 50ae9bca6bed4f749d54191ac1543410 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:39.373296 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 50ae9bca6bed4f749d54191ac1543410 (table default.Table_b42187d20a7e4f8783a8d25acb249350 [id=a5c73cb1e5a349ca8675f4ba4cb8a82d]) successfully deleted
I20251025 07:52:39.373661 1169 raft_consensus.cc:385] T 74e2d78c024849b0af5c0bde1f48c36f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.376211 2105 consensus_queue.cc:1048] T 8f2c8582fed34c848d9903b8ad8b4057 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:39.378270 2146 ts_tablet_manager.cc:1916] T a6ddefa1738f4ff98586900c344359d2 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:39.381986 2166 tablet_replica.cc:333] T 3abfd52327a04ba88625f391a8658fbb P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:39.382288 2166 raft_consensus.cc:2243] T 3abfd52327a04ba88625f391a8658fbb P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:39.382508 2166 raft_consensus.cc:2272] T 3abfd52327a04ba88625f391a8658fbb P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:39.382640 1659 raft_consensus.cc:493] T 88eb51447e0f4080baa49dd902deb24f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:39.383466 1659 raft_consensus.cc:515] T 88eb51447e0f4080baa49dd902deb24f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:39.384115 1659 leader_election.cc:290] T 88eb51447e0f4080baa49dd902deb24f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:39.384791 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88eb51447e0f4080baa49dd902deb24f" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:39.385509 710 leader_election.cc:343] T 88eb51447e0f4080baa49dd902deb24f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:39.386294 2166 ts_tablet_manager.cc:1916] T 3abfd52327a04ba88625f391a8658fbb P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:39.397969 442 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_a6c869949b744bc98efb735b0e73e7aa [id=bf77b5ff6f1048dd996d948437f9ff5e], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet d46673a283fe42b78ddeb99fbd2678eb consensus state has no leader
I20251025 07:52:39.397867 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88eb51447e0f4080baa49dd902deb24f" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:39.398592 1169 raft_consensus.cc:740] T 74e2d78c024849b0af5c0bde1f48c36f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.399199 1169 consensus_queue.cc:260] T 74e2d78c024849b0af5c0bde1f48c36f P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.400715 1169 ts_tablet_manager.cc:1434] T 74e2d78c024849b0af5c0bde1f48c36f P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.046s user 0.002s sys 0.000s
I20251025 07:52:39.401103 1169 tablet_bootstrap.cc:492] T fe5033e5ff404a5ebc6c8443d7548327 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:39.403230 1169 tablet_bootstrap.cc:654] T fe5033e5ff404a5ebc6c8443d7548327 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.403340 2146 ts_tablet_manager.cc:1929] T a6ddefa1738f4ff98586900c344359d2 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:39.403471 2146 log.cc:1199] T a6ddefa1738f4ff98586900c344359d2 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/a6ddefa1738f4ff98586900c344359d2
I20251025 07:52:39.403836 2146 ts_tablet_manager.cc:1950] T a6ddefa1738f4ff98586900c344359d2 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:39.405815 1166 tablet_bootstrap.cc:492] T 6a58931e7cd7454fb583f1c70cd2ad15 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:39.405942 1166 ts_tablet_manager.cc:1403] T 6a58931e7cd7454fb583f1c70cd2ad15 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.046s user 0.006s sys 0.004s
I20251025 07:52:39.406877 1166 raft_consensus.cc:359] T 6a58931e7cd7454fb583f1c70cd2ad15 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.407070 1166 raft_consensus.cc:385] T 6a58931e7cd7454fb583f1c70cd2ad15 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.407122 1166 raft_consensus.cc:740] T 6a58931e7cd7454fb583f1c70cd2ad15 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
W20251025 07:52:39.398432 712 leader_election.cc:343] T 88eb51447e0f4080baa49dd902deb24f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:39.407444 1166 consensus_queue.cc:260] T 6a58931e7cd7454fb583f1c70cd2ad15 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.407577 712 leader_election.cc:304] T 88eb51447e0f4080baa49dd902deb24f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:39.408004 1659 raft_consensus.cc:2749] T 88eb51447e0f4080baa49dd902deb24f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:39.408152 1166 ts_tablet_manager.cc:1434] T 6a58931e7cd7454fb583f1c70cd2ad15 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251025 07:52:39.417224 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 8c76d1b509f04ab1a10b83cff675c11e (table default.Table_cf686e1101314426a3ab24fd46a8974f [id=13fbed1c733c42e2b45e4ca69580b06d]) successfully deleted
I20251025 07:52:39.417713 2169 tablet_replica.cc:333] T b772025852794a49b072f37cccbd3291 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:39.418431 2169 raft_consensus.cc:2243] T b772025852794a49b072f37cccbd3291 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:39.418668 2146 tablet_replica.cc:333] T f0bdaa8cfd764f1b83f47881353d26b6 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:39.418670 2169 raft_consensus.cc:2272] T b772025852794a49b072f37cccbd3291 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:39.418860 2146 raft_consensus.cc:2243] T f0bdaa8cfd764f1b83f47881353d26b6 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:39.419013 2146 raft_consensus.cc:2272] T f0bdaa8cfd764f1b83f47881353d26b6 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:39.419242 2169 ts_tablet_manager.cc:1916] T b772025852794a49b072f37cccbd3291 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:39.419497 2146 ts_tablet_manager.cc:1916] T f0bdaa8cfd764f1b83f47881353d26b6 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:39.422415 1166 tablet_bootstrap.cc:492] T 943ec703e65f4fd3b0c690e08c0bd226 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:39.423120 1167 tablet_bootstrap.cc:492] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:39.423250 1167 ts_tablet_manager.cc:1403] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.104s user 0.006s sys 0.004s
I20251025 07:52:39.424115 1166 tablet_bootstrap.cc:654] T 943ec703e65f4fd3b0c690e08c0bd226 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.424084 1167 raft_consensus.cc:359] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:39.424249 1167 raft_consensus.cc:385] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.424300 1167 raft_consensus.cc:740] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.424502 1167 consensus_queue.cc:260] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:39.425004 1167 ts_tablet_manager.cc:1434] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 07:52:39.425154 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet a6ddefa1738f4ff98586900c344359d2 (table default.Table_c902adc24467448f9b5d4f585cf614f5 [id=56e169f79a37465d82dd1d1c3c285740]) successfully deleted
I20251025 07:52:39.425249 1167 tablet_bootstrap.cc:492] T 88bfc23acd274041a9457079c1709d82 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:39.426932 1167 tablet_bootstrap.cc:654] T 88bfc23acd274041a9457079c1709d82 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.429451 1659 raft_consensus.cc:493] T b3a0af893a8947848e12093d0a7f0229 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:39.429591 1659 raft_consensus.cc:515] T b3a0af893a8947848e12093d0a7f0229 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:39.430464 1659 leader_election.cc:290] T b3a0af893a8947848e12093d0a7f0229 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:39.431028 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3a0af893a8947848e12093d0a7f0229" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:39.431398 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3a0af893a8947848e12093d0a7f0229" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:39.431499 710 leader_election.cc:343] T b3a0af893a8947848e12093d0a7f0229 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:39.431924 712 leader_election.cc:343] T b3a0af893a8947848e12093d0a7f0229 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:39.432067 712 leader_election.cc:304] T b3a0af893a8947848e12093d0a7f0229 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:39.432300 1659 raft_consensus.cc:2749] T b3a0af893a8947848e12093d0a7f0229 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:39.433442 2146 ts_tablet_manager.cc:1929] T f0bdaa8cfd764f1b83f47881353d26b6 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:39.434907 2146 log.cc:1199] T f0bdaa8cfd764f1b83f47881353d26b6 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/f0bdaa8cfd764f1b83f47881353d26b6
I20251025 07:52:39.435433 2146 ts_tablet_manager.cc:1950] T f0bdaa8cfd764f1b83f47881353d26b6 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:39.436143 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 67) took 1010 ms. Trace:
I20251025 07:52:39.436606 2146 rpcz_store.cc:276] 1025 07:52:38.425336 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.426182 (+ 846us) service_pool.cc:225] Handling call
1025 07:52:39.418627 (+992445us) ts_tablet_manager.cc:1151] Deleting tablet f0bdaa8cfd764f1b83f47881353d26b6
1025 07:52:39.418634 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:39.428929 (+ 10295us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:39.433348 (+ 4419us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:39.436129 (+ 2781us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":992265}
I20251025 07:52:39.437430 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet f0bdaa8cfd764f1b83f47881353d26b6 (table default.Table_49beae553d03426aa395b7cdb6dfec9a [id=ab3328aa6f4148bfa2877cd4f0c6d324]) successfully deleted
I20251025 07:52:39.439356 2146 tablet_replica.cc:333] T 8956ad28b35d45f28c683ac055bba441 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:39.440100 2146 raft_consensus.cc:2243] T 8956ad28b35d45f28c683ac055bba441 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:39.441372 2105 raft_consensus.cc:493] T 5361f08f50b745eb98bf02e979d7c71e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:39.442293 2105 raft_consensus.cc:515] T 5361f08f50b745eb98bf02e979d7c71e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:39.442864 2105 leader_election.cc:290] T 5361f08f50b745eb98bf02e979d7c71e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
W20251025 07:52:39.443831 850 leader_election.cc:343] T 5361f08f50b745eb98bf02e979d7c71e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 5361f08f50b745eb98bf02e979d7c71e
W20251025 07:52:39.445307 847 leader_election.cc:343] T 5361f08f50b745eb98bf02e979d7c71e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: 5361f08f50b745eb98bf02e979d7c71e
I20251025 07:52:39.445420 847 leader_election.cc:304] T 5361f08f50b745eb98bf02e979d7c71e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:39.445842 2105 raft_consensus.cc:2749] T 5361f08f50b745eb98bf02e979d7c71e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:39.446229 2146 raft_consensus.cc:2272] T 8956ad28b35d45f28c683ac055bba441 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:39.447091 2146 ts_tablet_manager.cc:1916] T 8956ad28b35d45f28c683ac055bba441 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:39.455192 2166 ts_tablet_manager.cc:1929] T 3abfd52327a04ba88625f391a8658fbb P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:39.455327 2166 log.cc:1199] T 3abfd52327a04ba88625f391a8658fbb P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/3abfd52327a04ba88625f391a8658fbb
I20251025 07:52:39.455730 2166 ts_tablet_manager.cc:1950] T 3abfd52327a04ba88625f391a8658fbb P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:39.456710 2166 tablet_replica.cc:333] T b3a0af893a8947848e12093d0a7f0229 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:39.456867 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 3abfd52327a04ba88625f391a8658fbb (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]) successfully deleted
I20251025 07:52:39.456923 2166 raft_consensus.cc:2243] T b3a0af893a8947848e12093d0a7f0229 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:39.458693 1166 tablet_bootstrap.cc:492] T 943ec703e65f4fd3b0c690e08c0bd226 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:39.458729 2166 raft_consensus.cc:2272] T b3a0af893a8947848e12093d0a7f0229 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:39.458815 1166 ts_tablet_manager.cc:1403] T 943ec703e65f4fd3b0c690e08c0bd226 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.003s
I20251025 07:52:39.459198 2166 ts_tablet_manager.cc:1916] T b3a0af893a8947848e12093d0a7f0229 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:39.459532 1166 raft_consensus.cc:359] T 943ec703e65f4fd3b0c690e08c0bd226 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.459708 1166 raft_consensus.cc:385] T 943ec703e65f4fd3b0c690e08c0bd226 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.459770 1166 raft_consensus.cc:740] T 943ec703e65f4fd3b0c690e08c0bd226 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.460011 1166 consensus_queue.cc:260] T 943ec703e65f4fd3b0c690e08c0bd226 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.465025 1166 ts_tablet_manager.cc:1434] T 943ec703e65f4fd3b0c690e08c0bd226 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251025 07:52:39.465325 1166 tablet_bootstrap.cc:492] T b947ecc352e54f4caa73c6ca97372256 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:39.471632 1166 tablet_bootstrap.cc:654] T b947ecc352e54f4caa73c6ca97372256 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.457968 1167 tablet_bootstrap.cc:492] T 88bfc23acd274041a9457079c1709d82 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:39.472294 1167 ts_tablet_manager.cc:1403] T 88bfc23acd274041a9457079c1709d82 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.000s
I20251025 07:52:39.473034 1167 raft_consensus.cc:359] T 88bfc23acd274041a9457079c1709d82 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.473217 1167 raft_consensus.cc:385] T 88bfc23acd274041a9457079c1709d82 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.473273 1167 raft_consensus.cc:740] T 88bfc23acd274041a9457079c1709d82 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.476845 2169 ts_tablet_manager.cc:1929] T b772025852794a49b072f37cccbd3291 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:39.481030 1167 consensus_queue.cc:260] T 88bfc23acd274041a9457079c1709d82 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.481630 1167 ts_tablet_manager.cc:1434] T 88bfc23acd274041a9457079c1709d82 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20251025 07:52:39.477656 1169 tablet_bootstrap.cc:492] T fe5033e5ff404a5ebc6c8443d7548327 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:39.484339 1169 ts_tablet_manager.cc:1403] T fe5033e5ff404a5ebc6c8443d7548327 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.083s user 0.021s sys 0.008s
I20251025 07:52:39.484992 1169 raft_consensus.cc:359] T fe5033e5ff404a5ebc6c8443d7548327 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.485189 1169 raft_consensus.cc:385] T fe5033e5ff404a5ebc6c8443d7548327 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.485255 1169 raft_consensus.cc:740] T fe5033e5ff404a5ebc6c8443d7548327 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.485469 1169 consensus_queue.cc:260] T fe5033e5ff404a5ebc6c8443d7548327 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.485900 1169 ts_tablet_manager.cc:1434] T fe5033e5ff404a5ebc6c8443d7548327 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 07:52:39.486212 1169 tablet_bootstrap.cc:492] T 943ec703e65f4fd3b0c690e08c0bd226 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:39.487565 1169 tablet_bootstrap.cc:654] T 943ec703e65f4fd3b0c690e08c0bd226 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.488224 2169 log.cc:1199] T b772025852794a49b072f37cccbd3291 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/b772025852794a49b072f37cccbd3291
I20251025 07:52:39.489516 2169 ts_tablet_manager.cc:1950] T b772025852794a49b072f37cccbd3291 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:39.490367 1167 tablet_bootstrap.cc:492] T 0325d462441349ef81841f1b0cab2a20 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:39.490947 2182 raft_consensus.cc:493] T 80ee75935d7e461d9d4066fac9819bef P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:39.491096 2182 raft_consensus.cc:515] T 80ee75935d7e461d9d4066fac9819bef P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.491725 2182 leader_election.cc:290] T 80ee75935d7e461d9d4066fac9819bef P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:39.492084 1167 tablet_bootstrap.cc:654] T 0325d462441349ef81841f1b0cab2a20 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.492214 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "80ee75935d7e461d9d4066fac9819bef" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:39.492427 911 raft_consensus.cc:2468] T 80ee75935d7e461d9d4066fac9819bef P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:39.492833 983 leader_election.cc:304] T 80ee75935d7e461d9d4066fac9819bef P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
W20251025 07:52:39.493012 982 leader_election.cc:343] T 80ee75935d7e461d9d4066fac9819bef P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: 80ee75935d7e461d9d4066fac9819bef
I20251025 07:52:39.493412 2182 raft_consensus.cc:2804] T 80ee75935d7e461d9d4066fac9819bef P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:39.493460 2146 ts_tablet_manager.cc:1929] T 8956ad28b35d45f28c683ac055bba441 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:39.493484 2182 raft_consensus.cc:493] T 80ee75935d7e461d9d4066fac9819bef P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:39.493541 2182 raft_consensus.cc:3060] T 80ee75935d7e461d9d4066fac9819bef P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:39.493563 2146 log.cc:1199] T 8956ad28b35d45f28c683ac055bba441 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/8956ad28b35d45f28c683ac055bba441
I20251025 07:52:39.493886 2146 ts_tablet_manager.cc:1950] T 8956ad28b35d45f28c683ac055bba441 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:39.496217 2182 raft_consensus.cc:515] T 80ee75935d7e461d9d4066fac9819bef P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.497364 2166 ts_tablet_manager.cc:1929] T b3a0af893a8947848e12093d0a7f0229 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:39.497475 2166 log.cc:1199] T b3a0af893a8947848e12093d0a7f0229 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/b3a0af893a8947848e12093d0a7f0229
W20251025 07:52:39.497558 982 leader_election.cc:343] T 80ee75935d7e461d9d4066fac9819bef P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: 80ee75935d7e461d9d4066fac9819bef
I20251025 07:52:39.497768 2166 ts_tablet_manager.cc:1950] T b3a0af893a8947848e12093d0a7f0229 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:39.498025 2182 leader_election.cc:290] T 80ee75935d7e461d9d4066fac9819bef P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:39.494210 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 68) took 1068 ms. Trace:
I20251025 07:52:39.498548 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "80ee75935d7e461d9d4066fac9819bef" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:39.498740 911 raft_consensus.cc:3060] T 80ee75935d7e461d9d4066fac9819bef P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:39.498485 2146 rpcz_store.cc:276] 1025 07:52:38.425435 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.426407 (+ 972us) service_pool.cc:225] Handling call
1025 07:52:39.437829 (+1011422us) ts_tablet_manager.cc:1151] Deleting tablet 8956ad28b35d45f28c683ac055bba441
1025 07:52:39.437838 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:39.474295 (+ 36457us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:39.477375 (+ 3080us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:39.494193 (+ 16818us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1011269}
I20251025 07:52:39.499977 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f]). Expected version 1 got 0
I20251025 07:52:39.501230 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 4c155aa7fa5b4d68bbab1918b8432372 (table default.Table_6c5a24747ba14352b0285a41ec2985a4 [id=28bbaad474164a8c8c97a76bb8081791]). Expected version 2 got 0
I20251025 07:52:39.501958 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7]). Expected version 1 got 0
I20251025 07:52:39.501557 911 raft_consensus.cc:2468] T 80ee75935d7e461d9d4066fac9819bef P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:39.503525 2146 tablet_replica.cc:333] T 0b21b7da1e2f47829c3f6cdaee689e8a P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:39.504061 2146 raft_consensus.cc:2243] T 0b21b7da1e2f47829c3f6cdaee689e8a P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:39.504420 983 leader_election.cc:304] T 80ee75935d7e461d9d4066fac9819bef P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b
I20251025 07:52:39.504930 2182 raft_consensus.cc:2804] T 80ee75935d7e461d9d4066fac9819bef P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:39.505093 2146 raft_consensus.cc:2272] T 0b21b7da1e2f47829c3f6cdaee689e8a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:39.505147 2182 raft_consensus.cc:697] T 80ee75935d7e461d9d4066fac9819bef P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:39.505450 2182 consensus_queue.cc:237] T 80ee75935d7e461d9d4066fac9819bef P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.505618 2146 ts_tablet_manager.cc:1916] T 0b21b7da1e2f47829c3f6cdaee689e8a P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:39.504752 1166 tablet_bootstrap.cc:492] T b947ecc352e54f4caa73c6ca97372256 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:39.510394 1166 ts_tablet_manager.cc:1403] T b947ecc352e54f4caa73c6ca97372256 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.002s
I20251025 07:52:39.511126 2169 tablet_replica.cc:333] T db99a1e68daf4e948a3d66ad276cc5f9 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:39.511276 1166 raft_consensus.cc:359] T b947ecc352e54f4caa73c6ca97372256 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.511379 2169 raft_consensus.cc:2243] T db99a1e68daf4e948a3d66ad276cc5f9 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:39.511472 1166 raft_consensus.cc:385] T b947ecc352e54f4caa73c6ca97372256 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.511523 2169 raft_consensus.cc:2272] T db99a1e68daf4e948a3d66ad276cc5f9 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:39.511559 1166 raft_consensus.cc:740] T b947ecc352e54f4caa73c6ca97372256 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.511818 1166 consensus_queue.cc:260] T b947ecc352e54f4caa73c6ca97372256 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.512137 2169 ts_tablet_manager.cc:1916] T db99a1e68daf4e948a3d66ad276cc5f9 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:39.514665 1166 ts_tablet_manager.cc:1434] T b947ecc352e54f4caa73c6ca97372256 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251025 07:52:39.514994 1166 tablet_bootstrap.cc:492] T f4578ce69a0d4322944e5ce7c8a59831 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:39.517321 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 04d27d54fdb7483b869408af65ff85c9 (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]). Expected version 2 got 0
I20251025 07:52:39.517728 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet cb829649f73d4d3f8862d268c953bd86 (table default.Table_ba4b8cc7b930482690b7e192e0edff12 [id=7944225d3ce7439b881bfb9289d9f040]). Expected version 1 got 0
I20251025 07:52:39.517881 1169 tablet_bootstrap.cc:492] T 943ec703e65f4fd3b0c690e08c0bd226 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:39.518016 1169 ts_tablet_manager.cc:1403] T 943ec703e65f4fd3b0c690e08c0bd226 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20251025 07:52:39.503947 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 8956ad28b35d45f28c683ac055bba441 (table default.Table_f84af906af804c7a8e98ad602783d747 [id=39a63c8186374189ad21fbf8492d5d2b]) successfully deleted
I20251025 07:52:39.510855 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet b772025852794a49b072f37cccbd3291 (table default.Table_e50bf47b78d94bbc811e6f3fb3551edb [id=0827603890c8466082bc6c306c9c66af]) successfully deleted
I20251025 07:52:39.518675 1169 raft_consensus.cc:359] T 943ec703e65f4fd3b0c690e08c0bd226 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.518824 1169 raft_consensus.cc:385] T 943ec703e65f4fd3b0c690e08c0bd226 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.518867 1169 raft_consensus.cc:740] T 943ec703e65f4fd3b0c690e08c0bd226 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.519038 1169 consensus_queue.cc:260] T 943ec703e65f4fd3b0c690e08c0bd226 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.519585 1169 ts_tablet_manager.cc:1434] T 943ec703e65f4fd3b0c690e08c0bd226 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251025 07:52:39.519869 1169 tablet_bootstrap.cc:492] T ef9a2c9f2ccf432ab835396020872208 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:39.519946 1166 tablet_bootstrap.cc:654] T f4578ce69a0d4322944e5ce7c8a59831 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.521158 1169 tablet_bootstrap.cc:654] T ef9a2c9f2ccf432ab835396020872208 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.521538 1167 tablet_bootstrap.cc:492] T 0325d462441349ef81841f1b0cab2a20 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:39.521682 1167 ts_tablet_manager.cc:1403] T 0325d462441349ef81841f1b0cab2a20 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.001s
I20251025 07:52:39.523177 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet aa9d9ab8073945d28d52fb02213f201a (table default.Table_4791672658f043f6b97b9320a531ebb5 [id=dfd20cf9c3e541c4abe23d59e0aa7f2a]). Expected version 1 got 0
I20251025 07:52:39.523746 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 50) took 1011 ms. Trace:
I20251025 07:52:39.535454 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]). Expected version 1 got 0
I20251025 07:52:39.535632 2166 rpcz_store.cc:276] 1025 07:52:38.512035 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.514097 (+ 2062us) service_pool.cc:225] Handling call
1025 07:52:39.456661 (+942564us) ts_tablet_manager.cc:1151] Deleting tablet b3a0af893a8947848e12093d0a7f0229
1025 07:52:39.456673 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:39.494254 (+ 37581us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:39.497270 (+ 3016us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:39.523721 (+ 26451us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":940892}
I20251025 07:52:39.538882 2169 ts_tablet_manager.cc:1929] T db99a1e68daf4e948a3d66ad276cc5f9 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:39.539171 2146 ts_tablet_manager.cc:1929] T 0b21b7da1e2f47829c3f6cdaee689e8a P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:39.539162 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet ef9a2c9f2ccf432ab835396020872208 (table default.Table_2b9b772d4c234ac286bef3658482e490 [id=409fab60f9d744798e6f92b868f0f7c2]). Expected version 1 got 0
I20251025 07:52:39.539417 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]). Expected version 1 got 0
I20251025 07:52:39.539757 1659 raft_consensus.cc:493] T f97acc9a638b4d5a968dcc3412a384bc P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:39.539887 1659 raft_consensus.cc:515] T f97acc9a638b4d5a968dcc3412a384bc P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.534929 1167 raft_consensus.cc:359] T 0325d462441349ef81841f1b0cab2a20 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:39.540386 1167 raft_consensus.cc:385] T 0325d462441349ef81841f1b0cab2a20 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.540455 1167 raft_consensus.cc:740] T 0325d462441349ef81841f1b0cab2a20 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.540547 1659 leader_election.cc:290] T f97acc9a638b4d5a968dcc3412a384bc P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:39.540634 1167 consensus_queue.cc:260] T 0325d462441349ef81841f1b0cab2a20 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:39.541163 1167 ts_tablet_manager.cc:1434] T 0325d462441349ef81841f1b0cab2a20 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20251025 07:52:39.541162 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f97acc9a638b4d5a968dcc3412a384bc" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:39.541814 2105 raft_consensus.cc:493] T df1248f6dbd94af68c587da7d149e279 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:39.542217 2105 raft_consensus.cc:515] T df1248f6dbd94af68c587da7d149e279 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:39.542779 2105 leader_election.cc:290] T df1248f6dbd94af68c587da7d149e279 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:39.544260 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df1248f6dbd94af68c587da7d149e279" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:39.544477 774 raft_consensus.cc:2393] T df1248f6dbd94af68c587da7d149e279 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3aea2869ff814cc7b142012faa274382 in current term 1: Already voted for candidate db42be7ad3c741a0aa427f27e245993b in this term.
W20251025 07:52:39.545503 710 leader_election.cc:343] T f97acc9a638b4d5a968dcc3412a384bc P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:39.545743 2146 log.cc:1199] T 0b21b7da1e2f47829c3f6cdaee689e8a P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/0b21b7da1e2f47829c3f6cdaee689e8a
I20251025 07:52:39.546327 2146 ts_tablet_manager.cc:1950] T 0b21b7da1e2f47829c3f6cdaee689e8a P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:39.550252 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 66) took 1125 ms. Trace:
I20251025 07:52:39.550469 2146 rpcz_store.cc:276] 1025 07:52:38.425234 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.425999 (+ 765us) service_pool.cc:225] Handling call
1025 07:52:39.503477 (+1077478us) ts_tablet_manager.cc:1151] Deleting tablet 0b21b7da1e2f47829c3f6cdaee689e8a
1025 07:52:39.503487 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:39.508754 (+ 5267us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:39.539076 (+ 30322us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:39.550236 (+ 11160us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1076637}
I20251025 07:52:39.551563 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 69) took 1124 ms. Trace:
I20251025 07:52:39.551992 2146 rpcz_store.cc:276] 1025 07:52:38.426667 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.426907 (+ 240us) service_pool.cc:225] Handling call
1025 07:52:39.551395 (+1124488us) ts_tablet_manager.cc:1151] Deleting tablet c11f674385184ae6b235f1db8bf1fb91
1025 07:52:39.551413 (+ 18us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:39.551547 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1124268}
I20251025 07:52:39.553838 1167 tablet_bootstrap.cc:492] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:39.554910 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 70) took 1128 ms. Trace:
I20251025 07:52:39.555274 2146 rpcz_store.cc:276] 1025 07:52:38.426757 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.427419 (+ 662us) service_pool.cc:225] Handling call
1025 07:52:39.554789 (+1127370us) ts_tablet_manager.cc:1151] Deleting tablet fd265cdbf6a8440393a2b5d8240d1872
1025 07:52:39.554800 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:39.554896 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1127172}
I20251025 07:52:39.555516 1167 tablet_bootstrap.cc:654] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.556082 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 60) took 1172 ms. Trace:
I20251025 07:52:39.556485 2146 rpcz_store.cc:276] 1025 07:52:38.383678 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.383785 (+ 107us) service_pool.cc:225] Handling call
1025 07:52:39.555921 (+1172136us) ts_tablet_manager.cc:1151] Deleting tablet 48ff4ce7846b4e56a37eee9bcb17bd5e
1025 07:52:39.555931 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:39.556066 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1125220}
I20251025 07:52:39.557106 2166 tablet_replica.cc:333] T 34315f64b89148efbf1f20a43d5c5efb P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:39.557163 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 72) took 1128 ms. Trace:
I20251025 07:52:39.557340 2166 raft_consensus.cc:2243] T 34315f64b89148efbf1f20a43d5c5efb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:39.558076 2166 raft_consensus.cc:2272] T 34315f64b89148efbf1f20a43d5c5efb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:39.558606 2166 ts_tablet_manager.cc:1916] T 34315f64b89148efbf1f20a43d5c5efb P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:39.540030 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet b3a0af893a8947848e12093d0a7f0229 (table default.Table_0f2c837d88c043fda9d4eb94706a5398 [id=c837adb3bd5a4436a4cd8ea6f7be6b9e]) successfully deleted
I20251025 07:52:39.551177 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 0b21b7da1e2f47829c3f6cdaee689e8a (table default.Table_e61ae9065c604ad8b93af8146748cec9 [id=6d26779bac7243f19d534277dd1c986c]) successfully deleted
I20251025 07:52:39.563467 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_a6c869949b744bc98efb735b0e73e7aa [id=bf77b5ff6f1048dd996d948437f9ff5e]). Expected version 2 got 0
W20251025 07:52:39.563684 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet c11f674385184ae6b235f1db8bf1fb91 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c11f674385184ae6b235f1db8bf1fb91 already in progress: creating tablet
W20251025 07:52:39.563966 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet fd265cdbf6a8440393a2b5d8240d1872 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd265cdbf6a8440393a2b5d8240d1872 already in progress: creating tablet
W20251025 07:52:39.564245 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 48ff4ce7846b4e56a37eee9bcb17bd5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48ff4ce7846b4e56a37eee9bcb17bd5e already in progress: creating tablet
I20251025 07:52:39.563971 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet ef1854d35a3047e3a9fbe0166ab1295a (table default.Table_088aad2a46b041bfa29ed8e6929760b1 [id=834669c54dae449790818684fd8659bc]). Expected version 2 got 0
I20251025 07:52:39.564471 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet cd23b8094d7e472eba6bd170248858a9 (table default.Table_ca479cb5c1184c5391b7e1c9d8a08937 [id=d82b3e1317434cf08a0d51e8a9427224]). Expected version 3 got 2
I20251025 07:52:39.564746 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet bc2d16b33ca8426abbd7bd38339c9f1d (table default.Table_4ea985df4c5b415c9570c64506a98cc4 [id=98d2b7797e33485e8872054aa2eb8ec9]). Expected version 1 got 0
I20251025 07:52:39.565039 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e]). Expected version 1 got 0
I20251025 07:52:39.566164 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525]). Expected version 1 got 0
I20251025 07:52:39.570307 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84]). Expected version 1 got 0
I20251025 07:52:39.571807 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 5dab439bc50d4e869548154e1d222ad6 (table default.Table_bd82d871f84943f4baa967f5c0ec74a3 [id=45325c188eac4d34bb016d01de091f87]). Expected version 2 got 0
I20251025 07:52:39.541594 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f97acc9a638b4d5a968dcc3412a384bc" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:39.572023 2169 log.cc:1199] T db99a1e68daf4e948a3d66ad276cc5f9 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/db99a1e68daf4e948a3d66ad276cc5f9
I20251025 07:52:39.544797 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df1248f6dbd94af68c587da7d149e279" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:39.553161 1169 tablet_bootstrap.cc:492] T ef9a2c9f2ccf432ab835396020872208 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:39.572379 1169 ts_tablet_manager.cc:1403] T ef9a2c9f2ccf432ab835396020872208 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.001s
I20251025 07:52:39.572682 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_87d38832f85e4965bfab1a077990114f [id=ad812862698d421184ebf501ce79b7be]). Expected version 2 got 0
W20251025 07:52:39.573318 712 leader_election.cc:343] T f97acc9a638b4d5a968dcc3412a384bc P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:39.573299 1169 raft_consensus.cc:359] T ef9a2c9f2ccf432ab835396020872208 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.573482 1169 raft_consensus.cc:385] T ef9a2c9f2ccf432ab835396020872208 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.573442 712 leader_election.cc:304] T f97acc9a638b4d5a968dcc3412a384bc P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:39.573513 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5]). Expected version 1 got 0
I20251025 07:52:39.573558 1169 raft_consensus.cc:740] T ef9a2c9f2ccf432ab835396020872208 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.573781 1169 consensus_queue.cc:260] T ef9a2c9f2ccf432ab835396020872208 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.573904 1659 raft_consensus.cc:2749] T f97acc9a638b4d5a968dcc3412a384bc P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:39.573923 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 830fdbc6812e416690d0db42a73af503 (table default.Table_92824b3f2a774564b3aea04aaa25d204 [id=fb21a45b86ab446e8c3dae4512a9598f]). Expected version 1 got 0
I20251025 07:52:39.574302 2169 ts_tablet_manager.cc:1950] T db99a1e68daf4e948a3d66ad276cc5f9 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:39.574916 1169 ts_tablet_manager.cc:1434] T ef9a2c9f2ccf432ab835396020872208 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:39.577445 2146 rpcz_store.cc:276] 1025 07:52:38.428878 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.430806 (+ 1928us) service_pool.cc:225] Handling call
1025 07:52:39.557041 (+1126235us) ts_tablet_manager.cc:1151] Deleting tablet d77704b4fe6c4456a2ba40ffec0c60fc
1025 07:52:39.557050 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:39.557149 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1126075}
I20251025 07:52:39.572872 850 leader_election.cc:304] T df1248f6dbd94af68c587da7d149e279 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:39.581141 1169 tablet_bootstrap.cc:492] T e54fe7cbcffa40c4b04144b47bee2b48 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:39.583460 1169 tablet_bootstrap.cc:654] T e54fe7cbcffa40c4b04144b47bee2b48 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:39.585054 712 consensus_peers.cc:597] T 8c76d1b509f04ab1a10b83cff675c11e P 23eeffd3de254ce18db5febfde971c6b -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c76d1b509f04ab1a10b83cff675c11e. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:39.587213 1167 tablet_bootstrap.cc:492] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:39.587363 1167 ts_tablet_manager.cc:1403] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20251025 07:52:39.570644 1166 tablet_bootstrap.cc:492] T f4578ce69a0d4322944e5ce7c8a59831 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:39.587564 1166 ts_tablet_manager.cc:1403] T f4578ce69a0d4322944e5ce7c8a59831 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.073s user 0.003s sys 0.006s
I20251025 07:52:39.588119 1167 raft_consensus.cc:359] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.589000 1166 raft_consensus.cc:359] T f4578ce69a0d4322944e5ce7c8a59831 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.590508 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet db99a1e68daf4e948a3d66ad276cc5f9 (table default.Table_e1d698cd782e4d58b29e9cc7d1f56703 [id=e8dcb9f9f6554f5ebcdfee63c6259f03]) successfully deleted
I20251025 07:52:39.590760 1167 raft_consensus.cc:385] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.590935 1167 raft_consensus.cc:740] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.591207 1166 raft_consensus.cc:385] T f4578ce69a0d4322944e5ce7c8a59831 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.591220 1167 consensus_queue.cc:260] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.591331 1166 raft_consensus.cc:740] T f4578ce69a0d4322944e5ce7c8a59831 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.591620 1166 consensus_queue.cc:260] T f4578ce69a0d4322944e5ce7c8a59831 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.594391 1166 ts_tablet_manager.cc:1434] T f4578ce69a0d4322944e5ce7c8a59831 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20251025 07:52:39.594693 1166 tablet_bootstrap.cc:492] T aa9d9ab8073945d28d52fb02213f201a P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:39.595335 1096 heartbeater.cc:344] Connected to a master server at 127.0.104.61:42203
W20251025 07:52:39.595772 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet d77704b4fe6c4456a2ba40ffec0c60fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d77704b4fe6c4456a2ba40ffec0c60fc already in progress: creating tablet
I20251025 07:52:39.597328 2146 tablet_replica.cc:333] T 887e7c7a5d0e4aaf88f26246306df19c P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:39.600867 2182 raft_consensus.cc:493] T 74e2d78c024849b0af5c0bde1f48c36f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:39.601022 2182 raft_consensus.cc:515] T 74e2d78c024849b0af5c0bde1f48c36f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.601136 2146 raft_consensus.cc:2243] T 887e7c7a5d0e4aaf88f26246306df19c P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:39.601504 2146 raft_consensus.cc:2272] T 887e7c7a5d0e4aaf88f26246306df19c P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:39.601676 2182 leader_election.cc:290] T 74e2d78c024849b0af5c0bde1f48c36f P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:39.602741 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74e2d78c024849b0af5c0bde1f48c36f" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:39.602955 774 raft_consensus.cc:2468] T 74e2d78c024849b0af5c0bde1f48c36f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:39.602464 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74e2d78c024849b0af5c0bde1f48c36f" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:39.604178 911 raft_consensus.cc:2468] T 74e2d78c024849b0af5c0bde1f48c36f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:39.604604 983 leader_election.cc:304] T 74e2d78c024849b0af5c0bde1f48c36f P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:39.605324 2050 raft_consensus.cc:2804] T 74e2d78c024849b0af5c0bde1f48c36f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:39.605414 2050 raft_consensus.cc:493] T 74e2d78c024849b0af5c0bde1f48c36f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:39.605494 2050 raft_consensus.cc:3060] T 74e2d78c024849b0af5c0bde1f48c36f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:39.606581 2105 raft_consensus.cc:493] T b947ecc352e54f4caa73c6ca97372256 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:39.606694 2105 raft_consensus.cc:515] T b947ecc352e54f4caa73c6ca97372256 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.607420 2105 leader_election.cc:290] T b947ecc352e54f4caa73c6ca97372256 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:39.610000 2182 consensus_queue.cc:1048] T d66ad049d5484228a268b0facb00ecf1 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:39.610267 1169 tablet_bootstrap.cc:492] T e54fe7cbcffa40c4b04144b47bee2b48 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:39.610411 1169 ts_tablet_manager.cc:1403] T e54fe7cbcffa40c4b04144b47bee2b48 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.004s
W20251025 07:52:39.610735 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet ffe7c8e1d6c44872b7b65e512764a56a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ffe7c8e1d6c44872b7b65e512764a56a already in progress: creating tablet
I20251025 07:52:39.611382 1169 raft_consensus.cc:359] T e54fe7cbcffa40c4b04144b47bee2b48 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.611552 1169 raft_consensus.cc:385] T e54fe7cbcffa40c4b04144b47bee2b48 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.611631 1169 raft_consensus.cc:740] T e54fe7cbcffa40c4b04144b47bee2b48 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.612097 1169 consensus_queue.cc:260] T e54fe7cbcffa40c4b04144b47bee2b48 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
W20251025 07:52:39.612254 847 leader_election.cc:343] T b947ecc352e54f4caa73c6ca97372256 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: b947ecc352e54f4caa73c6ca97372256
I20251025 07:52:39.612593 1169 ts_tablet_manager.cc:1434] T e54fe7cbcffa40c4b04144b47bee2b48 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:39.614800 2050 raft_consensus.cc:515] T 74e2d78c024849b0af5c0bde1f48c36f P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.615505 2050 leader_election.cc:290] T 74e2d78c024849b0af5c0bde1f48c36f P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:39.615888 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74e2d78c024849b0af5c0bde1f48c36f" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:39.616070 776 raft_consensus.cc:3060] T 74e2d78c024849b0af5c0bde1f48c36f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:39.616423 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74e2d78c024849b0af5c0bde1f48c36f" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:39.616571 911 raft_consensus.cc:3060] T 74e2d78c024849b0af5c0bde1f48c36f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:39.618870 776 raft_consensus.cc:2468] T 74e2d78c024849b0af5c0bde1f48c36f P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:39.619716 982 leader_election.cc:304] T 74e2d78c024849b0af5c0bde1f48c36f P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:39.619853 911 raft_consensus.cc:2468] T 74e2d78c024849b0af5c0bde1f48c36f P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:39.620195 2182 raft_consensus.cc:2804] T 74e2d78c024849b0af5c0bde1f48c36f P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:39.620335 2182 raft_consensus.cc:697] T 74e2d78c024849b0af5c0bde1f48c36f P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:39.620500 1167 ts_tablet_manager.cc:1434] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.033s user 0.002s sys 0.000s
I20251025 07:52:39.620577 2182 consensus_queue.cc:237] T 74e2d78c024849b0af5c0bde1f48c36f P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.620795 1167 tablet_bootstrap.cc:492] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:39.622752 2146 ts_tablet_manager.cc:1916] T 887e7c7a5d0e4aaf88f26246306df19c P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:39.623369 1167 tablet_bootstrap.cc:654] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.625159 1612 consensus_queue.cc:1048] T d66ad049d5484228a268b0facb00ecf1 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:39.625826 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 0325d462441349ef81841f1b0cab2a20 (table default.Table_750ecb905839434fa9a1aa72f87394be [id=9ce3ba77c56a4ef09cb3b40ea0e488f6]). Expected version 1 got 0
W20251025 07:52:39.626693 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 4f0c4f0e8f014e59be9e46d2afe0be9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f0c4f0e8f014e59be9e46d2afe0be9d already in progress: creating tablet
W20251025 07:52:39.627507 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 80fe3255ed614270bd8352a2853fbc43 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80fe3255ed614270bd8352a2853fbc43 already in progress: creating tablet
W20251025 07:52:39.628103 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 5b6383c99e9e4c90a374da9ccd830a6f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b6383c99e9e4c90a374da9ccd830a6f already in progress: creating tablet
W20251025 07:52:39.629108 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet ebc824ec4c60411dab1a2ea4a385148e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ebc824ec4c60411dab1a2ea4a385148e already in progress: creating tablet
W20251025 07:52:39.629742 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet ce1ebb1ec14b4801930a72765bca1a7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce1ebb1ec14b4801930a72765bca1a7d already in progress: creating tablet
I20251025 07:52:39.623045 1166 tablet_bootstrap.cc:654] T aa9d9ab8073945d28d52fb02213f201a P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:39.634579 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 91728987303b41fd840eb56f3a311c5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91728987303b41fd840eb56f3a311c5f already in progress: creating tablet
I20251025 07:52:39.634982 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 22f7db660616457b83b24e947a0b0bae (table default.Table_4a86191163f2408f9bcc795cd8d1bbf6 [id=bb316a7e000748ca902a9bfb494802d7]). Expected version 1 got 0
W20251025 07:52:39.635468 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 050a5d3b5f934dd19bf2e221f2fe0780 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 050a5d3b5f934dd19bf2e221f2fe0780 already in progress: creating tablet
I20251025 07:52:39.635922 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c]). Expected version 1 got 0
I20251025 07:52:39.636451 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]). Expected version 1 got 0
I20251025 07:52:39.636732 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236]). Expected version 1 got 0
I20251025 07:52:39.637037 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]). Expected version 3 got 0
I20251025 07:52:39.637900 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_e90d7f5ff13b40878e5b625c97792c7d [id=fc8180c09d7b4e74b2e161f7acb8fe0e]). Expected version 1 got 0
W20251025 07:52:39.638314 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet aaef24ccdfba45fe8204327723df834e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aaef24ccdfba45fe8204327723df834e already in progress: creating tablet
W20251025 07:52:39.638682 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 0f5f6ca6c3cd4d369690e425387a4344 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f5f6ca6c3cd4d369690e425387a4344 already in progress: creating tablet
W20251025 07:52:39.638999 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet d9bfd7c0c3ad4e5b98b6708793d61d41 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9bfd7c0c3ad4e5b98b6708793d61d41 already in progress: creating tablet
W20251025 07:52:39.639216 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet b3a0af893a8947848e12093d0a7f0229 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3a0af893a8947848e12093d0a7f0229 already in progress: creating tablet
W20251025 07:52:39.639446 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 04449cc97f404a19b6a1b20dea074f95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04449cc97f404a19b6a1b20dea074f95 already in progress: creating tablet
W20251025 07:52:39.639642 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet f43ef51ee45d47a99ec732b49d50d3b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f43ef51ee45d47a99ec732b49d50d3b4 already in progress: creating tablet
W20251025 07:52:39.639914 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 7e6544202d4f489d82f22d6073ef240d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e6544202d4f489d82f22d6073ef240d already in progress: creating tablet
I20251025 07:52:39.639952 2169 tablet_replica.cc:333] T d37758f606344127b3db74c87babe69c P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:39.640192 2169 raft_consensus.cc:2243] T d37758f606344127b3db74c87babe69c P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:39.640244 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3]). Expected version 2 got 0
I20251025 07:52:39.640570 2169 raft_consensus.cc:2272] T d37758f606344127b3db74c87babe69c P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:39.641181 2169 ts_tablet_manager.cc:1916] T d37758f606344127b3db74c87babe69c P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:39.648134 1169 tablet_bootstrap.cc:492] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:39.649048 1166 tablet_bootstrap.cc:492] T aa9d9ab8073945d28d52fb02213f201a P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:39.649178 1166 ts_tablet_manager.cc:1403] T aa9d9ab8073945d28d52fb02213f201a P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.055s user 0.002s sys 0.000s
I20251025 07:52:39.649854 1169 tablet_bootstrap.cc:654] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.649888 1166 raft_consensus.cc:359] T aa9d9ab8073945d28d52fb02213f201a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.650069 1166 raft_consensus.cc:385] T aa9d9ab8073945d28d52fb02213f201a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.650154 1166 raft_consensus.cc:740] T aa9d9ab8073945d28d52fb02213f201a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.650421 822 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
I20251025 07:52:39.650426 1166 consensus_queue.cc:260] T aa9d9ab8073945d28d52fb02213f201a P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.651168 1166 ts_tablet_manager.cc:1434] T aa9d9ab8073945d28d52fb02213f201a P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:39.652149 1166 tablet_bootstrap.cc:492] T e54fe7cbcffa40c4b04144b47bee2b48 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:39.653587 1166 tablet_bootstrap.cc:654] T e54fe7cbcffa40c4b04144b47bee2b48 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.657042 823 heartbeater.cc:344] Connected to a master server at 127.0.104.61:42203
I20251025 07:52:39.665866 2105 raft_consensus.cc:3060] T df1248f6dbd94af68c587da7d149e279 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:39.668637 2105 raft_consensus.cc:2749] T df1248f6dbd94af68c587da7d149e279 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:39.669142 850 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:45417, user_credentials={real_user=slave}} blocked reactor thread for 96308.7us
I20251025 07:52:39.670198 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b947ecc352e54f4caa73c6ca97372256" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:39.677161 1169 tablet_bootstrap.cc:492] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:39.677340 1169 ts_tablet_manager.cc:1403] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.002s
I20251025 07:52:39.678119 1169 raft_consensus.cc:359] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.678351 1169 raft_consensus.cc:385] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.678429 1169 raft_consensus.cc:740] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.678678 1169 consensus_queue.cc:260] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.679176 1169 ts_tablet_manager.cc:1434] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:39.679613 1169 tablet_bootstrap.cc:492] T 88bfc23acd274041a9457079c1709d82 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:39.681216 1659 raft_consensus.cc:493] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:39.681375 1659 raft_consensus.cc:515] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.681993 1659 leader_election.cc:290] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:39.682767 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4578ce69a0d4322944e5ce7c8a59831" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:39.683017 911 raft_consensus.cc:2468] T f4578ce69a0d4322944e5ce7c8a59831 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:39.683472 710 leader_election.cc:304] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:39.683826 1659 raft_consensus.cc:2804] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:39.683902 1659 raft_consensus.cc:493] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:39.683956 1659 raft_consensus.cc:3060] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:39.682629 1169 tablet_bootstrap.cc:654] T 88bfc23acd274041a9457079c1709d82 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.683023 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4578ce69a0d4322944e5ce7c8a59831" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:39.686345 712 leader_election.cc:343] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:39.692193 1659 raft_consensus.cc:515] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.693109 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4578ce69a0d4322944e5ce7c8a59831" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:39.693343 911 raft_consensus.cc:3060] T f4578ce69a0d4322944e5ce7c8a59831 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:39.693617 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4578ce69a0d4322944e5ce7c8a59831" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:39.694157 712 leader_election.cc:343] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:39.694341 1659 leader_election.cc:290] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
W20251025 07:52:39.695850 850 leader_election.cc:343] T b947ecc352e54f4caa73c6ca97372256 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:39.695995 850 leader_election.cc:304] T b947ecc352e54f4caa73c6ca97372256 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:39.696203 2105 raft_consensus.cc:2749] T b947ecc352e54f4caa73c6ca97372256 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:39.696334 911 raft_consensus.cc:2468] T f4578ce69a0d4322944e5ce7c8a59831 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:39.659622 2146 ts_tablet_manager.cc:1929] T 887e7c7a5d0e4aaf88f26246306df19c P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:39.696745 710 leader_election.cc:304] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:39.697130 1659 raft_consensus.cc:2804] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:39.697256 1659 raft_consensus.cc:697] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:39.697563 1659 consensus_queue.cc:237] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.699318 960 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
I20251025 07:52:39.701541 2146 log.cc:1199] T 887e7c7a5d0e4aaf88f26246306df19c P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/887e7c7a5d0e4aaf88f26246306df19c
I20251025 07:52:39.702010 2146 ts_tablet_manager.cc:1950] T 887e7c7a5d0e4aaf88f26246306df19c P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:39.702379 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 71) took 1273 ms. Trace:
I20251025 07:52:39.703306 2146 rpcz_store.cc:276] 1025 07:52:38.428805 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.431230 (+ 2425us) service_pool.cc:225] Handling call
1025 07:52:39.597275 (+1166045us) ts_tablet_manager.cc:1151] Deleting tablet 887e7c7a5d0e4aaf88f26246306df19c
1025 07:52:39.597286 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:39.633088 (+ 35802us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:39.659507 (+ 26419us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:39.702360 (+ 42853us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2320,"tablet-delete.queue_time_us":1165665}
I20251025 07:52:39.704386 2166 ts_tablet_manager.cc:1929] T 34315f64b89148efbf1f20a43d5c5efb P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:39.704522 2166 log.cc:1199] T 34315f64b89148efbf1f20a43d5c5efb P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/34315f64b89148efbf1f20a43d5c5efb
I20251025 07:52:39.704834 2166 ts_tablet_manager.cc:1950] T 34315f64b89148efbf1f20a43d5c5efb P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:39.705188 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 887e7c7a5d0e4aaf88f26246306df19c (table default.Table_2c1cdea0079240b889637e5d8d13da95 [id=8e07d0ad2db84d1b8a3a3a3efa2d7f69]) successfully deleted
I20251025 07:52:39.705258 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 60) took 1192 ms. Trace:
I20251025 07:52:39.705332 2166 rpcz_store.cc:276] 1025 07:52:38.512816 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.515800 (+ 2984us) service_pool.cc:225] Handling call
1025 07:52:39.557053 (+1041253us) ts_tablet_manager.cc:1151] Deleting tablet 34315f64b89148efbf1f20a43d5c5efb
1025 07:52:39.557063 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:39.609960 (+ 52897us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:39.704268 (+ 94308us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:39.705242 (+ 974us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1041073}
I20251025 07:52:39.705673 2300 raft_consensus.cc:493] T 92aa605fabbc4257a6a204c087e7cc7f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:39.705731 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 73) took 1274 ms. Trace:
I20251025 07:52:39.705793 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 34315f64b89148efbf1f20a43d5c5efb (table default.Table_4b3d7b3a731942189cdc214938a45820 [id=88f1e8b1c5fa4913a5f7b5b2c6cbf6ec]) successfully deleted
I20251025 07:52:39.705801 2146 rpcz_store.cc:276] 1025 07:52:38.431165 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.431686 (+ 521us) service_pool.cc:225] Handling call
1025 07:52:39.705603 (+1273917us) ts_tablet_manager.cc:1151] Deleting tablet e4da89f863cc43f3bf6df908bb9dc6bf
1025 07:52:39.705613 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:39.705716 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1273716}
I20251025 07:52:39.705840 2300 raft_consensus.cc:515] T 92aa605fabbc4257a6a204c087e7cc7f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.706213 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 61) took 1190 ms. Trace:
W20251025 07:52:39.706223 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet e4da89f863cc43f3bf6df908bb9dc6bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4da89f863cc43f3bf6df908bb9dc6bf already in progress: creating tablet
I20251025 07:52:39.706281 2166 rpcz_store.cc:276] 1025 07:52:38.516201 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.516338 (+ 137us) service_pool.cc:225] Handling call
1025 07:52:39.706059 (+1189721us) ts_tablet_manager.cc:1151] Deleting tablet af743ee344e543a0aa6da3cd7a30e467
1025 07:52:39.706069 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:39.706194 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1189596}
W20251025 07:52:39.706674 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet af743ee344e543a0aa6da3cd7a30e467 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af743ee344e543a0aa6da3cd7a30e467 already in progress: creating tablet
I20251025 07:52:39.707209 1166 tablet_bootstrap.cc:492] T e54fe7cbcffa40c4b04144b47bee2b48 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
W20251025 07:52:39.707288 982 leader_election.cc:343] T 92aa605fabbc4257a6a204c087e7cc7f P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: 92aa605fabbc4257a6a204c087e7cc7f
I20251025 07:52:39.707439 2166 tablet_replica.cc:333] T 08da9d2687704725aacd11cd92c6a9f9 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:39.707639 2166 raft_consensus.cc:2243] T 08da9d2687704725aacd11cd92c6a9f9 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:39.708369 2300 leader_election.cc:290] T 92aa605fabbc4257a6a204c087e7cc7f P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:39.708772 1166 ts_tablet_manager.cc:1403] T e54fe7cbcffa40c4b04144b47bee2b48 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.057s user 0.005s sys 0.001s
I20251025 07:52:39.708194 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "92aa605fabbc4257a6a204c087e7cc7f" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:39.707808 2166 raft_consensus.cc:2272] T 08da9d2687704725aacd11cd92c6a9f9 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:39.709080 902 raft_consensus.cc:2468] T 92aa605fabbc4257a6a204c087e7cc7f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:39.709491 983 leader_election.cc:304] T 92aa605fabbc4257a6a204c087e7cc7f P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b
I20251025 07:52:39.709913 2300 raft_consensus.cc:2804] T 92aa605fabbc4257a6a204c087e7cc7f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:39.710055 2300 raft_consensus.cc:493] T 92aa605fabbc4257a6a204c087e7cc7f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:39.710194 2300 raft_consensus.cc:3060] T 92aa605fabbc4257a6a204c087e7cc7f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:39.713320 2300 raft_consensus.cc:515] T 92aa605fabbc4257a6a204c087e7cc7f P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.714277 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 87268e27540c4ec58ce1e62df819f19d (table default.Table_350383f0e98947eba77186e74ad88781 [id=96b0e47c666c4d8f982741aea7b17ae7]). Expected version 2 got 0
I20251025 07:52:39.714637 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 8f2c8582fed34c848d9903b8ad8b4057 (table default.Table_438f67f3dc62465999a917986922b3ed [id=7c7fcd265e624d7fab74661db505d9c3]). Expected version 1 got 0
W20251025 07:52:39.714689 982 leader_election.cc:343] T 92aa605fabbc4257a6a204c087e7cc7f P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: 92aa605fabbc4257a6a204c087e7cc7f
I20251025 07:52:39.715162 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet ab42909afc524f2fa3c1041495d13a59 (table default.Table_7eb8424d5192485da1a2d67bfdb2bbbf [id=40c29df99590408cbd5c583d4fc741eb]). Expected version 2 got 0
I20251025 07:52:39.715696 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 36262ab8c6964dd684c01b4db6219a4f (table default.Table_76158f8cf82e4cdb9a90580e46aaea48 [id=f249f30497bc450c887d659721469de8]). Expected version 1 got 0
I20251025 07:52:39.716017 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 2c50ec95e48e49b59a76fac4fc862d3d (table default.Table_32b54e82cbc84d39a0020826f8737d62 [id=c28b0b6acd61490caecfdaab803a9d92]). Expected version 2 got 0
I20251025 07:52:39.716241 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3]). Expected version 1 got 0
I20251025 07:52:39.716588 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet b97e39aaac3048ed93dbfe873b757254 (table default.Table_99fd36369f9a4e0fb14f234c98635203 [id=c8059ab6cec146efa631d06799d969a9]). Expected version 1 got 0
I20251025 07:52:39.716905 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "92aa605fabbc4257a6a204c087e7cc7f" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:39.717089 911 raft_consensus.cc:3060] T 92aa605fabbc4257a6a204c087e7cc7f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:39.719956 911 raft_consensus.cc:2468] T 92aa605fabbc4257a6a204c087e7cc7f P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:39.720371 983 leader_election.cc:304] T 92aa605fabbc4257a6a204c087e7cc7f P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b
I20251025 07:52:39.720649 1671 raft_consensus.cc:2804] T 92aa605fabbc4257a6a204c087e7cc7f P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:39.720700 2300 leader_election.cc:290] T 92aa605fabbc4257a6a204c087e7cc7f P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:39.720813 1671 raft_consensus.cc:697] T 92aa605fabbc4257a6a204c087e7cc7f P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:39.708088 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 74) took 1276 ms. Trace:
I20251025 07:52:39.721076 1671 consensus_queue.cc:237] T 92aa605fabbc4257a6a204c087e7cc7f P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.721029 2146 rpcz_store.cc:276] 1025 07:52:38.432053 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.432223 (+ 170us) service_pool.cc:225] Handling call
1025 07:52:39.707981 (+1275758us) ts_tablet_manager.cc:1151] Deleting tablet 4f7c15529be54a11961f385a953b0b03
1025 07:52:39.707989 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:39.708072 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1275578}
W20251025 07:52:39.721534 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 4f7c15529be54a11961f385a953b0b03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f7c15529be54a11961f385a953b0b03 already in progress: creating tablet
I20251025 07:52:39.722697 1166 raft_consensus.cc:359] T e54fe7cbcffa40c4b04144b47bee2b48 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.722863 1166 raft_consensus.cc:385] T e54fe7cbcffa40c4b04144b47bee2b48 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.722920 1166 raft_consensus.cc:740] T e54fe7cbcffa40c4b04144b47bee2b48 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.723117 1166 consensus_queue.cc:260] T e54fe7cbcffa40c4b04144b47bee2b48 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.724727 2166 ts_tablet_manager.cc:1916] T 08da9d2687704725aacd11cd92c6a9f9 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:39.726230 2124 raft_consensus.cc:493] T 88bfc23acd274041a9457079c1709d82 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:39.726393 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 75) took 1294 ms. Trace:
I20251025 07:52:39.726411 2124 raft_consensus.cc:515] T 88bfc23acd274041a9457079c1709d82 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.726475 2146 rpcz_store.cc:276] 1025 07:52:38.432168 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.432476 (+ 308us) service_pool.cc:225] Handling call
1025 07:52:39.726250 (+1293774us) ts_tablet_manager.cc:1151] Deleting tablet 72dc084f8efc45478771c247e07a16f4
1025 07:52:39.726261 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:39.726375 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1293614}
W20251025 07:52:39.726945 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 72dc084f8efc45478771c247e07a16f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72dc084f8efc45478771c247e07a16f4 already in progress: creating tablet
I20251025 07:52:39.727182 2124 leader_election.cc:290] T 88bfc23acd274041a9457079c1709d82 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:39.727278 2146 tablet_replica.cc:333] T be5ab0db6ac34c2c9a370445e251eb46 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:39.727456 2146 raft_consensus.cc:2243] T be5ab0db6ac34c2c9a370445e251eb46 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:39.728096 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88bfc23acd274041a9457079c1709d82" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:39.729169 712 leader_election.cc:343] T 88bfc23acd274041a9457079c1709d82 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:39.729491 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88bfc23acd274041a9457079c1709d82" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:39.729911 710 leader_election.cc:343] T 88bfc23acd274041a9457079c1709d82 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:39.730015 710 leader_election.cc:304] T 88bfc23acd274041a9457079c1709d82 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:39.730459 2124 raft_consensus.cc:2749] T 88bfc23acd274041a9457079c1709d82 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:39.734934 1167 tablet_bootstrap.cc:492] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:39.735117 1167 ts_tablet_manager.cc:1403] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.114s user 0.000s sys 0.004s
I20251025 07:52:39.735900 1167 raft_consensus.cc:359] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:39.736114 1167 raft_consensus.cc:385] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.736204 1167 raft_consensus.cc:740] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.736459 1167 consensus_queue.cc:260] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:39.737309 1167 ts_tablet_manager.cc:1434] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:39.737684 1167 tablet_bootstrap.cc:492] T 4b28215f61574cf48c447dd01955ac77 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:39.739420 1167 tablet_bootstrap.cc:654] T 4b28215f61574cf48c447dd01955ac77 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.743588 1095 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
I20251025 07:52:39.716974 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 8e6d76b1a97945cfb6c3055e1b622f8a (table default.Table_10e57b42f97544208675708b1e028846 [id=f5c1be203e2b4831a025871cf8a2ef17]). Expected version 2 got 0
I20251025 07:52:39.746307 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet b17d2a9ad6be49d29f2413666fba7e69 (table default.Table_fcddb31989ee4236a4981c12779b3f5d [id=f3794fa081364666b944c412415bdba0]). Expected version 3 got 0
I20251025 07:52:39.746600 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 292fe04055954afd8c08ac2636b68191 (table default.Table_9fa69b10b5e842639fc0b0a58cd9b6a5 [id=a253ce0253bd4da4814741c482193d8f]). Expected version 1 got 0
I20251025 07:52:39.747274 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 509c94f2a422493ea4390ecee576db72 (table default.Table_3f081b7bb4e1420c8121ea4115439f3d [id=696a1285fbfe4038a8b1b38708c8ea49]). Expected version 1 got 0
I20251025 07:52:39.771422 2117 raft_consensus.cc:493] T ebf3a2a926754305b5bd95fcebb25d29 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:39.771582 2117 raft_consensus.cc:515] T ebf3a2a926754305b5bd95fcebb25d29 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.772019 1659 raft_consensus.cc:493] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:39.772220 2117 leader_election.cc:290] T ebf3a2a926754305b5bd95fcebb25d29 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:39.772558 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebf3a2a926754305b5bd95fcebb25d29" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:39.773074 710 leader_election.cc:343] T ebf3a2a926754305b5bd95fcebb25d29 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:39.773247 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebf3a2a926754305b5bd95fcebb25d29" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:39.773257 1659 raft_consensus.cc:515] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:39.773818 712 leader_election.cc:343] T ebf3a2a926754305b5bd95fcebb25d29 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:39.773914 712 leader_election.cc:304] T ebf3a2a926754305b5bd95fcebb25d29 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:39.774566 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9bfd7c0c3ad4e5b98b6708793d61d41" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:39.774804 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9bfd7c0c3ad4e5b98b6708793d61d41" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:39.774995 710 leader_election.cc:343] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:39.775161 1659 leader_election.cc:290] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:39.775292 1659 raft_consensus.cc:2749] T ebf3a2a926754305b5bd95fcebb25d29 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:39.775329 712 leader_election.cc:343] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:39.775416 712 leader_election.cc:304] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:39.775749 2124 raft_consensus.cc:2749] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:39.786558 2182 consensus_queue.cc:1048] T fa998b9ca6724c4cb1eb68a3eecb0b11 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:39.788735 2300 raft_consensus.cc:493] T 6a58931e7cd7454fb583f1c70cd2ad15 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:39.788890 2300 raft_consensus.cc:515] T 6a58931e7cd7454fb583f1c70cd2ad15 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.789667 2300 leader_election.cc:290] T 6a58931e7cd7454fb583f1c70cd2ad15 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:39.791410 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a58931e7cd7454fb583f1c70cd2ad15" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:39.791574 773 raft_consensus.cc:2468] T 6a58931e7cd7454fb583f1c70cd2ad15 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:39.792315 982 leader_election.cc:304] T 6a58931e7cd7454fb583f1c70cd2ad15 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:39.793025 2182 raft_consensus.cc:2804] T 6a58931e7cd7454fb583f1c70cd2ad15 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:39.793123 2182 raft_consensus.cc:493] T 6a58931e7cd7454fb583f1c70cd2ad15 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:39.793200 2182 raft_consensus.cc:3060] T 6a58931e7cd7454fb583f1c70cd2ad15 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:39.796036 2182 raft_consensus.cc:515] T 6a58931e7cd7454fb583f1c70cd2ad15 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.798645 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a58931e7cd7454fb583f1c70cd2ad15" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:39.798810 773 raft_consensus.cc:3060] T 6a58931e7cd7454fb583f1c70cd2ad15 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:39.801707 773 raft_consensus.cc:2468] T 6a58931e7cd7454fb583f1c70cd2ad15 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:39.802042 2300 raft_consensus.cc:493] T fe5033e5ff404a5ebc6c8443d7548327 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:39.802194 2300 raft_consensus.cc:515] T fe5033e5ff404a5ebc6c8443d7548327 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.802780 2300 leader_election.cc:290] T fe5033e5ff404a5ebc6c8443d7548327 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:39.803565 982 leader_election.cc:304] T 6a58931e7cd7454fb583f1c70cd2ad15 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:39.804168 1671 raft_consensus.cc:2804] T 6a58931e7cd7454fb583f1c70cd2ad15 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:39.804309 1671 raft_consensus.cc:697] T 6a58931e7cd7454fb583f1c70cd2ad15 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:39.804545 1671 consensus_queue.cc:237] T 6a58931e7cd7454fb583f1c70cd2ad15 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.806962 1167 tablet_bootstrap.cc:492] T 4b28215f61574cf48c447dd01955ac77 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:39.807153 1167 ts_tablet_manager.cc:1403] T 4b28215f61574cf48c447dd01955ac77 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.070s user 0.003s sys 0.000s
I20251025 07:52:39.808497 1167 raft_consensus.cc:359] T 4b28215f61574cf48c447dd01955ac77 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.808760 1167 raft_consensus.cc:385] T 4b28215f61574cf48c447dd01955ac77 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.808856 1167 raft_consensus.cc:740] T 4b28215f61574cf48c447dd01955ac77 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.808897 2182 leader_election.cc:290] T 6a58931e7cd7454fb583f1c70cd2ad15 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:39.809180 1167 consensus_queue.cc:260] T 4b28215f61574cf48c447dd01955ac77 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.809859 1167 ts_tablet_manager.cc:1434] T 4b28215f61574cf48c447dd01955ac77 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251025 07:52:39.810240 1167 tablet_bootstrap.cc:492] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:39.813480 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5033e5ff404a5ebc6c8443d7548327" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:39.813760 1167 tablet_bootstrap.cc:654] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.813881 773 raft_consensus.cc:2468] T fe5033e5ff404a5ebc6c8443d7548327 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:39.814713 982 leader_election.cc:304] T fe5033e5ff404a5ebc6c8443d7548327 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:39.815450 2182 raft_consensus.cc:2804] T fe5033e5ff404a5ebc6c8443d7548327 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:39.815631 2182 raft_consensus.cc:493] T fe5033e5ff404a5ebc6c8443d7548327 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:39.815757 2182 raft_consensus.cc:3060] T fe5033e5ff404a5ebc6c8443d7548327 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:39.817155 1017 tablet_service.cc:1505] Processing CreateTablet for tablet c0ec5520674c457e94e51dd7481b4d5c (DEFAULT_TABLE table=default.Table_4f427d19e85b4ce19136b8a82cc5ef76 [id=88cbe93f753e4d5abc84b7c9e2b46fe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:39.824395 2166 ts_tablet_manager.cc:1929] T 08da9d2687704725aacd11cd92c6a9f9 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:39.829375 2182 raft_consensus.cc:515] T fe5033e5ff404a5ebc6c8443d7548327 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.831351 1018 tablet_service.cc:1505] Processing CreateTablet for tablet 9d829e6dd1224ad69e092476c281478c (DEFAULT_TABLE table=default.Table_4b11b64671124da3aa3ec46fe2cb9983 [id=1defec03c1c74085b3bef3ad6c25c041]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:39.831611 2182 leader_election.cc:290] T fe5033e5ff404a5ebc6c8443d7548327 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:39.825896 745 tablet_service.cc:1505] Processing CreateTablet for tablet c0ec5520674c457e94e51dd7481b4d5c (DEFAULT_TABLE table=default.Table_4f427d19e85b4ce19136b8a82cc5ef76 [id=88cbe93f753e4d5abc84b7c9e2b46fe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:39.833657 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0ec5520674c457e94e51dd7481b4d5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:39.833880 754 tablet_service.cc:1505] Processing CreateTablet for tablet 9d829e6dd1224ad69e092476c281478c (DEFAULT_TABLE table=default.Table_4b11b64671124da3aa3ec46fe2cb9983 [id=1defec03c1c74085b3bef3ad6c25c041]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:39.834232 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d829e6dd1224ad69e092476c281478c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:39.843117 2166 log.cc:1199] T 08da9d2687704725aacd11cd92c6a9f9 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/08da9d2687704725aacd11cd92c6a9f9
I20251025 07:52:39.831033 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5033e5ff404a5ebc6c8443d7548327" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:39.844713 773 raft_consensus.cc:3060] T fe5033e5ff404a5ebc6c8443d7548327 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:39.844990 2166 ts_tablet_manager.cc:1950] T 08da9d2687704725aacd11cd92c6a9f9 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:39.845427 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 62) took 1329 ms. Trace:
I20251025 07:52:39.845582 2166 rpcz_store.cc:276] 1025 07:52:38.516303 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.516490 (+ 187us) service_pool.cc:225] Handling call
1025 07:52:39.706879 (+1190389us) ts_tablet_manager.cc:1151] Deleting tablet 08da9d2687704725aacd11cd92c6a9f9
1025 07:52:39.706887 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:39.771531 (+ 64644us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:39.824271 (+ 52740us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:39.845408 (+ 21137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1190284}
I20251025 07:52:39.846201 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 08da9d2687704725aacd11cd92c6a9f9 (table default.Table_3055bcb146aa43d883f4e7208f6c6638 [id=78236f2583524e1fa95840afb7aef43f]) successfully deleted
I20251025 07:52:39.846596 2166 tablet_replica.cc:333] T d66ad049d5484228a268b0facb00ecf1 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:39.846890 2166 raft_consensus.cc:2243] T d66ad049d5484228a268b0facb00ecf1 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:39.856582 2166 raft_consensus.cc:2272] T d66ad049d5484228a268b0facb00ecf1 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:39.847705 773 raft_consensus.cc:2468] T fe5033e5ff404a5ebc6c8443d7548327 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:39.857470 982 leader_election.cc:304] T fe5033e5ff404a5ebc6c8443d7548327 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:39.857779 2300 raft_consensus.cc:2804] T fe5033e5ff404a5ebc6c8443d7548327 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:39.857899 2300 raft_consensus.cc:697] T fe5033e5ff404a5ebc6c8443d7548327 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:39.858271 2300 consensus_queue.cc:237] T fe5033e5ff404a5ebc6c8443d7548327 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.859321 2146 raft_consensus.cc:2272] T be5ab0db6ac34c2c9a370445e251eb46 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:39.859673 1169 tablet_bootstrap.cc:492] T 88bfc23acd274041a9457079c1709d82 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:39.859818 1169 ts_tablet_manager.cc:1403] T 88bfc23acd274041a9457079c1709d82 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.180s user 0.033s sys 0.049s
I20251025 07:52:39.860052 2146 ts_tablet_manager.cc:1916] T be5ab0db6ac34c2c9a370445e251eb46 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:39.860589 1169 raft_consensus.cc:359] T 88bfc23acd274041a9457079c1709d82 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.860759 1169 raft_consensus.cc:385] T 88bfc23acd274041a9457079c1709d82 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.860827 1169 raft_consensus.cc:740] T 88bfc23acd274041a9457079c1709d82 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.861027 1169 consensus_queue.cc:260] T 88bfc23acd274041a9457079c1709d82 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.861845 1169 ts_tablet_manager.cc:1434] T 88bfc23acd274041a9457079c1709d82 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:39.862195 1169 tablet_bootstrap.cc:492] T f4578ce69a0d4322944e5ce7c8a59831 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:39.863112 883 tablet_service.cc:1505] Processing CreateTablet for tablet 9d829e6dd1224ad69e092476c281478c (DEFAULT_TABLE table=default.Table_4b11b64671124da3aa3ec46fe2cb9983 [id=1defec03c1c74085b3bef3ad6c25c041]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:39.863588 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d829e6dd1224ad69e092476c281478c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:39.864265 2245 raft_consensus.cc:493] T aa9d9ab8073945d28d52fb02213f201a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:39.864401 2245 raft_consensus.cc:515] T aa9d9ab8073945d28d52fb02213f201a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.865082 2245 leader_election.cc:290] T aa9d9ab8073945d28d52fb02213f201a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:39.865202 2245 raft_consensus.cc:493] T 6a58931e7cd7454fb583f1c70cd2ad15 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:39.865330 2245 raft_consensus.cc:515] T 6a58931e7cd7454fb583f1c70cd2ad15 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.865831 2245 leader_election.cc:290] T 6a58931e7cd7454fb583f1c70cd2ad15 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:39.866554 894 tablet_service.cc:1505] Processing CreateTablet for tablet c0ec5520674c457e94e51dd7481b4d5c (DEFAULT_TABLE table=default.Table_4f427d19e85b4ce19136b8a82cc5ef76 [id=88cbe93f753e4d5abc84b7c9e2b46fe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:39.866936 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0ec5520674c457e94e51dd7481b4d5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:39.867708 1167 tablet_bootstrap.cc:492] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:39.867830 1167 ts_tablet_manager.cc:1403] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.000s
I20251025 07:52:39.868578 1167 raft_consensus.cc:359] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:39.868726 1167 raft_consensus.cc:385] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.868767 1167 raft_consensus.cc:740] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.869004 1167 consensus_queue.cc:260] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:39.869514 1167 ts_tablet_manager.cc:1434] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:39.869777 1167 tablet_bootstrap.cc:492] T 6dcab53a020847a1aa27930265ba0205 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:39.870613 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a58931e7cd7454fb583f1c70cd2ad15" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:39.870790 910 raft_consensus.cc:2468] T 6a58931e7cd7454fb583f1c70cd2ad15 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:39.871234 907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5033e5ff404a5ebc6c8443d7548327" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:39.871425 1167 tablet_bootstrap.cc:654] T 6dcab53a020847a1aa27930265ba0205 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.871534 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a58931e7cd7454fb583f1c70cd2ad15" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:39.871673 903 raft_consensus.cc:3060] T 6a58931e7cd7454fb583f1c70cd2ad15 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:39.872048 1169 tablet_bootstrap.cc:654] T f4578ce69a0d4322944e5ce7c8a59831 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.874622 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe5033e5ff404a5ebc6c8443d7548327" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:39.876386 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa9d9ab8073945d28d52fb02213f201a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:39.876731 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a58931e7cd7454fb583f1c70cd2ad15" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:39.879271 850 leader_election.cc:343] T aa9d9ab8073945d28d52fb02213f201a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:39.891356 850 consensus_peers.cc:597] T 467324e2432f4caca148951329caa292 P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 467324e2432f4caca148951329caa292. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:39.895144 1167 tablet_bootstrap.cc:492] T 6dcab53a020847a1aa27930265ba0205 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:39.895330 1167 ts_tablet_manager.cc:1403] T 6dcab53a020847a1aa27930265ba0205 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20251025 07:52:39.896303 1167 raft_consensus.cc:359] T 6dcab53a020847a1aa27930265ba0205 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:39.896530 1167 raft_consensus.cc:385] T 6dcab53a020847a1aa27930265ba0205 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.896612 1167 raft_consensus.cc:740] T 6dcab53a020847a1aa27930265ba0205 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.896859 1167 consensus_queue.cc:260] T 6dcab53a020847a1aa27930265ba0205 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:39.897529 1167 ts_tablet_manager.cc:1434] T 6dcab53a020847a1aa27930265ba0205 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 07:52:39.897815 903 raft_consensus.cc:2468] T 6a58931e7cd7454fb583f1c70cd2ad15 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:39.897898 1167 tablet_bootstrap.cc:492] T 4c155aa7fa5b4d68bbab1918b8432372 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:39.900349 1167 tablet_bootstrap.cc:654] T 4c155aa7fa5b4d68bbab1918b8432372 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.902148 2166 ts_tablet_manager.cc:1916] T d66ad049d5484228a268b0facb00ecf1 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:39.904649 2117 raft_consensus.cc:493] T 943ec703e65f4fd3b0c690e08c0bd226 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:39.904765 2117 raft_consensus.cc:515] T 943ec703e65f4fd3b0c690e08c0bd226 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.905318 2117 leader_election.cc:290] T 943ec703e65f4fd3b0c690e08c0bd226 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
W20251025 07:52:39.911592 983 consensus_peers.cc:597] T 8956ad28b35d45f28c683ac055bba441 P db42be7ad3c741a0aa427f27e245993b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8956ad28b35d45f28c683ac055bba441. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:39.912407 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f]). Expected version 1 got 0
I20251025 07:52:39.912791 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 4c155aa7fa5b4d68bbab1918b8432372 (table default.Table_6c5a24747ba14352b0285a41ec2985a4 [id=28bbaad474164a8c8c97a76bb8081791]). Expected version 2 got 0
I20251025 07:52:39.913161 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7]). Expected version 1 got 0
I20251025 07:52:39.913357 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 04d27d54fdb7483b869408af65ff85c9 (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]). Expected version 2 got 0
W20251025 07:52:39.913465 983 leader_election.cc:343] T fe5033e5ff404a5ebc6c8443d7548327 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:39.914078 983 leader_election.cc:343] T fe5033e5ff404a5ebc6c8443d7548327 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:39.914323 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "943ec703e65f4fd3b0c690e08c0bd226" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:39.914373 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet cb829649f73d4d3f8862d268c953bd86 (table default.Table_ba4b8cc7b930482690b7e192e0edff12 [id=7944225d3ce7439b881bfb9289d9f040]). Expected version 1 got 0
I20251025 07:52:39.914510 1049 raft_consensus.cc:2468] T 943ec703e65f4fd3b0c690e08c0bd226 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:39.914708 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 872cb7512d104827b1f1f186d33a4a44 (table default.Table_d5c65dce8b1b4214ba519d964012bfc0 [id=d4be274b36a845a99a4b96b8846248fd]). Expected version 1 got 0
I20251025 07:52:39.915195 712 leader_election.cc:304] T 943ec703e65f4fd3b0c690e08c0bd226 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:39.915711 2124 raft_consensus.cc:2804] T 943ec703e65f4fd3b0c690e08c0bd226 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:39.915783 2124 raft_consensus.cc:493] T 943ec703e65f4fd3b0c690e08c0bd226 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:39.915849 2124 raft_consensus.cc:3060] T 943ec703e65f4fd3b0c690e08c0bd226 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:39.916103 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet aa9d9ab8073945d28d52fb02213f201a (table default.Table_4791672658f043f6b97b9320a531ebb5 [id=dfd20cf9c3e541c4abe23d59e0aa7f2a]). Expected version 1 got 0
I20251025 07:52:39.916395 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]). Expected version 1 got 0
I20251025 07:52:39.916570 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet ef9a2c9f2ccf432ab835396020872208 (table default.Table_2b9b772d4c234ac286bef3658482e490 [id=409fab60f9d744798e6f92b868f0f7c2]). Expected version 1 got 0
I20251025 07:52:39.916772 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]). Expected version 1 got 0
I20251025 07:52:39.917052 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_a6c869949b744bc98efb735b0e73e7aa [id=bf77b5ff6f1048dd996d948437f9ff5e]). Expected version 2 got 0
I20251025 07:52:39.917553 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet ef1854d35a3047e3a9fbe0166ab1295a (table default.Table_088aad2a46b041bfa29ed8e6929760b1 [id=834669c54dae449790818684fd8659bc]). Expected version 2 got 0
I20251025 07:52:39.917743 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet cd23b8094d7e472eba6bd170248858a9 (table default.Table_ca479cb5c1184c5391b7e1c9d8a08937 [id=d82b3e1317434cf08a0d51e8a9427224]). Expected version 3 got 2
I20251025 07:52:39.922744 1659 raft_consensus.cc:493] T 80fe3255ed614270bd8352a2853fbc43 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:39.922876 1659 raft_consensus.cc:515] T 80fe3255ed614270bd8352a2853fbc43 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.923424 1659 leader_election.cc:290] T 80fe3255ed614270bd8352a2853fbc43 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:39.925561 2124 raft_consensus.cc:515] T 943ec703e65f4fd3b0c690e08c0bd226 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.925865 1167 tablet_bootstrap.cc:492] T 4c155aa7fa5b4d68bbab1918b8432372 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:39.926030 1167 ts_tablet_manager.cc:1403] T 4c155aa7fa5b4d68bbab1918b8432372 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.000s
I20251025 07:52:39.926419 2124 leader_election.cc:290] T 943ec703e65f4fd3b0c690e08c0bd226 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:39.927214 1167 raft_consensus.cc:359] T 4c155aa7fa5b4d68bbab1918b8432372 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.927441 1167 raft_consensus.cc:385] T 4c155aa7fa5b4d68bbab1918b8432372 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:39.927513 1167 raft_consensus.cc:740] T 4c155aa7fa5b4d68bbab1918b8432372 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:39.928066 1167 consensus_queue.cc:260] T 4c155aa7fa5b4d68bbab1918b8432372 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.906087 2146 ts_tablet_manager.cc:1929] T be5ab0db6ac34c2c9a370445e251eb46 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:39.928385 2146 log.cc:1199] T be5ab0db6ac34c2c9a370445e251eb46 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/be5ab0db6ac34c2c9a370445e251eb46
I20251025 07:52:39.928798 2146 ts_tablet_manager.cc:1950] T be5ab0db6ac34c2c9a370445e251eb46 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:39.928905 1167 ts_tablet_manager.cc:1434] T 4c155aa7fa5b4d68bbab1918b8432372 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251025 07:52:39.929246 1167 tablet_bootstrap.cc:492] T 66f0d398e59a4175aa9fe90715124d1a P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:39.929579 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 76) took 1496 ms. Trace:
I20251025 07:52:39.929667 2146 rpcz_store.cc:276] 1025 07:52:38.432814 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.432851 (+ 37us) service_pool.cc:225] Handling call
1025 07:52:39.727231 (+1294380us) ts_tablet_manager.cc:1151] Deleting tablet be5ab0db6ac34c2c9a370445e251eb46
1025 07:52:39.727241 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:39.886869 (+159628us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:39.905989 (+ 19120us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:39.929559 (+ 23570us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":131662,"tablet-delete.queue_time_us":1294212}
I20251025 07:52:39.930184 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet be5ab0db6ac34c2c9a370445e251eb46 (table default.Table_8faecb0cb37949a39f4ef5d9d2740c3d [id=f03dd08b39184812884db77faf4bb39e]) successfully deleted
I20251025 07:52:39.930621 2146 tablet_replica.cc:333] T b772025852794a49b072f37cccbd3291 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:39.930795 2146 raft_consensus.cc:2243] T b772025852794a49b072f37cccbd3291 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:39.931106 2146 raft_consensus.cc:2272] T b772025852794a49b072f37cccbd3291 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:39.931625 1167 tablet_bootstrap.cc:654] T 66f0d398e59a4175aa9fe90715124d1a P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.931641 2146 ts_tablet_manager.cc:1916] T b772025852794a49b072f37cccbd3291 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:39.934497 1166 ts_tablet_manager.cc:1434] T e54fe7cbcffa40c4b04144b47bee2b48 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.212s user 0.000s sys 0.001s
I20251025 07:52:39.934834 1166 tablet_bootstrap.cc:492] T ebf3a2a926754305b5bd95fcebb25d29 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:39.936486 1166 tablet_bootstrap.cc:654] T ebf3a2a926754305b5bd95fcebb25d29 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:39.918073 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet bc2d16b33ca8426abbd7bd38339c9f1d (table default.Table_4ea985df4c5b415c9570c64506a98cc4 [id=98d2b7797e33485e8872054aa2eb8ec9]). Expected version 1 got 0
I20251025 07:52:39.938514 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e]). Expected version 1 got 0
I20251025 07:52:39.938798 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet ab42909afc524f2fa3c1041495d13a59 (table default.Table_7eb8424d5192485da1a2d67bfdb2bbbf [id=40c29df99590408cbd5c583d4fc741eb]). Expected version 2 got 0
I20251025 07:52:39.938980 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "80fe3255ed614270bd8352a2853fbc43" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:39.939677 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "943ec703e65f4fd3b0c690e08c0bd226" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:39.939855 1049 raft_consensus.cc:3060] T 943ec703e65f4fd3b0c690e08c0bd226 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:39.941803 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 509c94f2a422493ea4390ecee576db72 (table default.Table_3f081b7bb4e1420c8121ea4115439f3d [id=696a1285fbfe4038a8b1b38708c8ea49]). Expected version 1 got 0
W20251025 07:52:39.942418 712 leader_election.cc:343] T 80fe3255ed614270bd8352a2853fbc43 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:39.942782 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525]). Expected version 1 got 0
I20251025 07:52:39.943357 1049 raft_consensus.cc:2468] T 943ec703e65f4fd3b0c690e08c0bd226 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:39.943727 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84]). Expected version 1 got 0
I20251025 07:52:39.943773 712 leader_election.cc:304] T 943ec703e65f4fd3b0c690e08c0bd226 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:39.944098 2124 raft_consensus.cc:2804] T 943ec703e65f4fd3b0c690e08c0bd226 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:39.944202 2124 raft_consensus.cc:697] T 943ec703e65f4fd3b0c690e08c0bd226 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:39.944415 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 5dab439bc50d4e869548154e1d222ad6 (table default.Table_bd82d871f84943f4baa967f5c0ec74a3 [id=45325c188eac4d34bb016d01de091f87]). Expected version 2 got 0
I20251025 07:52:39.944428 2124 consensus_queue.cc:237] T 943ec703e65f4fd3b0c690e08c0bd226 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.944734 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_87d38832f85e4965bfab1a077990114f [id=ad812862698d421184ebf501ce79b7be]). Expected version 2 got 0
I20251025 07:52:39.945240 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet f70091024c034c118fb86578db19a31c (table default.Table_332b308220f64c6dbb5f29711d8463b3 [id=e041946c093b4c789ce175c736cc230b]). Expected version 1 got 0
I20251025 07:52:39.945819 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5]). Expected version 1 got 0
I20251025 07:52:39.951175 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0ec5520674c457e94e51dd7481b4d5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:39.951951 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d829e6dd1224ad69e092476c281478c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:39.957067 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa9d9ab8073945d28d52fb02213f201a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:39.957276 773 raft_consensus.cc:2468] T aa9d9ab8073945d28d52fb02213f201a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:39.961061 2314 raft_consensus.cc:493] T e54fe7cbcffa40c4b04144b47bee2b48 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:39.961253 2314 raft_consensus.cc:515] T e54fe7cbcffa40c4b04144b47bee2b48 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.962357 2314 leader_election.cc:290] T e54fe7cbcffa40c4b04144b47bee2b48 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:39.962885 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e54fe7cbcffa40c4b04144b47bee2b48" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:39.963176 1049 raft_consensus.cc:2468] T e54fe7cbcffa40c4b04144b47bee2b48 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:39.963748 850 leader_election.cc:304] T e54fe7cbcffa40c4b04144b47bee2b48 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:39.964058 2314 raft_consensus.cc:2804] T e54fe7cbcffa40c4b04144b47bee2b48 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:39.964169 2314 raft_consensus.cc:493] T e54fe7cbcffa40c4b04144b47bee2b48 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:39.964269 2314 raft_consensus.cc:3060] T e54fe7cbcffa40c4b04144b47bee2b48 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:39.967713 903 raft_consensus.cc:3060] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:39.968398 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a58931e7cd7454fb583f1c70cd2ad15" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:39.968609 773 raft_consensus.cc:2393] T 6a58931e7cd7454fb583f1c70cd2ad15 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3aea2869ff814cc7b142012faa274382 in current term 1: Already voted for candidate db42be7ad3c741a0aa427f27e245993b in this term.
I20251025 07:52:39.974741 2182 consensus_queue.cc:1048] T fa998b9ca6724c4cb1eb68a3eecb0b11 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:39.977351 909 raft_consensus.cc:3060] T 5fca357d7e24494fb6af4bd73d98833c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:39.951179 2169 ts_tablet_manager.cc:1929] T d37758f606344127b3db74c87babe69c P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:39.983248 2169 log.cc:1199] T d37758f606344127b3db74c87babe69c P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/d37758f606344127b3db74c87babe69c
I20251025 07:52:39.983646 2314 raft_consensus.cc:515] T e54fe7cbcffa40c4b04144b47bee2b48 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:39.983832 2169 ts_tablet_manager.cc:1950] T d37758f606344127b3db74c87babe69c P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:39.985559 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e54fe7cbcffa40c4b04144b47bee2b48" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:39.986555 776 raft_consensus.cc:2468] T e54fe7cbcffa40c4b04144b47bee2b48 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:39.987584 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]). Expected version 3 got 0
I20251025 07:52:39.988085 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 755f0d6c87044e20a8020f8410940123 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 4 got 0
I20251025 07:52:39.988612 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 22f7db660616457b83b24e947a0b0bae (table default.Table_4a86191163f2408f9bcc795cd8d1bbf6 [id=bb316a7e000748ca902a9bfb494802d7]). Expected version 1 got 0
I20251025 07:52:39.989986 2117 raft_consensus.cc:493] T 0325d462441349ef81841f1b0cab2a20 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:39.990109 2117 raft_consensus.cc:515] T 0325d462441349ef81841f1b0cab2a20 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:39.990738 2117 leader_election.cc:290] T 0325d462441349ef81841f1b0cab2a20 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:39.991665 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 53) took 1331 ms. Trace:
I20251025 07:52:39.991812 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325d462441349ef81841f1b0cab2a20" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:39.991859 2169 rpcz_store.cc:276] 1025 07:52:38.659860 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.665644 (+ 5784us) service_pool.cc:225] Handling call
1025 07:52:39.639887 (+974243us) ts_tablet_manager.cc:1151] Deleting tablet d37758f606344127b3db74c87babe69c
1025 07:52:39.639894 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:39.691021 (+ 51127us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:39.951023 (+260002us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:39.991596 (+ 40573us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":965430}
W20251025 07:52:39.992411 712 leader_election.cc:343] T 0325d462441349ef81841f1b0cab2a20 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:39.992483 2169 tablet_replica.cc:333] T a6ddefa1738f4ff98586900c344359d2 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:39.992554 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet d37758f606344127b3db74c87babe69c (table default.Table_92cbcd8230f249a8b2821f69a81b41f1 [id=08e53121e86d405dbbd02a4605695fec]) successfully deleted
I20251025 07:52:39.992754 2169 raft_consensus.cc:2243] T a6ddefa1738f4ff98586900c344359d2 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:39.993180 2169 raft_consensus.cc:2272] T a6ddefa1738f4ff98586900c344359d2 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:39.993929 2169 ts_tablet_manager.cc:1916] T a6ddefa1738f4ff98586900c344359d2 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:40.006507 1671 raft_consensus.cc:493] T 22c97b3ee92140c68a7e1b93033cb299 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:40.007064 1671 raft_consensus.cc:515] T 22c97b3ee92140c68a7e1b93033cb299 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.008486 1671 leader_election.cc:290] T 22c97b3ee92140c68a7e1b93033cb299 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:40.011003 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22c97b3ee92140c68a7e1b93033cb299" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:40.011261 776 raft_consensus.cc:2468] T 22c97b3ee92140c68a7e1b93033cb299 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:40.011662 982 leader_election.cc:304] T 22c97b3ee92140c68a7e1b93033cb299 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:40.011922 1671 raft_consensus.cc:2804] T 22c97b3ee92140c68a7e1b93033cb299 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:40.012066 1671 raft_consensus.cc:493] T 22c97b3ee92140c68a7e1b93033cb299 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:40.012214 1671 raft_consensus.cc:3060] T 22c97b3ee92140c68a7e1b93033cb299 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:40.013697 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 0325d462441349ef81841f1b0cab2a20 (table default.Table_750ecb905839434fa9a1aa72f87394be [id=9ce3ba77c56a4ef09cb3b40ea0e488f6]). Expected version 1 got 0
I20251025 07:52:40.015434 1671 raft_consensus.cc:515] T 22c97b3ee92140c68a7e1b93033cb299 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.016471 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22c97b3ee92140c68a7e1b93033cb299" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:40.016690 776 raft_consensus.cc:3060] T 22c97b3ee92140c68a7e1b93033cb299 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:40.018975 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]). Expected version 1 got 0
I20251025 07:52:40.019681 776 raft_consensus.cc:2468] T 22c97b3ee92140c68a7e1b93033cb299 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:40.022971 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_e90d7f5ff13b40878e5b625c97792c7d [id=fc8180c09d7b4e74b2e161f7acb8fe0e]). Expected version 1 got 0
I20251025 07:52:39.985585 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "943ec703e65f4fd3b0c690e08c0bd226" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:40.023614 910 raft_consensus.cc:2468] T 943ec703e65f4fd3b0c690e08c0bd226 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:39.986073 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "943ec703e65f4fd3b0c690e08c0bd226" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:40.023854 902 raft_consensus.cc:3060] T 943ec703e65f4fd3b0c690e08c0bd226 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:40.024029 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 830fdbc6812e416690d0db42a73af503 (table default.Table_92824b3f2a774564b3aea04aaa25d204 [id=fb21a45b86ab446e8c3dae4512a9598f]). Expected version 1 got 0
I20251025 07:52:40.024547 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c]). Expected version 1 got 0
I20251025 07:52:40.024932 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236]). Expected version 1 got 0
I20251025 07:52:40.025831 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3]). Expected version 2 got 0
I20251025 07:52:39.986301 905 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "80fe3255ed614270bd8352a2853fbc43" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:40.026928 902 raft_consensus.cc:2468] T 943ec703e65f4fd3b0c690e08c0bd226 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
W20251025 07:52:40.031286 983 consensus_peers.cc:597] T f0bdaa8cfd764f1b83f47881353d26b6 P db42be7ad3c741a0aa427f27e245993b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f0bdaa8cfd764f1b83f47881353d26b6. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:39.989571 847 leader_election.cc:304] T aa9d9ab8073945d28d52fb02213f201a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:40.033726 2105 raft_consensus.cc:2804] T aa9d9ab8073945d28d52fb02213f201a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:40.033812 2105 raft_consensus.cc:493] T aa9d9ab8073945d28d52fb02213f201a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:40.033874 2105 raft_consensus.cc:3060] T aa9d9ab8073945d28d52fb02213f201a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:40.036576 2105 raft_consensus.cc:515] T aa9d9ab8073945d28d52fb02213f201a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.037695 2105 leader_election.cc:290] T aa9d9ab8073945d28d52fb02213f201a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:40.008843 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22c97b3ee92140c68a7e1b93033cb299" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:40.037806 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa9d9ab8073945d28d52fb02213f201a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:40.038305 850 leader_election.cc:343] T aa9d9ab8073945d28d52fb02213f201a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:40.039510 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 87268e27540c4ec58ce1e62df819f19d (table default.Table_350383f0e98947eba77186e74ad88781 [id=96b0e47c666c4d8f982741aea7b17ae7]). Expected version 2 got 0
I20251025 07:52:40.038020 903 raft_consensus.cc:2468] T 22c97b3ee92140c68a7e1b93033cb299 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
W20251025 07:52:40.040648 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:46231, user_credentials={real_user=slave}} blocked reactor thread for 51109.5us
I20251025 07:52:40.040772 2169 ts_tablet_manager.cc:1929] T a6ddefa1738f4ff98586900c344359d2 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:40.040910 847 leader_election.cc:304] T 6a58931e7cd7454fb583f1c70cd2ad15 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:40.041163 2105 raft_consensus.cc:2749] T 6a58931e7cd7454fb583f1c70cd2ad15 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:40.041582 847 consensus_peers.cc:597] T 50ae9bca6bed4f749d54191ac1543410 P 3aea2869ff814cc7b142012faa274382 -> Peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Couldn't send request to peer 23eeffd3de254ce18db5febfde971c6b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 50ae9bca6bed4f749d54191ac1543410. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:40.041776 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 8f2c8582fed34c848d9903b8ad8b4057 (table default.Table_438f67f3dc62465999a917986922b3ed [id=7c7fcd265e624d7fab74661db505d9c3]). Expected version 1 got 0
I20251025 07:52:40.043114 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 36262ab8c6964dd684c01b4db6219a4f (table default.Table_76158f8cf82e4cdb9a90580e46aaea48 [id=f249f30497bc450c887d659721469de8]). Expected version 1 got 0
I20251025 07:52:40.044519 2169 log.cc:1199] T a6ddefa1738f4ff98586900c344359d2 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/a6ddefa1738f4ff98586900c344359d2
I20251025 07:52:40.045058 2169 ts_tablet_manager.cc:1950] T a6ddefa1738f4ff98586900c344359d2 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:40.048170 2314 leader_election.cc:290] T e54fe7cbcffa40c4b04144b47bee2b48 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:40.048698 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e54fe7cbcffa40c4b04144b47bee2b48" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:40.048903 1049 raft_consensus.cc:3060] T e54fe7cbcffa40c4b04144b47bee2b48 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:40.048871 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 2c50ec95e48e49b59a76fac4fc862d3d (table default.Table_32b54e82cbc84d39a0020826f8737d62 [id=c28b0b6acd61490caecfdaab803a9d92]). Expected version 2 got 0
I20251025 07:52:40.049109 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3]). Expected version 1 got 0
I20251025 07:52:40.049436 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet b97e39aaac3048ed93dbfe873b757254 (table default.Table_99fd36369f9a4e0fb14f234c98635203 [id=c8059ab6cec146efa631d06799d969a9]). Expected version 1 got 0
I20251025 07:52:40.049784 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 8e6d76b1a97945cfb6c3055e1b622f8a (table default.Table_10e57b42f97544208675708b1e028846 [id=f5c1be203e2b4831a025871cf8a2ef17]). Expected version 2 got 0
I20251025 07:52:40.050171 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet b17d2a9ad6be49d29f2413666fba7e69 (table default.Table_fcddb31989ee4236a4981c12779b3f5d [id=f3794fa081364666b944c412415bdba0]). Expected version 3 got 0
I20251025 07:52:40.050348 451 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 292fe04055954afd8c08ac2636b68191 (table default.Table_9fa69b10b5e842639fc0b0a58cd9b6a5 [id=a253ce0253bd4da4814741c482193d8f]). Expected version 1 got 0
I20251025 07:52:40.051700 1049 raft_consensus.cc:2468] T e54fe7cbcffa40c4b04144b47bee2b48 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:40.052373 850 leader_election.cc:304] T e54fe7cbcffa40c4b04144b47bee2b48 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:40.054908 2105 raft_consensus.cc:2804] T e54fe7cbcffa40c4b04144b47bee2b48 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:40.055428 2105 raft_consensus.cc:697] T e54fe7cbcffa40c4b04144b47bee2b48 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:40.055693 2105 consensus_queue.cc:237] T e54fe7cbcffa40c4b04144b47bee2b48 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.057462 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 72) took 1381 ms. Trace:
I20251025 07:52:40.057641 2169 rpcz_store.cc:276] 1025 07:52:38.676520 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.676594 (+ 74us) service_pool.cc:225] Handling call
1025 07:52:39.992398 (+1315804us) ts_tablet_manager.cc:1151] Deleting tablet a6ddefa1738f4ff98586900c344359d2
1025 07:52:39.992412 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:40.014330 (+ 21918us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.040623 (+ 26293us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.057414 (+ 16791us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1315586}
I20251025 07:52:40.058326 2169 tablet_replica.cc:333] T fa77085e81c7440d9e255c44d1e80c26 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:40.058935 2169 raft_consensus.cc:2243] T fa77085e81c7440d9e255c44d1e80c26 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:40.058948 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet a6ddefa1738f4ff98586900c344359d2 (table default.Table_c902adc24467448f9b5d4f585cf614f5 [id=56e169f79a37465d82dd1d1c3c285740]) successfully deleted
I20251025 07:52:40.059083 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325d462441349ef81841f1b0cab2a20" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:40.059194 2169 raft_consensus.cc:2272] T fa77085e81c7440d9e255c44d1e80c26 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:40.059933 2169 ts_tablet_manager.cc:1916] T fa77085e81c7440d9e255c44d1e80c26 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:40.066797 1166 tablet_bootstrap.cc:492] T ebf3a2a926754305b5bd95fcebb25d29 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:40.066984 1166 ts_tablet_manager.cc:1403] T ebf3a2a926754305b5bd95fcebb25d29 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.132s user 0.000s sys 0.006s
I20251025 07:52:40.067802 1166 raft_consensus.cc:359] T ebf3a2a926754305b5bd95fcebb25d29 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.068087 1166 raft_consensus.cc:385] T ebf3a2a926754305b5bd95fcebb25d29 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:40.068234 1166 raft_consensus.cc:740] T ebf3a2a926754305b5bd95fcebb25d29 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:40.068614 1166 consensus_queue.cc:260] T ebf3a2a926754305b5bd95fcebb25d29 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.069458 1166 ts_tablet_manager.cc:1434] T ebf3a2a926754305b5bd95fcebb25d29 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:40.070048 1166 tablet_bootstrap.cc:492] T ef9a2c9f2ccf432ab835396020872208 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:40.073547 1169 tablet_bootstrap.cc:492] T f4578ce69a0d4322944e5ce7c8a59831 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:40.073706 1169 ts_tablet_manager.cc:1403] T f4578ce69a0d4322944e5ce7c8a59831 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.212s user 0.049s sys 0.116s
I20251025 07:52:40.074510 1169 raft_consensus.cc:359] T f4578ce69a0d4322944e5ce7c8a59831 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.074692 1169 raft_consensus.cc:385] T f4578ce69a0d4322944e5ce7c8a59831 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:40.074774 1169 raft_consensus.cc:740] T f4578ce69a0d4322944e5ce7c8a59831 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:40.074997 1169 consensus_queue.cc:260] T f4578ce69a0d4322944e5ce7c8a59831 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.075505 1169 ts_tablet_manager.cc:1434] T f4578ce69a0d4322944e5ce7c8a59831 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:40.075807 1169 tablet_bootstrap.cc:492] T aa9d9ab8073945d28d52fb02213f201a P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:40.078112 1169 tablet_bootstrap.cc:654] T aa9d9ab8073945d28d52fb02213f201a P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:40.087877 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_a6c869949b744bc98efb735b0e73e7aa [id=bf77b5ff6f1048dd996d948437f9ff5e]). Expected version 2 got 0
I20251025 07:52:40.091928 1166 tablet_bootstrap.cc:654] T ef9a2c9f2ccf432ab835396020872208 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:40.094291 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3]). Expected version 1 got 0
I20251025 07:52:40.023191 2146 ts_tablet_manager.cc:1929] T b772025852794a49b072f37cccbd3291 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:40.094859 2146 log.cc:1199] T b772025852794a49b072f37cccbd3291 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/b772025852794a49b072f37cccbd3291
I20251025 07:52:40.094949 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5]). Expected version 1 got 0
I20251025 07:52:40.095296 2146 ts_tablet_manager.cc:1950] T b772025852794a49b072f37cccbd3291 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:40.095650 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 63) took 1670 ms. Trace:
I20251025 07:52:40.095746 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f]). Expected version 1 got 0
I20251025 07:52:40.095752 2146 rpcz_store.cc:276] 1025 07:52:38.424872 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.433356 (+ 8484us) service_pool.cc:225] Handling call
1025 07:52:39.930498 (+1497142us) ts_tablet_manager.cc:1151] Deleting tablet b772025852794a49b072f37cccbd3291
1025 07:52:39.930580 (+ 82us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:39.962510 (+ 31930us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.023071 (+ 60561us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.095633 (+ 72562us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1496977}
I20251025 07:52:40.096251 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet b772025852794a49b072f37cccbd3291 (table default.Table_e50bf47b78d94bbc811e6f3fb3551edb [id=0827603890c8466082bc6c306c9c66af]) successfully deleted
I20251025 07:52:40.096464 2146 tablet_replica.cc:333] T 8c76d1b509f04ab1a10b83cff675c11e P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:40.096632 2146 raft_consensus.cc:2243] T 8c76d1b509f04ab1a10b83cff675c11e P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:40.096769 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7]). Expected version 1 got 0
I20251025 07:52:40.096818 2146 raft_consensus.cc:2272] T 8c76d1b509f04ab1a10b83cff675c11e P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:40.097388 2146 ts_tablet_manager.cc:1916] T 8c76d1b509f04ab1a10b83cff675c11e P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:40.097380 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 872cb7512d104827b1f1f186d33a4a44 (table default.Table_d5c65dce8b1b4214ba519d964012bfc0 [id=d4be274b36a845a99a4b96b8846248fd]). Expected version 1 got 0
I20251025 07:52:40.099810 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 04d27d54fdb7483b869408af65ff85c9 (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]). Expected version 2 got 0
I20251025 07:52:40.100450 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 4c155aa7fa5b4d68bbab1918b8432372 (table default.Table_6c5a24747ba14352b0285a41ec2985a4 [id=28bbaad474164a8c8c97a76bb8081791]). Expected version 2 got 0
W20251025 07:52:40.043567 710 leader_election.cc:343] T 80fe3255ed614270bd8352a2853fbc43 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:40.044342 1167 tablet_bootstrap.cc:492] T 66f0d398e59a4175aa9fe90715124d1a P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:40.100869 710 leader_election.cc:304] T 80fe3255ed614270bd8352a2853fbc43 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:40.100947 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet cb829649f73d4d3f8862d268c953bd86 (table default.Table_ba4b8cc7b930482690b7e192e0edff12 [id=7944225d3ce7439b881bfb9289d9f040]). Expected version 1 got 0
I20251025 07:52:40.100960 1167 ts_tablet_manager.cc:1403] T 66f0d398e59a4175aa9fe90715124d1a P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.172s user 0.004s sys 0.012s
W20251025 07:52:40.101234 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:43563, user_credentials={real_user=slave}} blocked reactor thread for 57658.4us
I20251025 07:52:40.054595 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa9d9ab8073945d28d52fb02213f201a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:40.101656 773 raft_consensus.cc:3060] T aa9d9ab8073945d28d52fb02213f201a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:40.101933 710 leader_election.cc:343] T 0325d462441349ef81841f1b0cab2a20 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:40.102068 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 292fe04055954afd8c08ac2636b68191 (table default.Table_9fa69b10b5e842639fc0b0a58cd9b6a5 [id=a253ce0253bd4da4814741c482193d8f]). Expected version 1 got 0
I20251025 07:52:40.102067 710 leader_election.cc:304] T 0325d462441349ef81841f1b0cab2a20 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:40.102310 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet aa9d9ab8073945d28d52fb02213f201a (table default.Table_4791672658f043f6b97b9320a531ebb5 [id=dfd20cf9c3e541c4abe23d59e0aa7f2a]). Expected version 1 got 0
I20251025 07:52:40.102514 2117 raft_consensus.cc:2749] T 80fe3255ed614270bd8352a2853fbc43 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:40.102700 2117 raft_consensus.cc:493] T 4b28215f61574cf48c447dd01955ac77 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:40.102645 448 catalog_manager.cc:5649] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 2, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 2 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } }
I20251025 07:52:40.102840 2117 raft_consensus.cc:515] T 4b28215f61574cf48c447dd01955ac77 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.103052 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet b17d2a9ad6be49d29f2413666fba7e69 (table default.Table_fcddb31989ee4236a4981c12779b3f5d [id=f3794fa081364666b944c412415bdba0]). Expected version 3 got 0
I20251025 07:52:40.103382 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]). Expected version 1 got 0
I20251025 07:52:40.103523 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet ef9a2c9f2ccf432ab835396020872208 (table default.Table_2b9b772d4c234ac286bef3658482e490 [id=409fab60f9d744798e6f92b868f0f7c2]). Expected version 1 got 0
I20251025 07:52:40.103667 2117 leader_election.cc:290] T 4b28215f61574cf48c447dd01955ac77 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:40.103684 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]). Expected version 1 got 0
I20251025 07:52:40.103837 2117 raft_consensus.cc:2749] T 0325d462441349ef81841f1b0cab2a20 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:40.104068 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84]). Expected version 1 got 0
I20251025 07:52:40.104260 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet ef1854d35a3047e3a9fbe0166ab1295a (table default.Table_088aad2a46b041bfa29ed8e6929760b1 [id=834669c54dae449790818684fd8659bc]). Expected version 2 got 0
I20251025 07:52:40.104389 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet cd23b8094d7e472eba6bd170248858a9 (table default.Table_ca479cb5c1184c5391b7e1c9d8a08937 [id=d82b3e1317434cf08a0d51e8a9427224]). Expected version 3 got 2
I20251025 07:52:40.059864 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e54fe7cbcffa40c4b04144b47bee2b48" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:40.104606 774 raft_consensus.cc:3060] T e54fe7cbcffa40c4b04144b47bee2b48 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:40.104661 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 5dab439bc50d4e869548154e1d222ad6 (table default.Table_bd82d871f84943f4baa967f5c0ec74a3 [id=45325c188eac4d34bb016d01de091f87]). Expected version 2 got 0
I20251025 07:52:40.072218 2166 ts_tablet_manager.cc:1929] T d66ad049d5484228a268b0facb00ecf1 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:40.104909 2166 log.cc:1199] T d66ad049d5484228a268b0facb00ecf1 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/d66ad049d5484228a268b0facb00ecf1
I20251025 07:52:40.104899 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_87d38832f85e4965bfab1a077990114f [id=ad812862698d421184ebf501ce79b7be]). Expected version 2 got 0
I20251025 07:52:40.105556 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet f70091024c034c118fb86578db19a31c (table default.Table_332b308220f64c6dbb5f29711d8463b3 [id=e041946c093b4c789ce175c736cc230b]). Expected version 1 got 0
I20251025 07:52:40.105927 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b28215f61574cf48c447dd01955ac77" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:40.106586 712 leader_election.cc:343] T 4b28215f61574cf48c447dd01955ac77 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:40.107100 1167 raft_consensus.cc:359] T 66f0d398e59a4175aa9fe90715124d1a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:40.107261 1167 raft_consensus.cc:385] T 66f0d398e59a4175aa9fe90715124d1a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:40.107340 1167 raft_consensus.cc:740] T 66f0d398e59a4175aa9fe90715124d1a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:40.107542 1167 consensus_queue.cc:260] T 66f0d398e59a4175aa9fe90715124d1a P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:40.107977 1167 ts_tablet_manager.cc:1434] T 66f0d398e59a4175aa9fe90715124d1a P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251025 07:52:40.108217 1167 tablet_bootstrap.cc:492] T a4e4b99be0434baaa75d988281797846 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:40.109795 2166 ts_tablet_manager.cc:1950] T d66ad049d5484228a268b0facb00ecf1 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:40.110888 1167 tablet_bootstrap.cc:654] T a4e4b99be0434baaa75d988281797846 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:40.111681 773 raft_consensus.cc:2468] T aa9d9ab8073945d28d52fb02213f201a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:40.113468 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22c97b3ee92140c68a7e1b93033cb299" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:40.113626 903 raft_consensus.cc:3060] T 22c97b3ee92140c68a7e1b93033cb299 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:40.114176 1671 leader_election.cc:290] T 22c97b3ee92140c68a7e1b93033cb299 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:40.114859 774 raft_consensus.cc:2468] T e54fe7cbcffa40c4b04144b47bee2b48 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:40.115482 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]). Expected version 3 got 0
I20251025 07:52:40.116292 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 22f7db660616457b83b24e947a0b0bae (table default.Table_4a86191163f2408f9bcc795cd8d1bbf6 [id=bb316a7e000748ca902a9bfb494802d7]). Expected version 1 got 0
I20251025 07:52:40.117528 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 0325d462441349ef81841f1b0cab2a20 (table default.Table_750ecb905839434fa9a1aa72f87394be [id=9ce3ba77c56a4ef09cb3b40ea0e488f6]). Expected version 1 got 0
I20251025 07:52:40.118635 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 830fdbc6812e416690d0db42a73af503 (table default.Table_92824b3f2a774564b3aea04aaa25d204 [id=fb21a45b86ab446e8c3dae4512a9598f]). Expected version 1 got 0
I20251025 07:52:40.119357 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 755f0d6c87044e20a8020f8410940123 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 4 got 0
I20251025 07:52:40.119686 982 leader_election.cc:304] T 22c97b3ee92140c68a7e1b93033cb299 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:40.119957 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]). Expected version 1 got 0
I20251025 07:52:40.120875 1167 tablet_bootstrap.cc:492] T a4e4b99be0434baaa75d988281797846 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:40.121038 1167 ts_tablet_manager.cc:1403] T a4e4b99be0434baaa75d988281797846 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.001s
I20251025 07:52:40.121040 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_e90d7f5ff13b40878e5b625c97792c7d [id=fc8180c09d7b4e74b2e161f7acb8fe0e]). Expected version 1 got 0
I20251025 07:52:40.121340 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236]). Expected version 1 got 0
I20251025 07:52:40.121783 1167 raft_consensus.cc:359] T a4e4b99be0434baaa75d988281797846 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.121978 1167 raft_consensus.cc:385] T a4e4b99be0434baaa75d988281797846 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:40.122071 1167 raft_consensus.cc:740] T a4e4b99be0434baaa75d988281797846 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:40.122360 1167 consensus_queue.cc:260] T a4e4b99be0434baaa75d988281797846 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.122677 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c]). Expected version 1 got 0
I20251025 07:52:40.122970 1167 ts_tablet_manager.cc:1434] T a4e4b99be0434baaa75d988281797846 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:40.123317 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3]). Expected version 2 got 0
I20251025 07:52:40.124349 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 87268e27540c4ec58ce1e62df819f19d (table default.Table_350383f0e98947eba77186e74ad88781 [id=96b0e47c666c4d8f982741aea7b17ae7]). Expected version 2 got 0
I20251025 07:52:40.125407 448 catalog_manager.cc:5649] T 8f2c8582fed34c848d9903b8ad8b4057 P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } }
I20251025 07:52:40.126246 1671 raft_consensus.cc:2804] T 22c97b3ee92140c68a7e1b93033cb299 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:40.126451 1671 raft_consensus.cc:697] T 22c97b3ee92140c68a7e1b93033cb299 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:40.126291 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 8f2c8582fed34c848d9903b8ad8b4057 (table default.Table_438f67f3dc62465999a917986922b3ed [id=7c7fcd265e624d7fab74661db505d9c3]). Expected version 1 got 0
I20251025 07:52:40.126788 1671 consensus_queue.cc:237] T 22c97b3ee92140c68a7e1b93033cb299 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.126876 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525]). Expected version 1 got 0
I20251025 07:52:40.127108 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 36262ab8c6964dd684c01b4db6219a4f (table default.Table_76158f8cf82e4cdb9a90580e46aaea48 [id=f249f30497bc450c887d659721469de8]). Expected version 1 got 0
I20251025 07:52:40.127703 448 catalog_manager.cc:5649] T 2c50ec95e48e49b59a76fac4fc862d3d P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } }
I20251025 07:52:40.127908 1167 tablet_bootstrap.cc:492] T 04449cc97f404a19b6a1b20dea074f95 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:40.128073 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 2c50ec95e48e49b59a76fac4fc862d3d (table default.Table_32b54e82cbc84d39a0020826f8737d62 [id=c28b0b6acd61490caecfdaab803a9d92]). Expected version 2 got 0
I20251025 07:52:40.128301 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 63) took 1611 ms. Trace:
I20251025 07:52:40.128381 2166 rpcz_store.cc:276] 1025 07:52:38.516704 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.516744 (+ 40us) service_pool.cc:225] Handling call
1025 07:52:39.846550 (+1329806us) ts_tablet_manager.cc:1151] Deleting tablet d66ad049d5484228a268b0facb00ecf1
1025 07:52:39.846559 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:40.032845 (+186286us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.072054 (+ 39209us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.128277 (+ 56223us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1329667}
I20251025 07:52:40.128644 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet b97e39aaac3048ed93dbfe873b757254 (table default.Table_99fd36369f9a4e0fb14f234c98635203 [id=c8059ab6cec146efa631d06799d969a9]). Expected version 1 got 0
I20251025 07:52:40.128712 2166 tablet_replica.cc:333] T 4b28215f61574cf48c447dd01955ac77 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:40.128842 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 8e6d76b1a97945cfb6c3055e1b622f8a (table default.Table_10e57b42f97544208675708b1e028846 [id=f5c1be203e2b4831a025871cf8a2ef17]). Expected version 2 got 0
I20251025 07:52:40.128891 2166 raft_consensus.cc:2243] T 4b28215f61574cf48c447dd01955ac77 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:40.129040 2166 raft_consensus.cc:2272] T 4b28215f61574cf48c447dd01955ac77 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:40.129400 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet d66ad049d5484228a268b0facb00ecf1 (table default.Table_9f52d94037424befbdea5c711068396a [id=55440328803042a28df1b6869d279f59]) successfully deleted
I20251025 07:52:40.129447 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 509c94f2a422493ea4390ecee576db72 (table default.Table_3f081b7bb4e1420c8121ea4115439f3d [id=696a1285fbfe4038a8b1b38708c8ea49]). Expected version 1 got 0
I20251025 07:52:40.129658 847 leader_election.cc:304] T aa9d9ab8073945d28d52fb02213f201a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:40.129684 1659 raft_consensus.cc:493] T 724d0b7e1b65445a9a1f056852983bdb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:40.129914 2314 raft_consensus.cc:2804] T aa9d9ab8073945d28d52fb02213f201a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:40.129880 1659 raft_consensus.cc:515] T 724d0b7e1b65445a9a1f056852983bdb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:40.130054 2314 raft_consensus.cc:697] T aa9d9ab8073945d28d52fb02213f201a P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:40.130268 2166 ts_tablet_manager.cc:1916] T 4b28215f61574cf48c447dd01955ac77 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:40.130897 1659 leader_election.cc:290] T 724d0b7e1b65445a9a1f056852983bdb P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:40.131423 905 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b28215f61574cf48c447dd01955ac77" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:40.132088 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724d0b7e1b65445a9a1f056852983bdb" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:40.132810 712 leader_election.cc:343] T 724d0b7e1b65445a9a1f056852983bdb P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:40.134361 710 leader_election.cc:343] T 4b28215f61574cf48c447dd01955ac77 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:40.134469 710 leader_election.cc:304] T 4b28215f61574cf48c447dd01955ac77 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:40.134639 710 raft_consensus.cc:2686] T 4b28215f61574cf48c447dd01955ac77 P 23eeffd3de254ce18db5febfde971c6b: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251025 07:52:40.135495 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet ab42909afc524f2fa3c1041495d13a59 (table default.Table_7eb8424d5192485da1a2d67bfdb2bbbf [id=40c29df99590408cbd5c583d4fc741eb]). Expected version 2 got 0
I20251025 07:52:40.135834 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e]). Expected version 1 got 0
I20251025 07:52:40.136085 448 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet bc2d16b33ca8426abbd7bd38339c9f1d (table default.Table_4ea985df4c5b415c9570c64506a98cc4 [id=98d2b7797e33485e8872054aa2eb8ec9]). Expected version 1 got 0
I20251025 07:52:40.137080 1167 tablet_bootstrap.cc:654] T 04449cc97f404a19b6a1b20dea074f95 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:40.130316 2314 consensus_queue.cc:237] T aa9d9ab8073945d28d52fb02213f201a P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.138176 1659 raft_consensus.cc:493] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:40.138316 1659 raft_consensus.cc:515] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:40.138653 453 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet ff3daed6c9af47e3976b82b4b9a41086
I20251025 07:52:40.138875 1659 leader_election.cc:290] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:40.138939 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724d0b7e1b65445a9a1f056852983bdb" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:40.139348 453 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:47610 (request call id 54) took 1911 ms. Trace:
I20251025 07:52:40.139425 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8eaf6c97ab9b41e9ae746a9c5abe1735" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:40.139878 712 leader_election.cc:343] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:40.139452 453 rpcz_store.cc:276] 1025 07:52:38.228061 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:39.172435 (+944374us) service_pool.cc:225] Handling call
1025 07:52:39.238516 (+ 66081us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:39.238688 (+ 172us) catalog_manager.cc:2815] Modifying in-memory table state
1025 07:52:39.238759 (+ 71us) catalog_manager.cc:2824] Locking tablets
1025 07:52:39.239035 (+ 276us) catalog_manager.cc:2838] Removing table and tablets from system table
1025 07:52:39.253814 (+ 14779us) catalog_manager.cc:2856] Removing table from by-name map
1025 07:52:39.253925 (+ 111us) catalog_manager.cc:2872] Committing in-memory state
1025 07:52:40.138604 (+884679us) catalog_manager.cc:2876] Aborting table tasks
1025 07:52:40.139285 (+ 681us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:39.240241 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:39.240367 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:39.240373 (+ 6us) write_op.cc:435] Acquired schema lock
1025 07:52:39.240375 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:39.240492 (+ 117us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:39.240506 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:39.240512 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:39.240610 (+ 98us) tablet.cc:699] Row locks acquired
1025 07:52:39.240612 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:39.240648 (+ 36us) write_op.cc:270] Start()
1025 07:52:39.240682 (+ 34us) write_op.cc:276] Timestamp: P: 1761378759240630 usec, L: 0
1025 07:52:39.240686 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:39.240858 (+ 172us) log.cc:844] Serialized 725 byte log entry
1025 07:52:39.244509 (+ 3651us) op_driver.cc:464] REPLICATION: finished
1025 07:52:39.244610 (+ 101us) write_op.cc:301] APPLY: starting
1025 07:52:39.244629 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:39.244678 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:39.244712 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:39.244784 (+ 72us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:39.244820 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1025 07:52:39.244828 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:39.244905 (+ 77us) log.cc:844] Serialized 29 byte log entry
1025 07:52:39.244968 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:39.244990 (+ 22us) write_op.cc:454] Released schema lock
1025 07:52:39.244997 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:39.245081 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:39.245085 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":38,"apply.run_cpu_time_us":470,"apply.run_wall_time_us":470,"num_ops":2,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":718,"prepare.run_wall_time_us":719,"raft.queue_time_us":416,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":274,"replication_time_us":3795}]]}
I20251025 07:52:40.140514 2169 ts_tablet_manager.cc:1929] T fa77085e81c7440d9e255c44d1e80c26 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:40.140676 2169 log.cc:1199] T fa77085e81c7440d9e255c44d1e80c26 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/fa77085e81c7440d9e255c44d1e80c26
W20251025 07:52:40.140858 710 leader_election.cc:343] T 724d0b7e1b65445a9a1f056852983bdb P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:40.140947 710 leader_election.cc:304] T 724d0b7e1b65445a9a1f056852983bdb P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:40.141327 1659 raft_consensus.cc:2749] T 724d0b7e1b65445a9a1f056852983bdb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:40.141306 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8eaf6c97ab9b41e9ae746a9c5abe1735" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:40.142076 2169 ts_tablet_manager.cc:1950] T fa77085e81c7440d9e255c44d1e80c26 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:40.142534 894 tablet_service.cc:1552] Processing DeleteTablet for tablet ff3daed6c9af47e3976b82b4b9a41086 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:40.142794 903 raft_consensus.cc:2468] T 22c97b3ee92140c68a7e1b93033cb299 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:40.142966 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet ff3daed6c9af47e3976b82b4b9a41086 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:40.143369 754 tablet_service.cc:1552] Processing DeleteTablet for tablet ff3daed6c9af47e3976b82b4b9a41086 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:40.144667 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 73) took 1453 ms. Trace:
I20251025 07:52:40.144805 2169 rpcz_store.cc:276] 1025 07:52:38.691402 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.691647 (+ 245us) service_pool.cc:225] Handling call
1025 07:52:40.058242 (+1366595us) ts_tablet_manager.cc:1151] Deleting tablet fa77085e81c7440d9e255c44d1e80c26
1025 07:52:40.058256 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:40.120144 (+ 61888us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.140388 (+ 20244us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.144633 (+ 4245us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1366331}
I20251025 07:52:40.145246 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet fa77085e81c7440d9e255c44d1e80c26 (table default.Table_936e8f1bd9344704940225361305cf5b [id=4998c2abb8fb48db83483bc0aa0170eb]) successfully deleted
I20251025 07:52:40.145323 2169 tablet_replica.cc:333] T 0b21b7da1e2f47829c3f6cdaee689e8a P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
W20251025 07:52:40.145390 710 leader_election.cc:343] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:40.145494 710 leader_election.cc:304] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:40.145573 2169 raft_consensus.cc:2243] T 0b21b7da1e2f47829c3f6cdaee689e8a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:40.145696 1659 raft_consensus.cc:2749] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:40.146077 2169 raft_consensus.cc:2272] T 0b21b7da1e2f47829c3f6cdaee689e8a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:40.148617 982 consensus_peers.cc:597] T 08da9d2687704725aacd11cd92c6a9f9 P db42be7ad3c741a0aa427f27e245993b -> Peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Couldn't send request to peer 23eeffd3de254ce18db5febfde971c6b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08da9d2687704725aacd11cd92c6a9f9. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:40.149559 2105 consensus_queue.cc:1048] T 50ae9bca6bed4f749d54191ac1543410 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:40.149936 2169 ts_tablet_manager.cc:1916] T 0b21b7da1e2f47829c3f6cdaee689e8a P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:40.157660 2166 ts_tablet_manager.cc:1929] T 4b28215f61574cf48c447dd01955ac77 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:40.158223 2166 log.cc:1199] T 4b28215f61574cf48c447dd01955ac77 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/4b28215f61574cf48c447dd01955ac77
I20251025 07:52:40.162617 2166 ts_tablet_manager.cc:1950] T 4b28215f61574cf48c447dd01955ac77 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:40.165162 1169 tablet_bootstrap.cc:492] T aa9d9ab8073945d28d52fb02213f201a P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:40.165380 1169 ts_tablet_manager.cc:1403] T aa9d9ab8073945d28d52fb02213f201a P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.090s user 0.007s sys 0.022s
I20251025 07:52:40.166460 1169 raft_consensus.cc:359] T aa9d9ab8073945d28d52fb02213f201a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.166694 1169 raft_consensus.cc:385] T aa9d9ab8073945d28d52fb02213f201a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:40.166785 1169 raft_consensus.cc:740] T aa9d9ab8073945d28d52fb02213f201a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:40.167032 1169 consensus_queue.cc:260] T aa9d9ab8073945d28d52fb02213f201a P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.167344 2300 raft_consensus.cc:493] T 88bfc23acd274041a9457079c1709d82 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:40.167485 2300 raft_consensus.cc:515] T 88bfc23acd274041a9457079c1709d82 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.167640 1169 ts_tablet_manager.cc:1434] T aa9d9ab8073945d28d52fb02213f201a P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:40.167924 1169 tablet_bootstrap.cc:492] T ebf3a2a926754305b5bd95fcebb25d29 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:40.168157 2300 leader_election.cc:290] T 88bfc23acd274041a9457079c1709d82 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:40.168871 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88bfc23acd274041a9457079c1709d82" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:40.169129 774 raft_consensus.cc:2468] T 88bfc23acd274041a9457079c1709d82 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:40.169667 982 leader_election.cc:304] T 88bfc23acd274041a9457079c1709d82 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:40.170058 2300 raft_consensus.cc:2804] T 88bfc23acd274041a9457079c1709d82 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:40.170224 2300 raft_consensus.cc:493] T 88bfc23acd274041a9457079c1709d82 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:40.170334 2300 raft_consensus.cc:3060] T 88bfc23acd274041a9457079c1709d82 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:40.171703 1166 tablet_bootstrap.cc:492] T ef9a2c9f2ccf432ab835396020872208 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:40.171837 1166 ts_tablet_manager.cc:1403] T ef9a2c9f2ccf432ab835396020872208 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.102s user 0.002s sys 0.009s
I20251025 07:52:40.173934 2300 raft_consensus.cc:515] T 88bfc23acd274041a9457079c1709d82 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.174960 2146 ts_tablet_manager.cc:1929] T 8c76d1b509f04ab1a10b83cff675c11e P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:40.175271 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88bfc23acd274041a9457079c1709d82" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:40.175485 774 raft_consensus.cc:3060] T 88bfc23acd274041a9457079c1709d82 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:40.178280 1659 raft_consensus.cc:493] T f43ef51ee45d47a99ec732b49d50d3b4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:40.178416 1659 raft_consensus.cc:515] T f43ef51ee45d47a99ec732b49d50d3b4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.179114 1659 leader_election.cc:290] T f43ef51ee45d47a99ec732b49d50d3b4 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:40.180114 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f43ef51ee45d47a99ec732b49d50d3b4" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:40.181450 712 leader_election.cc:343] T f43ef51ee45d47a99ec732b49d50d3b4 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:40.182644 2300 leader_election.cc:290] T 88bfc23acd274041a9457079c1709d82 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:40.185220 774 raft_consensus.cc:2468] T 88bfc23acd274041a9457079c1709d82 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:40.185799 982 leader_election.cc:304] T 88bfc23acd274041a9457079c1709d82 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:40.186303 2316 raft_consensus.cc:2804] T 88bfc23acd274041a9457079c1709d82 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:40.186479 2316 raft_consensus.cc:697] T 88bfc23acd274041a9457079c1709d82 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:40.186790 2316 consensus_queue.cc:237] T 88bfc23acd274041a9457079c1709d82 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.202176 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5]). Expected version 1 got 0
I20251025 07:52:40.202658 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3]). Expected version 1 got 0
I20251025 07:52:40.203088 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_a6c869949b744bc98efb735b0e73e7aa [id=bf77b5ff6f1048dd996d948437f9ff5e]). Expected version 2 got 0
I20251025 07:52:40.203881 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f]). Expected version 1 got 0
I20251025 07:52:40.204911 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7]). Expected version 1 got 0
I20251025 07:52:40.205211 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 872cb7512d104827b1f1f186d33a4a44 (table default.Table_d5c65dce8b1b4214ba519d964012bfc0 [id=d4be274b36a845a99a4b96b8846248fd]). Expected version 1 got 0
I20251025 07:52:40.205447 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet cb829649f73d4d3f8862d268c953bd86 (table default.Table_ba4b8cc7b930482690b7e192e0edff12 [id=7944225d3ce7439b881bfb9289d9f040]). Expected version 1 got 0
I20251025 07:52:40.205715 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 04d27d54fdb7483b869408af65ff85c9 (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]). Expected version 2 got 0
I20251025 07:52:40.206020 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 4c155aa7fa5b4d68bbab1918b8432372 (table default.Table_6c5a24747ba14352b0285a41ec2985a4 [id=28bbaad474164a8c8c97a76bb8081791]). Expected version 2 got 0
I20251025 07:52:40.207413 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet aa9d9ab8073945d28d52fb02213f201a (table default.Table_4791672658f043f6b97b9320a531ebb5 [id=dfd20cf9c3e541c4abe23d59e0aa7f2a]). Expected version 1 got 0
I20251025 07:52:40.207844 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 292fe04055954afd8c08ac2636b68191 (table default.Table_9fa69b10b5e842639fc0b0a58cd9b6a5 [id=a253ce0253bd4da4814741c482193d8f]). Expected version 1 got 0
I20251025 07:52:40.208189 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]). Expected version 1 got 0
I20251025 07:52:40.208412 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet ef9a2c9f2ccf432ab835396020872208 (table default.Table_2b9b772d4c234ac286bef3658482e490 [id=409fab60f9d744798e6f92b868f0f7c2]). Expected version 1 got 0
I20251025 07:52:40.211580 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]). Expected version 1 got 0
I20251025 07:52:40.212110 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 22f7db660616457b83b24e947a0b0bae (table default.Table_4a86191163f2408f9bcc795cd8d1bbf6 [id=bb316a7e000748ca902a9bfb494802d7]). Expected version 1 got 0
I20251025 07:52:40.212347 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84]). Expected version 1 got 0
I20251025 07:52:40.212682 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 5dab439bc50d4e869548154e1d222ad6 (table default.Table_bd82d871f84943f4baa967f5c0ec74a3 [id=45325c188eac4d34bb016d01de091f87]). Expected version 2 got 0
I20251025 07:52:40.213138 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_87d38832f85e4965bfab1a077990114f [id=ad812862698d421184ebf501ce79b7be]). Expected version 2 got 0
I20251025 07:52:40.213440 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet f70091024c034c118fb86578db19a31c (table default.Table_332b308220f64c6dbb5f29711d8463b3 [id=e041946c093b4c789ce175c736cc230b]). Expected version 1 got 0
I20251025 07:52:40.214452 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_e90d7f5ff13b40878e5b625c97792c7d [id=fc8180c09d7b4e74b2e161f7acb8fe0e]). Expected version 1 got 0
I20251025 07:52:40.214921 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 0325d462441349ef81841f1b0cab2a20 (table default.Table_750ecb905839434fa9a1aa72f87394be [id=9ce3ba77c56a4ef09cb3b40ea0e488f6]). Expected version 1 got 0
I20251025 07:52:40.215612 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]). Expected version 1 got 0
I20251025 07:52:40.216068 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 830fdbc6812e416690d0db42a73af503 (table default.Table_92824b3f2a774564b3aea04aaa25d204 [id=fb21a45b86ab446e8c3dae4512a9598f]). Expected version 1 got 0
I20251025 07:52:40.216308 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 755f0d6c87044e20a8020f8410940123 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 4 got 0
I20251025 07:52:40.216934 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c]). Expected version 1 got 0
I20251025 07:52:40.217449 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]). Expected version 3 got 0
I20251025 07:52:40.217763 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236]). Expected version 1 got 0
I20251025 07:52:40.218209 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 87268e27540c4ec58ce1e62df819f19d (table default.Table_350383f0e98947eba77186e74ad88781 [id=96b0e47c666c4d8f982741aea7b17ae7]). Expected version 2 got 0
I20251025 07:52:40.218474 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 8f2c8582fed34c848d9903b8ad8b4057 (table default.Table_438f67f3dc62465999a917986922b3ed [id=7c7fcd265e624d7fab74661db505d9c3]). Expected version 1 got 0
I20251025 07:52:40.218755 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525]). Expected version 1 got 0
I20251025 07:52:40.219310 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 36262ab8c6964dd684c01b4db6219a4f (table default.Table_76158f8cf82e4cdb9a90580e46aaea48 [id=f249f30497bc450c887d659721469de8]). Expected version 1 got 0
I20251025 07:52:40.219782 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 2c50ec95e48e49b59a76fac4fc862d3d (table default.Table_32b54e82cbc84d39a0020826f8737d62 [id=c28b0b6acd61490caecfdaab803a9d92]). Expected version 2 got 0
I20251025 07:52:40.220110 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet b97e39aaac3048ed93dbfe873b757254 (table default.Table_99fd36369f9a4e0fb14f234c98635203 [id=c8059ab6cec146efa631d06799d969a9]). Expected version 1 got 0
I20251025 07:52:40.220544 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 8e6d76b1a97945cfb6c3055e1b622f8a (table default.Table_10e57b42f97544208675708b1e028846 [id=f5c1be203e2b4831a025871cf8a2ef17]). Expected version 2 got 0
I20251025 07:52:40.220904 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet b17d2a9ad6be49d29f2413666fba7e69 (table default.Table_fcddb31989ee4236a4981c12779b3f5d [id=f3794fa081364666b944c412415bdba0]). Expected version 3 got 0
I20251025 07:52:40.221194 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3]). Expected version 2 got 0
I20251025 07:52:40.221824 446 catalog_manager.cc:5649] T 509c94f2a422493ea4390ecee576db72 P db42be7ad3c741a0aa427f27e245993b reported cstate change: term changed from 0 to 1, leader changed from <none> to db42be7ad3c741a0aa427f27e245993b (127.0.104.3). New cstate: current_term: 1 leader_uuid: "db42be7ad3c741a0aa427f27e245993b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: UNKNOWN } } }
I20251025 07:52:40.222396 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 509c94f2a422493ea4390ecee576db72 (table default.Table_3f081b7bb4e1420c8121ea4115439f3d [id=696a1285fbfe4038a8b1b38708c8ea49]). Expected version 1 got 0
I20251025 07:52:40.222702 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet ab42909afc524f2fa3c1041495d13a59 (table default.Table_7eb8424d5192485da1a2d67bfdb2bbbf [id=40c29df99590408cbd5c583d4fc741eb]). Expected version 2 got 0
I20251025 07:52:40.223104 446 catalog_manager.cc:5649] T ef1854d35a3047e3a9fbe0166ab1295a P db42be7ad3c741a0aa427f27e245993b reported cstate change: term changed from 0 to 1, leader changed from <none> to db42be7ad3c741a0aa427f27e245993b (127.0.104.3). New cstate: current_term: 1 leader_uuid: "db42be7ad3c741a0aa427f27e245993b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: UNKNOWN } } }
I20251025 07:52:40.223546 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet ef1854d35a3047e3a9fbe0166ab1295a (table default.Table_088aad2a46b041bfa29ed8e6929760b1 [id=834669c54dae449790818684fd8659bc]). Expected version 2 got 0
I20251025 07:52:40.223798 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet bc2d16b33ca8426abbd7bd38339c9f1d (table default.Table_4ea985df4c5b415c9570c64506a98cc4 [id=98d2b7797e33485e8872054aa2eb8ec9]). Expected version 1 got 0
I20251025 07:52:40.223856 2316 raft_consensus.cc:493] T ef9a2c9f2ccf432ab835396020872208 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:40.224069 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e]). Expected version 1 got 0
I20251025 07:52:40.224643 2316 raft_consensus.cc:515] T ef9a2c9f2ccf432ab835396020872208 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.225636 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef9a2c9f2ccf432ab835396020872208" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:40.225869 774 raft_consensus.cc:2468] T ef9a2c9f2ccf432ab835396020872208 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:40.226285 982 leader_election.cc:304] T ef9a2c9f2ccf432ab835396020872208 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:40.226722 1671 raft_consensus.cc:2804] T ef9a2c9f2ccf432ab835396020872208 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:40.226863 1671 raft_consensus.cc:493] T ef9a2c9f2ccf432ab835396020872208 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:40.227012 1671 raft_consensus.cc:3060] T ef9a2c9f2ccf432ab835396020872208 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:40.229818 1671 raft_consensus.cc:515] T ef9a2c9f2ccf432ab835396020872208 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.230921 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef9a2c9f2ccf432ab835396020872208" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:40.231153 774 raft_consensus.cc:3060] T ef9a2c9f2ccf432ab835396020872208 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:40.232374 2146 log.cc:1199] T 8c76d1b509f04ab1a10b83cff675c11e P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/8c76d1b509f04ab1a10b83cff675c11e
I20251025 07:52:40.232779 2146 ts_tablet_manager.cc:1950] T 8c76d1b509f04ab1a10b83cff675c11e P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:40.233589 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 62) took 1813 ms. Trace:
I20251025 07:52:40.233682 2146 rpcz_store.cc:276] 1025 07:52:38.420375 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.420434 (+ 59us) service_pool.cc:225] Handling call
1025 07:52:40.096419 (+1675985us) ts_tablet_manager.cc:1151] Deleting tablet 8c76d1b509f04ab1a10b83cff675c11e
1025 07:52:40.096429 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:40.153814 (+ 57385us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.174846 (+ 21032us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.233570 (+ 58724us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1662693}
I20251025 07:52:40.234162 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 8c76d1b509f04ab1a10b83cff675c11e (table default.Table_cf686e1101314426a3ab24fd46a8974f [id=13fbed1c733c42e2b45e4ca69580b06d]) successfully deleted
I20251025 07:52:40.172683 1166 raft_consensus.cc:359] T ef9a2c9f2ccf432ab835396020872208 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.180857 905 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88bfc23acd274041a9457079c1709d82" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:40.234542 1166 raft_consensus.cc:385] T ef9a2c9f2ccf432ab835396020872208 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:40.234617 1166 raft_consensus.cc:740] T ef9a2c9f2ccf432ab835396020872208 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:40.182467 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88bfc23acd274041a9457079c1709d82" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:40.234848 1166 consensus_queue.cc:260] T ef9a2c9f2ccf432ab835396020872208 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.188638 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f43ef51ee45d47a99ec732b49d50d3b4" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:40.235345 1166 ts_tablet_manager.cc:1434] T ef9a2c9f2ccf432ab835396020872208 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.063s user 0.000s sys 0.001s
W20251025 07:52:40.235435 983 leader_election.cc:343] T 88bfc23acd274041a9457079c1709d82 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:40.235826 983 leader_election.cc:343] T 88bfc23acd274041a9457079c1709d82 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:40.236549 710 leader_election.cc:343] T f43ef51ee45d47a99ec732b49d50d3b4 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:40.236694 710 leader_election.cc:304] T f43ef51ee45d47a99ec732b49d50d3b4 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:40.236953 1659 raft_consensus.cc:2749] T f43ef51ee45d47a99ec732b49d50d3b4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:40.237453 774 raft_consensus.cc:2468] T ef9a2c9f2ccf432ab835396020872208 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:40.238823 1166 tablet_bootstrap.cc:492] T 88bfc23acd274041a9457079c1709d82 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:40.240496 1166 tablet_bootstrap.cc:654] T 88bfc23acd274041a9457079c1709d82 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:40.241397 982 leader_election.cc:304] T ef9a2c9f2ccf432ab835396020872208 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:40.241887 2300 raft_consensus.cc:2804] T ef9a2c9f2ccf432ab835396020872208 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:40.242228 2300 raft_consensus.cc:697] T ef9a2c9f2ccf432ab835396020872208 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:40.242638 2300 consensus_queue.cc:237] T ef9a2c9f2ccf432ab835396020872208 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.243505 961 heartbeater.cc:344] Connected to a master server at 127.0.104.61:42203
I20251025 07:52:40.247473 1659 raft_consensus.cc:493] T c2713942584e4734acbc0775ead772fa P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:40.247615 1659 raft_consensus.cc:515] T c2713942584e4734acbc0775ead772fa P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:40.248200 1659 leader_election.cc:290] T c2713942584e4734acbc0775ead772fa P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:40.248816 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2713942584e4734acbc0775ead772fa" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:40.248827 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2713942584e4734acbc0775ead772fa" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:40.249135 2146 tablet_replica.cc:333] T 4037dee385d04e85bb78c8e77cb00379 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
W20251025 07:52:40.249284 710 leader_election.cc:343] T c2713942584e4734acbc0775ead772fa P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:40.250672 712 leader_election.cc:343] T c2713942584e4734acbc0775ead772fa P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:40.250794 712 leader_election.cc:304] T c2713942584e4734acbc0775ead772fa P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:40.249370 2146 raft_consensus.cc:2243] T 4037dee385d04e85bb78c8e77cb00379 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:40.251233 1659 raft_consensus.cc:2749] T c2713942584e4734acbc0775ead772fa P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:40.251610 2146 raft_consensus.cc:2272] T 4037dee385d04e85bb78c8e77cb00379 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:40.253968 2316 leader_election.cc:290] T ef9a2c9f2ccf432ab835396020872208 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:40.253813 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef9a2c9f2ccf432ab835396020872208" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:40.254199 902 raft_consensus.cc:2468] T ef9a2c9f2ccf432ab835396020872208 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:40.255304 2146 ts_tablet_manager.cc:1916] T 4037dee385d04e85bb78c8e77cb00379 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:40.255486 1167 tablet_bootstrap.cc:492] T 04449cc97f404a19b6a1b20dea074f95 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:40.255623 1167 ts_tablet_manager.cc:1403] T 04449cc97f404a19b6a1b20dea074f95 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.128s user 0.002s sys 0.002s
I20251025 07:52:40.256455 1167 raft_consensus.cc:359] T 04449cc97f404a19b6a1b20dea074f95 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:40.256625 1167 raft_consensus.cc:385] T 04449cc97f404a19b6a1b20dea074f95 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:40.256691 1167 raft_consensus.cc:740] T 04449cc97f404a19b6a1b20dea074f95 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:40.257051 1167 consensus_queue.cc:260] T 04449cc97f404a19b6a1b20dea074f95 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:40.257661 1167 ts_tablet_manager.cc:1434] T 04449cc97f404a19b6a1b20dea074f95 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:40.257936 1167 tablet_bootstrap.cc:492] T cb829649f73d4d3f8862d268c953bd86 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:40.259330 1167 tablet_bootstrap.cc:654] T cb829649f73d4d3f8862d268c953bd86 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:40.261279 2050 consensus_queue.cc:1048] T df1248f6dbd94af68c587da7d149e279 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:40.273715 1166 tablet_bootstrap.cc:492] T 88bfc23acd274041a9457079c1709d82 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:40.273874 1166 ts_tablet_manager.cc:1403] T 88bfc23acd274041a9457079c1709d82 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.001s
I20251025 07:52:40.274660 1166 raft_consensus.cc:359] T 88bfc23acd274041a9457079c1709d82 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.274835 1166 raft_consensus.cc:385] T 88bfc23acd274041a9457079c1709d82 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:40.274890 1166 raft_consensus.cc:740] T 88bfc23acd274041a9457079c1709d82 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:40.275136 1166 consensus_queue.cc:260] T 88bfc23acd274041a9457079c1709d82 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.275835 1166 ts_tablet_manager.cc:1434] T 88bfc23acd274041a9457079c1709d82 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:40.276113 1166 tablet_bootstrap.cc:492] T b94634ac5077439fadfbbce569f60e8b P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:40.276811 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 64) took 1759 ms. Trace:
I20251025 07:52:40.278335 1166 tablet_bootstrap.cc:654] T b94634ac5077439fadfbbce569f60e8b P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:40.277853 2166 rpcz_store.cc:276] 1025 07:52:38.516991 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.517029 (+ 38us) service_pool.cc:225] Handling call
1025 07:52:40.128662 (+1611633us) ts_tablet_manager.cc:1151] Deleting tablet 4b28215f61574cf48c447dd01955ac77
1025 07:52:40.128673 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:40.136135 (+ 7462us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.157554 (+ 21419us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.276779 (+119225us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1611525}
I20251025 07:52:40.279125 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 4b28215f61574cf48c447dd01955ac77 (table default.Table_a437a2623d704e3ea3c5b8c65c88b936 [id=51b12903e6024e1a9adc2f9fa7400937]) successfully deleted
I20251025 07:52:40.279325 2166 tablet_replica.cc:333] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:40.279521 2166 raft_consensus.cc:2243] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:40.279824 2166 raft_consensus.cc:2272] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:40.280396 2166 ts_tablet_manager.cc:1916] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:40.280829 894 tablet_service.cc:1505] Processing CreateTablet for tablet a95a449e02b44ed58a877b8da05c28ea (DEFAULT_TABLE table=default.Table_a88d62768a0043398282bc79b0d5c96f [id=88b19cf775b34475b3d42219ad35d78e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:40.281196 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a95a449e02b44ed58a877b8da05c28ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:40.281972 754 tablet_service.cc:1505] Processing CreateTablet for tablet a95a449e02b44ed58a877b8da05c28ea (DEFAULT_TABLE table=default.Table_a88d62768a0043398282bc79b0d5c96f [id=88b19cf775b34475b3d42219ad35d78e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:40.283123 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a95a449e02b44ed58a877b8da05c28ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:40.283116 883 tablet_service.cc:1505] Processing CreateTablet for tablet 14dd1307f1244355b2efab4bb60107bd (DEFAULT_TABLE table=default.Table_1a207ab585e8456bad8fc10a6244d621 [id=d3764ccae44342d0b2c505f0ee377954]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:40.283435 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14dd1307f1244355b2efab4bb60107bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:40.286536 1017 tablet_service.cc:1505] Processing CreateTablet for tablet a95a449e02b44ed58a877b8da05c28ea (DEFAULT_TABLE table=default.Table_a88d62768a0043398282bc79b0d5c96f [id=88b19cf775b34475b3d42219ad35d78e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:40.291512 1018 tablet_service.cc:1505] Processing CreateTablet for tablet 14dd1307f1244355b2efab4bb60107bd (DEFAULT_TABLE table=default.Table_1a207ab585e8456bad8fc10a6244d621 [id=d3764ccae44342d0b2c505f0ee377954]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:40.292775 1010 tablet_service.cc:1505] Processing CreateTablet for tablet 8c2b78a664a14fb2b3c6ba27c69db908 (DEFAULT_TABLE table=default.Table_f56958ab784443f39df5bef925069408 [id=3e8c5a34005d4c5385f0c96d3b06fe49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:40.295219 883 tablet_service.cc:1505] Processing CreateTablet for tablet d8174034c1e74dc095a17cee4c5555e9 (DEFAULT_TABLE table=default.Table_834c71dad0864775a527dda8c2247132 [id=4ab62120b8394a07ae0b7452c8f166eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:40.295595 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8174034c1e74dc095a17cee4c5555e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:40.295560 745 tablet_service.cc:1505] Processing CreateTablet for tablet 8c2b78a664a14fb2b3c6ba27c69db908 (DEFAULT_TABLE table=default.Table_f56958ab784443f39df5bef925069408 [id=3e8c5a34005d4c5385f0c96d3b06fe49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:40.295915 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c2b78a664a14fb2b3c6ba27c69db908. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:40.302774 1011 tablet_service.cc:1505] Processing CreateTablet for tablet d8174034c1e74dc095a17cee4c5555e9 (DEFAULT_TABLE table=default.Table_834c71dad0864775a527dda8c2247132 [id=4ab62120b8394a07ae0b7452c8f166eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:40.303544 746 tablet_service.cc:1505] Processing CreateTablet for tablet d8174034c1e74dc095a17cee4c5555e9 (DEFAULT_TABLE table=default.Table_834c71dad0864775a527dda8c2247132 [id=4ab62120b8394a07ae0b7452c8f166eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:40.303957 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8174034c1e74dc095a17cee4c5555e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:40.307344 875 tablet_service.cc:1505] Processing CreateTablet for tablet b56209e4934e4407a8b4518ca4af0d46 (DEFAULT_TABLE table=default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:40.307708 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b56209e4934e4407a8b4518ca4af0d46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:40.309391 754 tablet_service.cc:1505] Processing CreateTablet for tablet 14dd1307f1244355b2efab4bb60107bd (DEFAULT_TABLE table=default.Table_1a207ab585e8456bad8fc10a6244d621 [id=d3764ccae44342d0b2c505f0ee377954]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:40.310375 894 tablet_service.cc:1505] Processing CreateTablet for tablet 8c2b78a664a14fb2b3c6ba27c69db908 (DEFAULT_TABLE table=default.Table_f56958ab784443f39df5bef925069408 [id=3e8c5a34005d4c5385f0c96d3b06fe49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:40.311478 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14dd1307f1244355b2efab4bb60107bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:40.315881 1021 tablet_service.cc:1505] Processing CreateTablet for tablet b56209e4934e4407a8b4518ca4af0d46 (DEFAULT_TABLE table=default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:40.316505 1167 tablet_bootstrap.cc:492] T cb829649f73d4d3f8862d268c953bd86 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:40.316633 1167 ts_tablet_manager.cc:1403] T cb829649f73d4d3f8862d268c953bd86 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.059s user 0.002s sys 0.002s
I20251025 07:52:40.316633 885 tablet_service.cc:1505] Processing CreateTablet for tablet 783487b60c144f719ed2eb3c67b5ccab (DEFAULT_TABLE table=default.Table_915572e49894471d8933816077123d6c [id=e372ca4384644779a6b37f389f549dec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:40.317040 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 783487b60c144f719ed2eb3c67b5ccab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:40.317297 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c2b78a664a14fb2b3c6ba27c69db908. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:40.318104 746 tablet_service.cc:1505] Processing CreateTablet for tablet b56209e4934e4407a8b4518ca4af0d46 (DEFAULT_TABLE table=default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:40.319198 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef9a2c9f2ccf432ab835396020872208" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:40.319363 903 raft_consensus.cc:3060] T ef9a2c9f2ccf432ab835396020872208 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:40.320034 2146 ts_tablet_manager.cc:1929] T 4037dee385d04e85bb78c8e77cb00379 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:40.320138 2146 log.cc:1199] T 4037dee385d04e85bb78c8e77cb00379 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/4037dee385d04e85bb78c8e77cb00379
I20251025 07:52:40.320482 2146 ts_tablet_manager.cc:1950] T 4037dee385d04e85bb78c8e77cb00379 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:40.321214 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 78) took 1880 ms. Trace:
I20251025 07:52:40.321316 2146 rpcz_store.cc:276] 1025 07:52:38.441259 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.441580 (+ 321us) service_pool.cc:225] Handling call
1025 07:52:40.249086 (+1807506us) ts_tablet_manager.cc:1151] Deleting tablet 4037dee385d04e85bb78c8e77cb00379
1025 07:52:40.249097 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:40.279387 (+ 30290us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.319941 (+ 40554us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.321195 (+ 1254us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1807324}
I20251025 07:52:40.321727 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 4037dee385d04e85bb78c8e77cb00379 (table default.Table_892da226f7054e61b5ffc882b1d5c35a [id=c13e40cbb3624b3997e907d4f2c1f706]) successfully deleted
I20251025 07:52:40.321995 903 raft_consensus.cc:2468] T ef9a2c9f2ccf432ab835396020872208 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:40.323797 1671 leader_election.cc:290] T ef9a2c9f2ccf432ab835396020872208 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:40.325731 1167 raft_consensus.cc:359] T cb829649f73d4d3f8862d268c953bd86 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:40.326395 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 783487b60c144f719ed2eb3c67b5ccab (DEFAULT_TABLE table=default.Table_915572e49894471d8933816077123d6c [id=e372ca4384644779a6b37f389f549dec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:40.326536 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a95a449e02b44ed58a877b8da05c28ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:40.326695 875 tablet_service.cc:1505] Processing CreateTablet for tablet 181059cc79d54999bf58504c85b739e9 (DEFAULT_TABLE table=default.Table_19a3326e4da345c1bbeb2cf083d28e24 [id=11c088570f6940b1895d1691465c10b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:40.326795 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 783487b60c144f719ed2eb3c67b5ccab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:40.326978 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8174034c1e74dc095a17cee4c5555e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:40.327010 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 181059cc79d54999bf58504c85b739e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:40.327425 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c2b78a664a14fb2b3c6ba27c69db908. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:40.328202 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b56209e4934e4407a8b4518ca4af0d46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:40.328644 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14dd1307f1244355b2efab4bb60107bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:40.332650 1019 tablet_service.cc:1505] Processing CreateTablet for tablet 181059cc79d54999bf58504c85b739e9 (DEFAULT_TABLE table=default.Table_19a3326e4da345c1bbeb2cf083d28e24 [id=11c088570f6940b1895d1691465c10b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:40.332993 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 181059cc79d54999bf58504c85b739e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:40.333474 885 tablet_service.cc:1505] Processing CreateTablet for tablet 70070d912e414b54bf6d07b597690a49 (DEFAULT_TABLE table=default.Table_dd5cb130dd9441929fa1e3efe915743b [id=8c21f3a42a9b42d09aa834b3b15f8a72]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:40.333774 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70070d912e414b54bf6d07b597690a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:40.340396 1019 tablet_service.cc:1505] Processing CreateTablet for tablet 70070d912e414b54bf6d07b597690a49 (DEFAULT_TABLE table=default.Table_dd5cb130dd9441929fa1e3efe915743b [id=8c21f3a42a9b42d09aa834b3b15f8a72]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:40.340749 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70070d912e414b54bf6d07b597690a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:40.342905 883 tablet_service.cc:1505] Processing CreateTablet for tablet 5895665ba20f4287b7212003f0d39af9 (DEFAULT_TABLE table=default.Table_1a037b3b43c74608a119c8f6c2007fcc [id=3aed5c86095a4e5bba85822bfd084c8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:40.343227 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5895665ba20f4287b7212003f0d39af9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:40.347571 1023 tablet_service.cc:1505] Processing CreateTablet for tablet 5895665ba20f4287b7212003f0d39af9 (DEFAULT_TABLE table=default.Table_1a037b3b43c74608a119c8f6c2007fcc [id=3aed5c86095a4e5bba85822bfd084c8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:40.347918 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5895665ba20f4287b7212003f0d39af9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:40.356637 875 tablet_service.cc:1505] Processing CreateTablet for tablet 6fe43ccecf7047319090ebd035fe24a5 (DEFAULT_TABLE table=default.Table_617594400ddd4cf4a8743365e27af63f [id=c95fc6d40893462eacee1276fb9cacfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:40.357549 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fe43ccecf7047319090ebd035fe24a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:40.357709 1023 tablet_service.cc:1505] Processing CreateTablet for tablet 6fe43ccecf7047319090ebd035fe24a5 (DEFAULT_TABLE table=default.Table_617594400ddd4cf4a8743365e27af63f [id=c95fc6d40893462eacee1276fb9cacfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:40.358747 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fe43ccecf7047319090ebd035fe24a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:40.362959 878 tablet_service.cc:1505] Processing CreateTablet for tablet 2ef3bffa73524aa29c311e24eb70e548 (DEFAULT_TABLE table=default.Table_ae530c9ed4814fb999a45b84c6c01634 [id=3ed746e8ad674278ad155425af7478f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:40.363373 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ef3bffa73524aa29c311e24eb70e548. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:40.372342 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 79) took 1930 ms. Trace:
I20251025 07:52:40.372450 2146 rpcz_store.cc:276] 1025 07:52:38.441379 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.442215 (+ 836us) service_pool.cc:225] Handling call
1025 07:52:40.372148 (+1929933us) ts_tablet_manager.cc:1151] Deleting tablet e47c4f741f354a19b49777bd06e54764
1025 07:52:40.372157 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:40.372316 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1929766}
W20251025 07:52:40.372938 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet e47c4f741f354a19b49777bd06e54764 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e47c4f741f354a19b49777bd06e54764 already in progress: creating tablet
I20251025 07:52:40.373032 2300 consensus_queue.cc:1048] T df1248f6dbd94af68c587da7d149e279 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:40.376837 1166 tablet_bootstrap.cc:492] T b94634ac5077439fadfbbce569f60e8b P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:40.376971 1166 ts_tablet_manager.cc:1403] T b94634ac5077439fadfbbce569f60e8b P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.101s user 0.000s sys 0.003s
I20251025 07:52:40.377722 1166 raft_consensus.cc:359] T b94634ac5077439fadfbbce569f60e8b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.377898 1166 raft_consensus.cc:385] T b94634ac5077439fadfbbce569f60e8b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:40.377964 1166 raft_consensus.cc:740] T b94634ac5077439fadfbbce569f60e8b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:40.378262 1166 consensus_queue.cc:260] T b94634ac5077439fadfbbce569f60e8b P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.378834 1166 ts_tablet_manager.cc:1434] T b94634ac5077439fadfbbce569f60e8b P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251025 07:52:40.378985 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 2ef3bffa73524aa29c311e24eb70e548 (DEFAULT_TABLE table=default.Table_ae530c9ed4814fb999a45b84c6c01634 [id=3ed746e8ad674278ad155425af7478f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:40.379097 1166 tablet_bootstrap.cc:492] T 88eb51447e0f4080baa49dd902deb24f P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:40.379468 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ef3bffa73524aa29c311e24eb70e548. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:40.380892 1166 tablet_bootstrap.cc:654] T 88eb51447e0f4080baa49dd902deb24f P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:40.386415 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 81) took 1944 ms. Trace:
I20251025 07:52:40.386523 2146 rpcz_store.cc:276] 1025 07:52:38.441541 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.443425 (+ 1884us) service_pool.cc:225] Handling call
1025 07:52:40.386254 (+1942829us) ts_tablet_manager.cc:1151] Deleting tablet cc407c7eb12e4eadb41fb21385611c94
1025 07:52:40.386266 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:40.386387 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1942601}
I20251025 07:52:40.386652 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5]). Expected version 1 got 0
W20251025 07:52:40.387035 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet cc407c7eb12e4eadb41fb21385611c94 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc407c7eb12e4eadb41fb21385611c94 already in progress: creating tablet
I20251025 07:52:40.387238 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3]). Expected version 1 got 0
I20251025 07:52:40.387508 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_a6c869949b744bc98efb735b0e73e7aa [id=bf77b5ff6f1048dd996d948437f9ff5e]). Expected version 2 got 0
I20251025 07:52:40.387729 2169 ts_tablet_manager.cc:1929] T 0b21b7da1e2f47829c3f6cdaee689e8a P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:40.388166 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f]). Expected version 1 got 0
I20251025 07:52:40.388697 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7]). Expected version 1 got 0
W20251025 07:52:40.327316 710 consensus_peers.cc:597] T fa77085e81c7440d9e255c44d1e80c26 P 23eeffd3de254ce18db5febfde971c6b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fa77085e81c7440d9e255c44d1e80c26. This is attempt 1: this message will repeat every 5th retry.
W20251025 07:52:40.389811 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:43563, user_credentials={real_user=slave}} blocked reactor thread for 62532.4us
I20251025 07:52:40.328796 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b56209e4934e4407a8b4518ca4af0d46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:40.338615 752 tablet_service.cc:1505] Processing CreateTablet for tablet 181059cc79d54999bf58504c85b739e9 (DEFAULT_TABLE table=default.Table_19a3326e4da345c1bbeb2cf083d28e24 [id=11c088570f6940b1895d1691465c10b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:40.342553 741 tablet_service.cc:1505] Processing CreateTablet for tablet 70070d912e414b54bf6d07b597690a49 (DEFAULT_TABLE table=default.Table_dd5cb130dd9441929fa1e3efe915743b [id=8c21f3a42a9b42d09aa834b3b15f8a72]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:40.356134 755 tablet_service.cc:1505] Processing CreateTablet for tablet 5895665ba20f4287b7212003f0d39af9 (DEFAULT_TABLE table=default.Table_1a037b3b43c74608a119c8f6c2007fcc [id=3aed5c86095a4e5bba85822bfd084c8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:40.358471 756 tablet_service.cc:1505] Processing CreateTablet for tablet 6fe43ccecf7047319090ebd035fe24a5 (DEFAULT_TABLE table=default.Table_617594400ddd4cf4a8743365e27af63f [id=c95fc6d40893462eacee1276fb9cacfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:40.391103 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 872cb7512d104827b1f1f186d33a4a44 (table default.Table_d5c65dce8b1b4214ba519d964012bfc0 [id=d4be274b36a845a99a4b96b8846248fd]). Expected version 1 got 0
I20251025 07:52:40.391287 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 181059cc79d54999bf58504c85b739e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:40.391314 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet cb829649f73d4d3f8862d268c953bd86 (table default.Table_ba4b8cc7b930482690b7e192e0edff12 [id=7944225d3ce7439b881bfb9289d9f040]). Expected version 1 got 0
I20251025 07:52:40.391481 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 04d27d54fdb7483b869408af65ff85c9 (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]). Expected version 2 got 0
I20251025 07:52:40.391721 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 4c155aa7fa5b4d68bbab1918b8432372 (table default.Table_6c5a24747ba14352b0285a41ec2985a4 [id=28bbaad474164a8c8c97a76bb8081791]). Expected version 2 got 0
I20251025 07:52:40.392735 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet aa9d9ab8073945d28d52fb02213f201a (table default.Table_4791672658f043f6b97b9320a531ebb5 [id=dfd20cf9c3e541c4abe23d59e0aa7f2a]). Expected version 1 got 0
I20251025 07:52:40.393082 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 292fe04055954afd8c08ac2636b68191 (table default.Table_9fa69b10b5e842639fc0b0a58cd9b6a5 [id=a253ce0253bd4da4814741c482193d8f]). Expected version 1 got 0
I20251025 07:52:40.393316 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]). Expected version 1 got 0
I20251025 07:52:40.393497 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet ef9a2c9f2ccf432ab835396020872208 (table default.Table_2b9b772d4c234ac286bef3658482e490 [id=409fab60f9d744798e6f92b868f0f7c2]). Expected version 1 got 0
I20251025 07:52:40.393643 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]). Expected version 1 got 0
I20251025 07:52:40.393887 453 catalog_manager.cc:5649] T df1248f6dbd94af68c587da7d149e279 P db42be7ad3c741a0aa427f27e245993b reported cstate change: term changed from 0 to 1, leader changed from <none> to db42be7ad3c741a0aa427f27e245993b (127.0.104.3). New cstate: current_term: 1 leader_uuid: "db42be7ad3c741a0aa427f27e245993b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: UNKNOWN } } }
I20251025 07:52:40.394447 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5895665ba20f4287b7212003f0d39af9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:40.394601 1659 raft_consensus.cc:493] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:40.394801 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 22f7db660616457b83b24e947a0b0bae (table default.Table_4a86191163f2408f9bcc795cd8d1bbf6 [id=bb316a7e000748ca902a9bfb494802d7]). Expected version 1 got 0
I20251025 07:52:40.394739 1659 raft_consensus.cc:515] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:40.394964 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84]). Expected version 1 got 0
I20251025 07:52:40.395277 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 5dab439bc50d4e869548154e1d222ad6 (table default.Table_bd82d871f84943f4baa967f5c0ec74a3 [id=45325c188eac4d34bb016d01de091f87]). Expected version 2 got 0
I20251025 07:52:40.395411 1659 leader_election.cc:290] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:40.395515 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_87d38832f85e4965bfab1a077990114f [id=ad812862698d421184ebf501ce79b7be]). Expected version 2 got 0
I20251025 07:52:40.395753 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet f70091024c034c118fb86578db19a31c (table default.Table_332b308220f64c6dbb5f29711d8463b3 [id=e041946c093b4c789ce175c736cc230b]). Expected version 1 got 0
I20251025 07:52:40.396895 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 755f0d6c87044e20a8020f8410940123 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 4 got 0
I20251025 07:52:40.397064 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236]). Expected version 1 got 0
I20251025 07:52:40.397804 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 0325d462441349ef81841f1b0cab2a20 (table default.Table_750ecb905839434fa9a1aa72f87394be [id=9ce3ba77c56a4ef09cb3b40ea0e488f6]). Expected version 1 got 0
I20251025 07:52:40.398032 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]). Expected version 1 got 0
I20251025 07:52:40.398348 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fe43ccecf7047319090ebd035fe24a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:40.398507 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 830fdbc6812e416690d0db42a73af503 (table default.Table_92824b3f2a774564b3aea04aaa25d204 [id=fb21a45b86ab446e8c3dae4512a9598f]). Expected version 1 got 0
I20251025 07:52:40.398676 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_e90d7f5ff13b40878e5b625c97792c7d [id=fc8180c09d7b4e74b2e161f7acb8fe0e]). Expected version 1 got 0
I20251025 07:52:40.399245 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c]). Expected version 1 got 0
I20251025 07:52:40.399633 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]). Expected version 3 got 0
I20251025 07:52:40.399890 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 8f2c8582fed34c848d9903b8ad8b4057 (table default.Table_438f67f3dc62465999a917986922b3ed [id=7c7fcd265e624d7fab74661db505d9c3]). Expected version 1 got 0
I20251025 07:52:40.400038 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525]). Expected version 1 got 0
I20251025 07:52:40.400373 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 36262ab8c6964dd684c01b4db6219a4f (table default.Table_76158f8cf82e4cdb9a90580e46aaea48 [id=f249f30497bc450c887d659721469de8]). Expected version 1 got 0
I20251025 07:52:40.400765 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 2c50ec95e48e49b59a76fac4fc862d3d (table default.Table_32b54e82cbc84d39a0020826f8737d62 [id=c28b0b6acd61490caecfdaab803a9d92]). Expected version 2 got 0
I20251025 07:52:40.400982 453 catalog_manager.cc:5649] T fa998b9ca6724c4cb1eb68a3eecb0b11 P db42be7ad3c741a0aa427f27e245993b reported cstate change: term changed from 0 to 1, leader changed from <none> to db42be7ad3c741a0aa427f27e245993b (127.0.104.3). New cstate: current_term: 1 leader_uuid: "db42be7ad3c741a0aa427f27e245993b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: UNKNOWN } } }
I20251025 07:52:40.401752 1167 raft_consensus.cc:385] T cb829649f73d4d3f8862d268c953bd86 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:40.401829 1167 raft_consensus.cc:740] T cb829649f73d4d3f8862d268c953bd86 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:40.402204 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet b17d2a9ad6be49d29f2413666fba7e69 (table default.Table_fcddb31989ee4236a4981c12779b3f5d [id=f3794fa081364666b944c412415bdba0]). Expected version 3 got 0
I20251025 07:52:40.402410 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3]). Expected version 2 got 0
I20251025 07:52:40.390745 1169 tablet_bootstrap.cc:654] T ebf3a2a926754305b5bd95fcebb25d29 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:40.391027 2300 raft_consensus.cc:493] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:40.402697 2300 raft_consensus.cc:515] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.403487 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70070d912e414b54bf6d07b597690a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:40.403620 2300 leader_election.cc:290] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:40.396042 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c87ab6a3d08f43e0b2fe58a6d7b11588" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:40.404625 712 leader_election.cc:343] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:40.405378 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 509c94f2a422493ea4390ecee576db72 (table default.Table_3f081b7bb4e1420c8121ea4115439f3d [id=696a1285fbfe4038a8b1b38708c8ea49]). Expected version 1 got 0
I20251025 07:52:40.405668 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet ab42909afc524f2fa3c1041495d13a59 (table default.Table_7eb8424d5192485da1a2d67bfdb2bbbf [id=40c29df99590408cbd5c583d4fc741eb]). Expected version 2 got 0
I20251025 07:52:40.406455 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet ef1854d35a3047e3a9fbe0166ab1295a (table default.Table_088aad2a46b041bfa29ed8e6929760b1 [id=834669c54dae449790818684fd8659bc]). Expected version 2 got 0
I20251025 07:52:40.406852 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet bc2d16b33ca8426abbd7bd38339c9f1d (table default.Table_4ea985df4c5b415c9570c64506a98cc4 [id=98d2b7797e33485e8872054aa2eb8ec9]). Expected version 1 got 0
I20251025 07:52:40.407148 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e]). Expected version 1 got 0
I20251025 07:52:40.407613 454 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet b02f5c0ab0b24288bb50a3ba0e6bd78a
I20251025 07:52:40.408326 454 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:47610 (request call id 53) took 2202 ms. Trace:
I20251025 07:52:40.408710 875 tablet_service.cc:1552] Processing DeleteTablet for tablet b02f5c0ab0b24288bb50a3ba0e6bd78a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:40.408443 454 rpcz_store.cc:276] 1025 07:52:38.205888 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:39.158721 (+952833us) service_pool.cc:225] Handling call
1025 07:52:39.159270 (+ 549us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:39.159417 (+ 147us) catalog_manager.cc:2815] Modifying in-memory table state
1025 07:52:39.159481 (+ 64us) catalog_manager.cc:2824] Locking tablets
1025 07:52:39.159617 (+ 136us) catalog_manager.cc:2838] Removing table and tablets from system table
1025 07:52:39.167643 (+ 8026us) catalog_manager.cc:2856] Removing table from by-name map
1025 07:52:39.237834 (+ 70191us) catalog_manager.cc:2872] Committing in-memory state
1025 07:52:40.407559 (+1169725us) catalog_manager.cc:2876] Aborting table tasks
1025 07:52:40.408254 (+ 695us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:39.160384 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:39.160500 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:39.160506 (+ 6us) write_op.cc:435] Acquired schema lock
1025 07:52:39.160508 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:39.160621 (+ 113us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:39.160637 (+ 16us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:39.160644 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:39.160773 (+ 129us) tablet.cc:699] Row locks acquired
1025 07:52:39.160775 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:39.160809 (+ 34us) write_op.cc:270] Start()
1025 07:52:39.160848 (+ 39us) write_op.cc:276] Timestamp: P: 1761378759160792 usec, L: 0
1025 07:52:39.160853 (+ 5us) op_driver.cc:348] REPLICATION: starting
1025 07:52:39.161057 (+ 204us) log.cc:844] Serialized 725 byte log entry
1025 07:52:39.164645 (+ 3588us) op_driver.cc:464] REPLICATION: finished
1025 07:52:39.164953 (+ 308us) write_op.cc:301] APPLY: starting
1025 07:52:39.164971 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:39.165015 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:39.165049 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:39.165125 (+ 76us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:39.165165 (+ 40us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1025 07:52:39.165173 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:39.165251 (+ 78us) log.cc:844] Serialized 29 byte log entry
1025 07:52:39.165315 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:39.165340 (+ 25us) write_op.cc:454] Released schema lock
1025 07:52:39.165349 (+ 9us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:39.165429 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:39.165435 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":244,"apply.run_cpu_time_us":470,"apply.run_wall_time_us":469,"num_ops":2,"prepare.queue_time_us":48,"prepare.run_cpu_time_us":781,"prepare.run_wall_time_us":782,"raft.queue_time_us":434,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":239,"replication_time_us":3759}]]}
I20251025 07:52:40.410722 752 tablet_service.cc:1552] Processing DeleteTablet for tablet b02f5c0ab0b24288bb50a3ba0e6bd78a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:58412
W20251025 07:52:40.413367 712 consensus_peers.cc:597] T fa77085e81c7440d9e255c44d1e80c26 P 23eeffd3de254ce18db5febfde971c6b -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fa77085e81c7440d9e255c44d1e80c26. This is attempt 1: this message will repeat every 5th retry.
W20251025 07:52:40.415702 982 leader_election.cc:343] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: b947ecc352e54f4caa73c6ca97372256
I20251025 07:52:40.417263 2169 log.cc:1199] T 0b21b7da1e2f47829c3f6cdaee689e8a P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/0b21b7da1e2f47829c3f6cdaee689e8a
I20251025 07:52:40.417981 2169 ts_tablet_manager.cc:1950] T 0b21b7da1e2f47829c3f6cdaee689e8a P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:40.371898 745 tablet_service.cc:1505] Processing CreateTablet for tablet 783487b60c144f719ed2eb3c67b5ccab (DEFAULT_TABLE table=default.Table_915572e49894471d8933816077123d6c [id=e372ca4384644779a6b37f389f549dec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:40.376782 754 tablet_service.cc:1505] Processing CreateTablet for tablet 2ef3bffa73524aa29c311e24eb70e548 (DEFAULT_TABLE table=default.Table_ae530c9ed4814fb999a45b84c6c01634 [id=3ed746e8ad674278ad155425af7478f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:40.418329 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ef3bffa73524aa29c311e24eb70e548. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:40.418416 1167 consensus_queue.cc:260] T cb829649f73d4d3f8862d268c953bd86 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:40.423629 452 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet a77eb82f26ae43b48de1e85693372d34
I20251025 07:52:40.424444 452 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:47610 (request call id 51) took 2268 ms. Trace:
I20251025 07:52:40.424576 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 783487b60c144f719ed2eb3c67b5ccab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:40.424580 452 rpcz_store.cc:276] 1025 07:52:38.156366 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.544164 (+387798us) service_pool.cc:225] Handling call
1025 07:52:38.544983 (+ 819us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:38.545256 (+ 273us) catalog_manager.cc:2815] Modifying in-memory table state
1025 07:52:38.545385 (+ 129us) catalog_manager.cc:2824] Locking tablets
1025 07:52:38.951915 (+406530us) catalog_manager.cc:2838] Removing table and tablets from system table
1025 07:52:38.960956 (+ 9041us) catalog_manager.cc:2856] Removing table from by-name map
1025 07:52:39.153441 (+192485us) catalog_manager.cc:2872] Committing in-memory state
1025 07:52:40.423562 (+1270121us) catalog_manager.cc:2876] Aborting table tasks
1025 07:52:40.424371 (+ 809us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:38.953138 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:38.953316 (+ 178us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:38.953325 (+ 9us) write_op.cc:435] Acquired schema lock
1025 07:52:38.953329 (+ 4us) tablet.cc:660] Decoding operations
1025 07:52:38.953480 (+ 151us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:38.953497 (+ 17us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:38.953504 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:38.953685 (+ 181us) tablet.cc:699] Row locks acquired
1025 07:52:38.953690 (+ 5us) write_op.cc:260] PREPARE: finished
1025 07:52:38.953738 (+ 48us) write_op.cc:270] Start()
1025 07:52:38.953776 (+ 38us) write_op.cc:276] Timestamp: P: 1761378758953718 usec, L: 0
1025 07:52:38.953783 (+ 7us) op_driver.cc:348] REPLICATION: starting
1025 07:52:38.954097 (+ 314us) log.cc:844] Serialized 725 byte log entry
1025 07:52:38.960239 (+ 6142us) op_driver.cc:464] REPLICATION: finished
1025 07:52:38.960375 (+ 136us) write_op.cc:301] APPLY: starting
1025 07:52:38.960414 (+ 39us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:38.960460 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:38.960490 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:38.960557 (+ 67us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:38.960589 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1025 07:52:38.960596 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:38.960663 (+ 67us) log.cc:844] Serialized 29 byte log entry
1025 07:52:38.960727 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.960748 (+ 21us) write_op.cc:454] Released schema lock
1025 07:52:38.960754 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:38.960826 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.960830 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":82,"apply.run_cpu_time_us":442,"apply.run_wall_time_us":442,"num_ops":2,"prepare.queue_time_us":224,"prepare.run_cpu_time_us":997,"prepare.run_wall_time_us":1864,"raft.queue_time_us":51,"raft.run_cpu_time_us":341,"raft.run_wall_time_us":341,"replication_time_us":6397}]]}
I20251025 07:52:40.430528 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c87ab6a3d08f43e0b2fe58a6d7b11588" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:40.431144 2146 tablet_replica.cc:333] T 7f29b64704e147aa9b0b5f6cda860f09 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:40.431318 2146 raft_consensus.cc:2243] T 7f29b64704e147aa9b0b5f6cda860f09 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:40.431479 2146 raft_consensus.cc:2272] T 7f29b64704e147aa9b0b5f6cda860f09 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:40.432049 2146 ts_tablet_manager.cc:1916] T 7f29b64704e147aa9b0b5f6cda860f09 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:40.432529 905 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b947ecc352e54f4caa73c6ca97372256" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:40.432636 754 tablet_service.cc:1552] Processing DeleteTablet for tablet a77eb82f26ae43b48de1e85693372d34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:40.432696 905 raft_consensus.cc:2468] T b947ecc352e54f4caa73c6ca97372256 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:40.433152 875 tablet_service.cc:1552] Processing DeleteTablet for tablet a77eb82f26ae43b48de1e85693372d34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:59616
W20251025 07:52:40.433461 710 leader_election.cc:343] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:40.433566 710 leader_election.cc:304] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:40.438230 1659 raft_consensus.cc:2749] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:40.438645 442 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_a6c869949b744bc98efb735b0e73e7aa [id=bf77b5ff6f1048dd996d948437f9ff5e], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet d46673a283fe42b78ddeb99fbd2678eb consensus state has no leader
I20251025 07:52:40.450484 1166 tablet_bootstrap.cc:492] T 88eb51447e0f4080baa49dd902deb24f P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:40.450778 1166 ts_tablet_manager.cc:1403] T 88eb51447e0f4080baa49dd902deb24f P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.072s user 0.001s sys 0.002s
I20251025 07:52:40.451720 1166 raft_consensus.cc:359] T 88eb51447e0f4080baa49dd902deb24f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:40.452009 1166 raft_consensus.cc:385] T 88eb51447e0f4080baa49dd902deb24f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:40.452109 1166 raft_consensus.cc:740] T 88eb51447e0f4080baa49dd902deb24f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:40.452368 1166 consensus_queue.cc:260] T 88eb51447e0f4080baa49dd902deb24f P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:40.452711 710 consensus_peers.cc:597] T 7f29b64704e147aa9b0b5f6cda860f09 P 23eeffd3de254ce18db5febfde971c6b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:40.453039 1166 ts_tablet_manager.cc:1434] T 88eb51447e0f4080baa49dd902deb24f P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:40.453455 1166 tablet_bootstrap.cc:492] T b3a0af893a8947848e12093d0a7f0229 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:40.456238 1166 tablet_bootstrap.cc:654] T b3a0af893a8947848e12093d0a7f0229 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:40.462343 2105 raft_consensus.cc:493] T 88bfc23acd274041a9457079c1709d82 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:40.462466 2105 raft_consensus.cc:515] T 88bfc23acd274041a9457079c1709d82 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.463054 2105 leader_election.cc:290] T 88bfc23acd274041a9457079c1709d82 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:40.464367 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88bfc23acd274041a9457079c1709d82" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:40.464639 776 raft_consensus.cc:2393] T 88bfc23acd274041a9457079c1709d82 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3aea2869ff814cc7b142012faa274382 in current term 1: Already voted for candidate db42be7ad3c741a0aa427f27e245993b in this term.
I20251025 07:52:40.460723 2146 ts_tablet_manager.cc:1929] T 7f29b64704e147aa9b0b5f6cda860f09 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:40.465903 2146 log.cc:1199] T 7f29b64704e147aa9b0b5f6cda860f09 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/7f29b64704e147aa9b0b5f6cda860f09
I20251025 07:52:40.466420 2146 ts_tablet_manager.cc:1950] T 7f29b64704e147aa9b0b5f6cda860f09 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:40.468711 823 heartbeater.cc:344] Connected to a master server at 127.0.104.61:42203
I20251025 07:52:40.471132 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 82) took 2027 ms. Trace:
I20251025 07:52:40.471293 2146 rpcz_store.cc:276] 1025 07:52:38.443864 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.444217 (+ 353us) service_pool.cc:225] Handling call
1025 07:52:40.390495 (+1946278us) ts_tablet_manager.cc:1151] Deleting tablet 7f29b64704e147aa9b0b5f6cda860f09
1025 07:52:40.430406 (+ 39911us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:40.436063 (+ 5657us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.460617 (+ 24554us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.471111 (+ 10494us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1946121}
W20251025 07:52:40.471599 710 consensus_peers.cc:597] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: be5ab0db6ac34c2c9a370445e251eb46. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:40.471972 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 7f29b64704e147aa9b0b5f6cda860f09 (table default.Table_2253f6700c2f4a3a9847023ba98b80ea [id=17c206c8e24647ab9f483d9944dd3191]) successfully deleted
I20251025 07:52:40.472469 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 83) took 2027 ms. Trace:
I20251025 07:52:40.472605 2146 rpcz_store.cc:276] 1025 07:52:38.444555 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.445998 (+ 1443us) service_pool.cc:225] Handling call
1025 07:52:40.472329 (+2026331us) ts_tablet_manager.cc:1151] Deleting tablet 716fb61312e94a4d9eb8ab8336ca140c
1025 07:52:40.472339 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:40.472453 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2026092}
W20251025 07:52:40.473132 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 716fb61312e94a4d9eb8ab8336ca140c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 716fb61312e94a4d9eb8ab8336ca140c already in progress: creating tablet
I20251025 07:52:40.473481 2146 tablet_replica.cc:333] T bef0ef23d90d480d9c99aef21d7271fd P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:40.473724 2146 raft_consensus.cc:2243] T bef0ef23d90d480d9c99aef21d7271fd P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:40.474010 2146 raft_consensus.cc:2272] T bef0ef23d90d480d9c99aef21d7271fd P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:40.474646 2146 ts_tablet_manager.cc:1916] T bef0ef23d90d480d9c99aef21d7271fd P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:40.475111 1166 tablet_bootstrap.cc:492] T b3a0af893a8947848e12093d0a7f0229 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:40.475319 1166 ts_tablet_manager.cc:1403] T b3a0af893a8947848e12093d0a7f0229 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20251025 07:52:40.476576 1166 raft_consensus.cc:359] T b3a0af893a8947848e12093d0a7f0229 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:40.476907 1166 raft_consensus.cc:385] T b3a0af893a8947848e12093d0a7f0229 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:40.477043 1166 raft_consensus.cc:740] T b3a0af893a8947848e12093d0a7f0229 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:40.477397 1166 consensus_queue.cc:260] T b3a0af893a8947848e12093d0a7f0229 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:40.478294 1166 ts_tablet_manager.cc:1434] T b3a0af893a8947848e12093d0a7f0229 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251025 07:52:40.479283 1166 tablet_bootstrap.cc:492] T fe5033e5ff404a5ebc6c8443d7548327 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:40.480211 447 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet 6897405bbc1c47a4ab0aa3af80e60e3d
I20251025 07:52:40.480896 447 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:47610 (request call id 52) took 2276 ms. Trace:
I20251025 07:52:40.481242 875 tablet_service.cc:1552] Processing DeleteTablet for tablet 6897405bbc1c47a4ab0aa3af80e60e3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:40.481010 447 rpcz_store.cc:276] 1025 07:52:38.204469 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.550735 (+346266us) service_pool.cc:225] Handling call
1025 07:52:38.551663 (+ 928us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:38.551953 (+ 290us) catalog_manager.cc:2815] Modifying in-memory table state
1025 07:52:38.552093 (+ 140us) catalog_manager.cc:2824] Locking tablets
1025 07:52:38.935614 (+383521us) catalog_manager.cc:2838] Removing table and tablets from system table
1025 07:52:38.949607 (+ 13993us) catalog_manager.cc:2856] Removing table from by-name map
1025 07:52:39.153376 (+203769us) catalog_manager.cc:2872] Committing in-memory state
1025 07:52:40.480151 (+1326775us) catalog_manager.cc:2876] Aborting table tasks
1025 07:52:40.480826 (+ 675us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:38.940807 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:38.940932 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:38.940939 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:38.940943 (+ 4us) tablet.cc:660] Decoding operations
1025 07:52:38.941052 (+ 109us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:38.941066 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:38.941071 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:38.941182 (+ 111us) tablet.cc:699] Row locks acquired
1025 07:52:38.941185 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:38.941222 (+ 37us) write_op.cc:270] Start()
1025 07:52:38.941254 (+ 32us) write_op.cc:276] Timestamp: P: 1761378758941205 usec, L: 0
1025 07:52:38.941260 (+ 6us) op_driver.cc:348] REPLICATION: starting
1025 07:52:38.941483 (+ 223us) log.cc:844] Serialized 790 byte log entry
1025 07:52:38.948881 (+ 7398us) op_driver.cc:464] REPLICATION: finished
1025 07:52:38.949020 (+ 139us) write_op.cc:301] APPLY: starting
1025 07:52:38.949050 (+ 30us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:38.949095 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:38.949129 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:38.949199 (+ 70us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:38.949232 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1025 07:52:38.949239 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:38.949323 (+ 84us) log.cc:844] Serialized 29 byte log entry
1025 07:52:38.949391 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.949416 (+ 25us) write_op.cc:454] Released schema lock
1025 07:52:38.949439 (+ 23us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:38.949507 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.949511 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":80,"apply.run_cpu_time_us":483,"apply.run_wall_time_us":483,"num_ops":2,"prepare.queue_time_us":4481,"prepare.run_cpu_time_us":785,"prepare.run_wall_time_us":785,"raft.queue_time_us":424,"raft.run_cpu_time_us":305,"raft.run_wall_time_us":304,"replication_time_us":7587}]]}
I20251025 07:52:40.481797 447 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_4976d4f5d05540258d144751d129bdb8" } modify_external_catalogs: true
I20251025 07:52:40.481967 447 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_4976d4f5d05540258d144751d129bdb8" } modify_external_catalogs: true
I20251025 07:52:40.483073 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 6897405bbc1c47a4ab0aa3af80e60e3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:40.484952 1167 ts_tablet_manager.cc:1434] T cb829649f73d4d3f8862d268c953bd86 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.168s user 0.001s sys 0.001s
I20251025 07:52:40.485246 1167 tablet_bootstrap.cc:492] T ab42909afc524f2fa3c1041495d13a59 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:40.430290 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet b02f5c0ab0b24288bb50a3ba0e6bd78a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:40.433444 983 leader_election.cc:304] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b
W20251025 07:52:40.489076 983 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:43563, user_credentials={real_user=slave}} blocked reactor thread for 55663.3us
I20251025 07:52:40.491189 1612 raft_consensus.cc:2804] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:40.491290 1612 raft_consensus.cc:493] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:40.491367 1612 raft_consensus.cc:3060] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:40.491493 2146 ts_tablet_manager.cc:1929] T bef0ef23d90d480d9c99aef21d7271fd P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:40.491590 2146 log.cc:1199] T bef0ef23d90d480d9c99aef21d7271fd P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/bef0ef23d90d480d9c99aef21d7271fd
I20251025 07:52:40.491954 2146 ts_tablet_manager.cc:1950] T bef0ef23d90d480d9c99aef21d7271fd P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:40.492257 1166 tablet_bootstrap.cc:654] T fe5033e5ff404a5ebc6c8443d7548327 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:40.492327 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 84) took 2045 ms. Trace:
I20251025 07:52:40.492390 2146 rpcz_store.cc:276] 1025 07:52:38.446404 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.446443 (+ 39us) service_pool.cc:225] Handling call
1025 07:52:40.473437 (+2026994us) ts_tablet_manager.cc:1151] Deleting tablet bef0ef23d90d480d9c99aef21d7271fd
1025 07:52:40.473446 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:40.477890 (+ 4444us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.491381 (+ 13491us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.492312 (+ 931us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2026855}
I20251025 07:52:40.492671 2146 tablet_replica.cc:333] T abf94c32ff2945119b710905c075d63c P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:40.494786 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet bef0ef23d90d480d9c99aef21d7271fd (table default.Table_e3b368ac698c46f4a8c947d02f9a2680 [id=ce5f22039d104e518ef6f40468b404cc]) successfully deleted
I20251025 07:52:40.494030 1612 raft_consensus.cc:515] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.497602 1612 leader_election.cc:290] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:40.498210 905 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b947ecc352e54f4caa73c6ca97372256" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:40.498361 905 raft_consensus.cc:3060] T b947ecc352e54f4caa73c6ca97372256 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:40.501185 905 raft_consensus.cc:2468] T b947ecc352e54f4caa73c6ca97372256 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:40.502307 983 leader_election.cc:304] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:40.502661 1612 consensus_queue.cc:1048] T 80ee75935d7e461d9d4066fac9819bef P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:40.503108 2050 raft_consensus.cc:2804] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:40.503397 2050 raft_consensus.cc:697] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:40.503746 2050 consensus_queue.cc:237] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.453075 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet a77eb82f26ae43b48de1e85693372d34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:40.463776 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88bfc23acd274041a9457079c1709d82" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:40.508852 850 leader_election.cc:304] T 88bfc23acd274041a9457079c1709d82 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:40.509348 2105 raft_consensus.cc:3060] T 88bfc23acd274041a9457079c1709d82 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:40.511071 2146 raft_consensus.cc:2243] T abf94c32ff2945119b710905c075d63c P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:40.511245 2146 raft_consensus.cc:2272] T abf94c32ff2945119b710905c075d63c P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:40.512320 2105 raft_consensus.cc:2749] T 88bfc23acd274041a9457079c1709d82 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:40.514866 2146 ts_tablet_manager.cc:1916] T abf94c32ff2945119b710905c075d63c P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:40.515389 1166 tablet_bootstrap.cc:492] T fe5033e5ff404a5ebc6c8443d7548327 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:40.515645 1166 ts_tablet_manager.cc:1403] T fe5033e5ff404a5ebc6c8443d7548327 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.037s user 0.000s sys 0.008s
I20251025 07:52:40.518590 2117 consensus_queue.cc:1048] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:40.519381 1166 raft_consensus.cc:359] T fe5033e5ff404a5ebc6c8443d7548327 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.519690 1166 raft_consensus.cc:385] T fe5033e5ff404a5ebc6c8443d7548327 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:40.519850 1166 raft_consensus.cc:740] T fe5033e5ff404a5ebc6c8443d7548327 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:40.520383 1166 consensus_queue.cc:260] T fe5033e5ff404a5ebc6c8443d7548327 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.524016 1166 ts_tablet_manager.cc:1434] T fe5033e5ff404a5ebc6c8443d7548327 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251025 07:52:40.525872 1166 tablet_bootstrap.cc:492] T e9f7cd765d614d59a7c190b744b0844d P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:40.481343 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 6897405bbc1c47a4ab0aa3af80e60e3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:40.539472 1166 tablet_bootstrap.cc:654] T e9f7cd765d614d59a7c190b744b0844d P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:40.542520 2146 ts_tablet_manager.cc:1929] T abf94c32ff2945119b710905c075d63c P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:40.544816 1048 raft_consensus.cc:3060] T f4578ce69a0d4322944e5ce7c8a59831 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:40.550855 1659 consensus_queue.cc:1048] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:40.561718 1659 raft_consensus.cc:493] T cb829649f73d4d3f8862d268c953bd86 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:40.561918 1659 raft_consensus.cc:515] T cb829649f73d4d3f8862d268c953bd86 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:40.562709 1659 leader_election.cc:290] T cb829649f73d4d3f8862d268c953bd86 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:40.563503 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb829649f73d4d3f8862d268c953bd86" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:40.564461 710 leader_election.cc:343] T cb829649f73d4d3f8862d268c953bd86 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:40.565196 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb829649f73d4d3f8862d268c953bd86" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:40.566025 1166 tablet_bootstrap.cc:492] T e9f7cd765d614d59a7c190b744b0844d P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:40.566257 1166 ts_tablet_manager.cc:1403] T e9f7cd765d614d59a7c190b744b0844d P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.041s user 0.007s sys 0.002s
W20251025 07:52:40.566411 712 leader_election.cc:343] T cb829649f73d4d3f8862d268c953bd86 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:40.566548 712 leader_election.cc:304] T cb829649f73d4d3f8862d268c953bd86 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:40.567023 1659 raft_consensus.cc:2749] T cb829649f73d4d3f8862d268c953bd86 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:40.567462 1166 raft_consensus.cc:359] T e9f7cd765d614d59a7c190b744b0844d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.567683 1166 raft_consensus.cc:385] T e9f7cd765d614d59a7c190b744b0844d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:40.567788 1166 raft_consensus.cc:740] T e9f7cd765d614d59a7c190b744b0844d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:40.568121 1166 consensus_queue.cc:260] T e9f7cd765d614d59a7c190b744b0844d P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.568964 1166 ts_tablet_manager.cc:1434] T e9f7cd765d614d59a7c190b744b0844d P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251025 07:52:40.569902 1166 tablet_bootstrap.cc:492] T 7c7a9d7a4fb3449bb57eafa83230a839 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:40.571602 2146 log.cc:1199] T abf94c32ff2945119b710905c075d63c P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/abf94c32ff2945119b710905c075d63c
I20251025 07:52:40.574508 1166 tablet_bootstrap.cc:654] T 7c7a9d7a4fb3449bb57eafa83230a839 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:40.589051 2146 ts_tablet_manager.cc:1950] T abf94c32ff2945119b710905c075d63c P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:40.589480 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 85) took 2142 ms. Trace:
I20251025 07:52:40.589653 2146 rpcz_store.cc:276] 1025 07:52:38.446730 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.446787 (+ 57us) service_pool.cc:225] Handling call
1025 07:52:40.492625 (+2045838us) ts_tablet_manager.cc:1151] Deleting tablet abf94c32ff2945119b710905c075d63c
1025 07:52:40.492634 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:40.527022 (+ 34388us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.542393 (+ 15371us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.589456 (+ 47063us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2045687}
I20251025 07:52:40.590327 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet abf94c32ff2945119b710905c075d63c (table default.Table_4b63840ca3ac4296b23104bf46030bb4 [id=0010b5f16dd14d64b8c93d941803cc75]) successfully deleted
I20251025 07:52:40.590850 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 86) took 2143 ms. Trace:
I20251025 07:52:40.590992 2146 rpcz_store.cc:276] 1025 07:52:38.447135 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.447171 (+ 36us) service_pool.cc:225] Handling call
1025 07:52:40.590694 (+2143523us) ts_tablet_manager.cc:1151] Deleting tablet 1912e59548874e0e8e989050ec0064c5
1025 07:52:40.590709 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:40.590828 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2143386}
I20251025 07:52:40.591259 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 74) took 1898 ms. Trace:
I20251025 07:52:40.591481 2169 rpcz_store.cc:276] 1025 07:52:38.692951 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.693045 (+ 94us) service_pool.cc:225] Handling call
1025 07:52:40.145259 (+1452214us) ts_tablet_manager.cc:1151] Deleting tablet 0b21b7da1e2f47829c3f6cdaee689e8a
1025 07:52:40.145271 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:40.153270 (+ 7999us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.387611 (+234341us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.591150 (+203539us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1442232}
W20251025 07:52:40.592187 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 1912e59548874e0e8e989050ec0064c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1912e59548874e0e8e989050ec0064c5 already in progress: creating tablet
I20251025 07:52:40.592290 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 0b21b7da1e2f47829c3f6cdaee689e8a (table default.Table_e61ae9065c604ad8b93af8146748cec9 [id=6d26779bac7243f19d534277dd1c986c]) successfully deleted
I20251025 07:52:40.592622 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 87) took 2144 ms. Trace:
I20251025 07:52:40.592793 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 75) took 1889 ms. Trace:
I20251025 07:52:40.593004 2169 rpcz_store.cc:276] 1025 07:52:38.704260 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.704320 (+ 60us) service_pool.cc:225] Handling call
1025 07:52:40.592590 (+1888270us) ts_tablet_manager.cc:1151] Deleting tablet fd265cdbf6a8440393a2b5d8240d1872
1025 07:52:40.592602 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:40.592723 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1887927}
W20251025 07:52:40.593521 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet fd265cdbf6a8440393a2b5d8240d1872 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd265cdbf6a8440393a2b5d8240d1872 already in progress: creating tablet
I20251025 07:52:40.593956 2169 tablet_replica.cc:333] T 887e7c7a5d0e4aaf88f26246306df19c P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:40.594503 2169 raft_consensus.cc:2243] T 887e7c7a5d0e4aaf88f26246306df19c P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:40.598177 2146 rpcz_store.cc:276] 1025 07:52:38.448282 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.448346 (+ 64us) service_pool.cc:225] Handling call
1025 07:52:40.592491 (+2144145us) ts_tablet_manager.cc:1151] Deleting tablet 2e098c9c4613459295b6043048cba587
1025 07:52:40.592504 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:40.592602 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2143953}
W20251025 07:52:40.598730 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 2e098c9c4613459295b6043048cba587 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e098c9c4613459295b6043048cba587 already in progress: creating tablet
I20251025 07:52:40.599179 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 88) took 2150 ms. Trace:
I20251025 07:52:40.599285 2146 rpcz_store.cc:276] 1025 07:52:38.448710 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.448923 (+ 213us) service_pool.cc:225] Handling call
1025 07:52:40.599050 (+2150127us) ts_tablet_manager.cc:1151] Deleting tablet 6dcab53a020847a1aa27930265ba0205
1025 07:52:40.599061 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:40.599159 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2149966}
W20251025 07:52:40.599772 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 6dcab53a020847a1aa27930265ba0205 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6dcab53a020847a1aa27930265ba0205 already in progress: creating tablet
I20251025 07:52:40.600111 2146 tablet_replica.cc:333] T 4f25c9caa05b4da4bd56a8a21fde423c P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:40.600596 2146 raft_consensus.cc:2243] T 4f25c9caa05b4da4bd56a8a21fde423c P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:40.600816 2146 raft_consensus.cc:2272] T 4f25c9caa05b4da4bd56a8a21fde423c P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:40.600795 710 consensus_peers.cc:597] T 8c76d1b509f04ab1a10b83cff675c11e P 23eeffd3de254ce18db5febfde971c6b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c76d1b509f04ab1a10b83cff675c11e. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:40.601387 2146 ts_tablet_manager.cc:1916] T 4f25c9caa05b4da4bd56a8a21fde423c P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:40.602530 1671 consensus_queue.cc:1048] T 74e2d78c024849b0af5c0bde1f48c36f P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:40.607088 1659 raft_consensus.cc:493] T a4e4b99be0434baaa75d988281797846 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:40.607247 1659 raft_consensus.cc:515] T a4e4b99be0434baaa75d988281797846 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.607872 1659 leader_election.cc:290] T a4e4b99be0434baaa75d988281797846 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:40.608716 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e4b99be0434baaa75d988281797846" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:40.609004 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e4b99be0434baaa75d988281797846" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:40.609442 710 leader_election.cc:343] T a4e4b99be0434baaa75d988281797846 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:40.610416 712 leader_election.cc:343] T a4e4b99be0434baaa75d988281797846 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:40.610550 712 leader_election.cc:304] T a4e4b99be0434baaa75d988281797846 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:40.611213 1659 raft_consensus.cc:2749] T a4e4b99be0434baaa75d988281797846 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:40.617938 1166 tablet_bootstrap.cc:492] T 7c7a9d7a4fb3449bb57eafa83230a839 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:40.618204 1166 ts_tablet_manager.cc:1403] T 7c7a9d7a4fb3449bb57eafa83230a839 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.049s user 0.006s sys 0.008s
I20251025 07:52:40.620464 1166 raft_consensus.cc:359] T 7c7a9d7a4fb3449bb57eafa83230a839 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.620765 1166 raft_consensus.cc:385] T 7c7a9d7a4fb3449bb57eafa83230a839 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:40.620862 1166 raft_consensus.cc:740] T 7c7a9d7a4fb3449bb57eafa83230a839 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:40.621368 1166 consensus_queue.cc:260] T 7c7a9d7a4fb3449bb57eafa83230a839 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.622296 1166 ts_tablet_manager.cc:1434] T 7c7a9d7a4fb3449bb57eafa83230a839 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251025 07:52:40.623014 1166 tablet_bootstrap.cc:492] T 80fe3255ed614270bd8352a2853fbc43 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:40.624572 2146 ts_tablet_manager.cc:1929] T 4f25c9caa05b4da4bd56a8a21fde423c P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:40.628501 1166 tablet_bootstrap.cc:654] T 80fe3255ed614270bd8352a2853fbc43 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:40.648298 2146 log.cc:1199] T 4f25c9caa05b4da4bd56a8a21fde423c P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/4f25c9caa05b4da4bd56a8a21fde423c
I20251025 07:52:40.648959 2146 ts_tablet_manager.cc:1950] T 4f25c9caa05b4da4bd56a8a21fde423c P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:40.649775 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 89) took 2200 ms. Trace:
I20251025 07:52:40.650259 2146 rpcz_store.cc:276] 1025 07:52:38.448809 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.449165 (+ 356us) service_pool.cc:225] Handling call
1025 07:52:40.600058 (+2150893us) ts_tablet_manager.cc:1151] Deleting tablet 4f25c9caa05b4da4bd56a8a21fde423c
1025 07:52:40.600068 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:40.616056 (+ 15988us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.624467 (+ 8411us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.649762 (+ 25295us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2150743}
I20251025 07:52:40.650949 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 90) took 2202 ms. Trace:
I20251025 07:52:40.651083 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 4f25c9caa05b4da4bd56a8a21fde423c (table default.Table_60bfcc345b8649e9bc73fe3d077f46c7 [id=ed06dbbb96a24ab39619f4186ba93191]) successfully deleted
I20251025 07:52:40.651360 2146 rpcz_store.cc:276] 1025 07:52:38.448889 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.449983 (+ 1094us) service_pool.cc:225] Handling call
1025 07:52:40.650818 (+2200835us) ts_tablet_manager.cc:1151] Deleting tablet 1b5ac7e845a24b2eb5b2429d47361232
1025 07:52:40.650827 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:40.650934 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2200646}
I20251025 07:52:40.651849 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 92) took 2199 ms. Trace:
W20251025 07:52:40.652058 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 1b5ac7e845a24b2eb5b2429d47361232 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b5ac7e845a24b2eb5b2429d47361232 already in progress: creating tablet
I20251025 07:52:40.652194 2146 rpcz_store.cc:276] 1025 07:52:38.451956 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.452108 (+ 152us) service_pool.cc:225] Handling call
1025 07:52:40.651759 (+2199651us) ts_tablet_manager.cc:1151] Deleting tablet a7ab3150a35b4caca9119a1374d3f2f7
1025 07:52:40.651766 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:40.651835 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2199509}
I20251025 07:52:40.652618 2146 tablet_replica.cc:333] T cb824bd780524832b2b60f5618a39da5 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
W20251025 07:52:40.652772 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet a7ab3150a35b4caca9119a1374d3f2f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7ab3150a35b4caca9119a1374d3f2f7 already in progress: creating tablet
I20251025 07:52:40.652992 2146 raft_consensus.cc:2243] T cb824bd780524832b2b60f5618a39da5 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:40.653162 2146 raft_consensus.cc:2272] T cb824bd780524832b2b60f5618a39da5 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:40.653730 2146 ts_tablet_manager.cc:1916] T cb824bd780524832b2b60f5618a39da5 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:40.655471 1169 tablet_bootstrap.cc:492] T ebf3a2a926754305b5bd95fcebb25d29 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:40.655699 1169 ts_tablet_manager.cc:1403] T ebf3a2a926754305b5bd95fcebb25d29 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.488s user 0.008s sys 0.006s
I20251025 07:52:40.656857 1169 raft_consensus.cc:359] T ebf3a2a926754305b5bd95fcebb25d29 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.657150 1169 raft_consensus.cc:385] T ebf3a2a926754305b5bd95fcebb25d29 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:40.657258 1169 raft_consensus.cc:740] T ebf3a2a926754305b5bd95fcebb25d29 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:40.657552 1169 consensus_queue.cc:260] T ebf3a2a926754305b5bd95fcebb25d29 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.658389 1169 ts_tablet_manager.cc:1434] T ebf3a2a926754305b5bd95fcebb25d29 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251025 07:52:40.658761 1169 tablet_bootstrap.cc:492] T b94634ac5077439fadfbbce569f60e8b P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:40.669236 2146 ts_tablet_manager.cc:1929] T cb824bd780524832b2b60f5618a39da5 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:40.672991 1169 tablet_bootstrap.cc:654] T b94634ac5077439fadfbbce569f60e8b P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:40.678387 2146 log.cc:1199] T cb824bd780524832b2b60f5618a39da5 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/cb824bd780524832b2b60f5618a39da5
I20251025 07:52:40.678990 2146 ts_tablet_manager.cc:1950] T cb824bd780524832b2b60f5618a39da5 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:40.677999 2361 tablet.cc:1722] T 2c50ec95e48e49b59a76fac4fc862d3d P 3aea2869ff814cc7b142012faa274382: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251025 07:52:40.680173 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 93) took 2228 ms. Trace:
I20251025 07:52:40.682847 2362 tablet.cc:1722] T 2c50ec95e48e49b59a76fac4fc862d3d P db42be7ad3c741a0aa427f27e245993b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251025 07:52:40.683544 2146 rpcz_store.cc:276] 1025 07:52:38.452073 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.453626 (+ 1553us) service_pool.cc:225] Handling call
1025 07:52:40.652577 (+2198951us) ts_tablet_manager.cc:1151] Deleting tablet cb824bd780524832b2b60f5618a39da5
1025 07:52:40.652585 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:40.656990 (+ 4405us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.669119 (+ 12129us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.680152 (+ 11033us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2198787}
I20251025 07:52:40.684240 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet cb824bd780524832b2b60f5618a39da5 (table default.Table_c1743cb60e4d45939f387235c1726ef3 [id=18e29b94f8a6443fb8e6a876119fd9a3]) successfully deleted
I20251025 07:52:40.684703 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 94) took 2230 ms. Trace:
I20251025 07:52:40.685086 2146 rpcz_store.cc:276] 1025 07:52:38.453989 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.454313 (+ 324us) service_pool.cc:225] Handling call
1025 07:52:40.684576 (+2230263us) ts_tablet_manager.cc:1151] Deleting tablet ee1c8ea694284d80a30f9c05be4cbf49
1025 07:52:40.684588 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:40.684688 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2230112}
W20251025 07:52:40.685700 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet ee1c8ea694284d80a30f9c05be4cbf49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee1c8ea694284d80a30f9c05be4cbf49 already in progress: creating tablet
I20251025 07:52:40.686069 2146 tablet_replica.cc:333] T fab67b56eaa44b54886c17181d8c52b2 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:40.686848 2146 raft_consensus.cc:2243] T fab67b56eaa44b54886c17181d8c52b2 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251025 07:52:40.687990 982 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:46231, user_credentials={real_user=slave}} blocked reactor thread for 191541us
I20251025 07:52:40.687348 2146 raft_consensus.cc:2272] T fab67b56eaa44b54886c17181d8c52b2 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:40.690958 2146 ts_tablet_manager.cc:1916] T fab67b56eaa44b54886c17181d8c52b2 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:40.707552 1659 raft_consensus.cc:493] T 6dcab53a020847a1aa27930265ba0205 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:40.707769 2146 ts_tablet_manager.cc:1929] T fab67b56eaa44b54886c17181d8c52b2 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:40.707722 1659 raft_consensus.cc:515] T 6dcab53a020847a1aa27930265ba0205 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:40.707859 2146 log.cc:1199] T fab67b56eaa44b54886c17181d8c52b2 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/fab67b56eaa44b54886c17181d8c52b2
I20251025 07:52:40.708192 2146 ts_tablet_manager.cc:1950] T fab67b56eaa44b54886c17181d8c52b2 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:40.708460 1659 leader_election.cc:290] T 6dcab53a020847a1aa27930265ba0205 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:40.708495 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 95) took 2253 ms. Trace:
I20251025 07:52:40.708563 2146 rpcz_store.cc:276] 1025 07:52:38.454705 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.455098 (+ 393us) service_pool.cc:225] Handling call
1025 07:52:40.686024 (+2230926us) ts_tablet_manager.cc:1151] Deleting tablet fab67b56eaa44b54886c17181d8c52b2
1025 07:52:40.686033 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:40.704637 (+ 18604us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.707681 (+ 3044us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.708476 (+ 795us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2230763}
I20251025 07:52:40.716907 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dcab53a020847a1aa27930265ba0205" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:40.718173 712 leader_election.cc:343] T 6dcab53a020847a1aa27930265ba0205 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:40.709286 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dcab53a020847a1aa27930265ba0205" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:40.727202 710 leader_election.cc:343] T 6dcab53a020847a1aa27930265ba0205 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:40.727350 710 leader_election.cc:304] T 6dcab53a020847a1aa27930265ba0205 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:40.727715 1659 raft_consensus.cc:2749] T 6dcab53a020847a1aa27930265ba0205 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:40.728099 2146 tablet_replica.cc:333] T 27651312cc2e42a8bfcd245f5d301679 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:40.728304 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet fab67b56eaa44b54886c17181d8c52b2 (table default.Table_6ca844d1e78c4f19939985aec32a8afd [id=1e933863244e4792b4c1b4d6d5c674b7]) successfully deleted
I20251025 07:52:40.728551 2146 raft_consensus.cc:2243] T 27651312cc2e42a8bfcd245f5d301679 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:40.728726 2146 raft_consensus.cc:2272] T 27651312cc2e42a8bfcd245f5d301679 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:40.729103 1167 tablet_bootstrap.cc:654] T ab42909afc524f2fa3c1041495d13a59 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:40.729277 2146 ts_tablet_manager.cc:1916] T 27651312cc2e42a8bfcd245f5d301679 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:40.734882 2361 tablet.cc:1722] T b17d2a9ad6be49d29f2413666fba7e69 P 3aea2869ff814cc7b142012faa274382: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251025 07:52:40.740347 2362 tablet.cc:1722] T b17d2a9ad6be49d29f2413666fba7e69 P db42be7ad3c741a0aa427f27e245993b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251025 07:52:40.745237 2166 ts_tablet_manager.cc:1929] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:40.752657 2146 ts_tablet_manager.cc:1929] T 27651312cc2e42a8bfcd245f5d301679 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:40.753221 1838 raft_consensus.cc:493] T b3a0af893a8947848e12093d0a7f0229 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:40.754580 2146 log.cc:1199] T 27651312cc2e42a8bfcd245f5d301679 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/27651312cc2e42a8bfcd245f5d301679
I20251025 07:52:40.754616 1838 raft_consensus.cc:515] T b3a0af893a8947848e12093d0a7f0229 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:40.754958 2146 ts_tablet_manager.cc:1950] T 27651312cc2e42a8bfcd245f5d301679 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:40.755288 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 96) took 2299 ms. Trace:
I20251025 07:52:40.755311 1838 leader_election.cc:290] T b3a0af893a8947848e12093d0a7f0229 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:40.755357 2146 rpcz_store.cc:276] 1025 07:52:38.455467 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.455803 (+ 336us) service_pool.cc:225] Handling call
1025 07:52:40.728046 (+2272243us) ts_tablet_manager.cc:1151] Deleting tablet 27651312cc2e42a8bfcd245f5d301679
1025 07:52:40.728057 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:40.732478 (+ 4421us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.752539 (+ 20061us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.755271 (+ 2732us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2272100}
I20251025 07:52:40.757781 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3a0af893a8947848e12093d0a7f0229" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:40.756506 847 leader_election.cc:343] T b3a0af893a8947848e12093d0a7f0229 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: b3a0af893a8947848e12093d0a7f0229
W20251025 07:52:40.762838 850 leader_election.cc:343] T b3a0af893a8947848e12093d0a7f0229 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:40.763211 850 leader_election.cc:304] T b3a0af893a8947848e12093d0a7f0229 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:40.763717 2314 raft_consensus.cc:2749] T b3a0af893a8947848e12093d0a7f0229 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:40.764254 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 27651312cc2e42a8bfcd245f5d301679 (table default.Table_dac528b081844b7c86027169c6234c27 [id=b0caa78ee8c84aa8bea4818cf9feb70c]) successfully deleted
I20251025 07:52:40.764665 2146 tablet_replica.cc:333] T b947ecc352e54f4caa73c6ca97372256 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:40.769802 2146 raft_consensus.cc:2243] T b947ecc352e54f4caa73c6ca97372256 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:40.770330 2146 raft_consensus.cc:2272] T b947ecc352e54f4caa73c6ca97372256 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:40.771075 2146 ts_tablet_manager.cc:1916] T b947ecc352e54f4caa73c6ca97372256 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:40.779011 1095 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
I20251025 07:52:40.789683 2314 raft_consensus.cc:493] T e9f7cd765d614d59a7c190b744b0844d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:40.789861 2314 raft_consensus.cc:515] T e9f7cd765d614d59a7c190b744b0844d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.790524 2314 leader_election.cc:290] T e9f7cd765d614d59a7c190b744b0844d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:40.790951 1166 tablet_bootstrap.cc:492] T 80fe3255ed614270bd8352a2853fbc43 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:40.791080 1166 ts_tablet_manager.cc:1403] T 80fe3255ed614270bd8352a2853fbc43 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.168s user 0.024s sys 0.019s
I20251025 07:52:40.791361 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9f7cd765d614d59a7c190b744b0844d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:40.791841 850 leader_election.cc:343] T e9f7cd765d614d59a7c190b744b0844d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:40.795269 2379 tablet.cc:1722] T 8f2c8582fed34c848d9903b8ad8b4057 P db42be7ad3c741a0aa427f27e245993b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 07:52:40.797865 1166 raft_consensus.cc:359] T 80fe3255ed614270bd8352a2853fbc43 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.798043 1166 raft_consensus.cc:385] T 80fe3255ed614270bd8352a2853fbc43 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:40.798090 1166 raft_consensus.cc:740] T 80fe3255ed614270bd8352a2853fbc43 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:40.798290 1166 consensus_queue.cc:260] T 80fe3255ed614270bd8352a2853fbc43 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.798710 1166 ts_tablet_manager.cc:1434] T 80fe3255ed614270bd8352a2853fbc43 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251025 07:52:40.798960 1166 tablet_bootstrap.cc:492] T f97acc9a638b4d5a968dcc3412a384bc P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:40.800496 1166 tablet_bootstrap.cc:654] T f97acc9a638b4d5a968dcc3412a384bc P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:40.805341 2361 tablet.cc:1722] T 8f2c8582fed34c848d9903b8ad8b4057 P 3aea2869ff814cc7b142012faa274382: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 07:52:40.808601 902 raft_consensus.cc:3060] T fe5033e5ff404a5ebc6c8443d7548327 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:40.813683 2166 log.cc:1199] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/7fbae8be73bf4e18b5a5c4ec4cdebc12
I20251025 07:52:40.758749 2370 tablet.cc:1722] T 2c50ec95e48e49b59a76fac4fc862d3d P 23eeffd3de254ce18db5febfde971c6b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251025 07:52:40.816009 822 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
I20251025 07:52:40.816192 2385 tablet.cc:1722] T 8f2c8582fed34c848d9903b8ad8b4057 P 23eeffd3de254ce18db5febfde971c6b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 07:52:40.791796 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9f7cd765d614d59a7c190b744b0844d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:40.818257 772 raft_consensus.cc:2468] T e9f7cd765d614d59a7c190b744b0844d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:40.818785 847 leader_election.cc:304] T e9f7cd765d614d59a7c190b744b0844d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:40.818924 2146 ts_tablet_manager.cc:1929] T b947ecc352e54f4caa73c6ca97372256 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:40.819028 2146 log.cc:1199] T b947ecc352e54f4caa73c6ca97372256 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/b947ecc352e54f4caa73c6ca97372256
I20251025 07:52:40.819260 2105 raft_consensus.cc:2804] T e9f7cd765d614d59a7c190b744b0844d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:40.819314 2146 ts_tablet_manager.cc:1950] T b947ecc352e54f4caa73c6ca97372256 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:40.819422 2348 consensus_queue.cc:1048] T fe5033e5ff404a5ebc6c8443d7548327 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:40.819360 2105 raft_consensus.cc:493] T e9f7cd765d614d59a7c190b744b0844d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:40.822296 2105 raft_consensus.cc:3060] T e9f7cd765d614d59a7c190b744b0844d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:40.823243 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f]). Expected version 1 got 0
I20251025 07:52:40.824015 2166 ts_tablet_manager.cc:1950] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:40.825004 2105 raft_consensus.cc:515] T e9f7cd765d614d59a7c190b744b0844d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.825726 1612 consensus_queue.cc:1048] T 6a58931e7cd7454fb583f1c70cd2ad15 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:40.825976 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9f7cd765d614d59a7c190b744b0844d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:40.826177 772 raft_consensus.cc:3060] T e9f7cd765d614d59a7c190b744b0844d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:40.826480 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9f7cd765d614d59a7c190b744b0844d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:40.826905 850 leader_election.cc:343] T e9f7cd765d614d59a7c190b744b0844d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:40.827107 2105 leader_election.cc:290] T e9f7cd765d614d59a7c190b744b0844d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:40.827497 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 61) took 2443 ms. Trace:
I20251025 07:52:40.827595 2146 rpcz_store.cc:276] 1025 07:52:38.420255 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.459086 (+ 38831us) service_pool.cc:225] Handling call
1025 07:52:40.764609 (+2305523us) ts_tablet_manager.cc:1151] Deleting tablet b947ecc352e54f4caa73c6ca97372256
1025 07:52:40.764622 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:40.776117 (+ 11495us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.818811 (+ 42694us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.827484 (+ 8673us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2305303}
I20251025 07:52:40.828841 772 raft_consensus.cc:2468] T e9f7cd765d614d59a7c190b744b0844d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:40.829535 847 leader_election.cc:304] T e9f7cd765d614d59a7c190b744b0844d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:40.829684 1166 tablet_bootstrap.cc:492] T f97acc9a638b4d5a968dcc3412a384bc P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:40.830020 2362 tablet.cc:1722] T ef1854d35a3047e3a9fbe0166ab1295a P db42be7ad3c741a0aa427f27e245993b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251025 07:52:40.830194 2314 raft_consensus.cc:2804] T e9f7cd765d614d59a7c190b744b0844d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:40.830303 2314 raft_consensus.cc:697] T e9f7cd765d614d59a7c190b744b0844d P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:40.830531 2314 consensus_queue.cc:237] T e9f7cd765d614d59a7c190b744b0844d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.830457 1166 ts_tablet_manager.cc:1403] T f97acc9a638b4d5a968dcc3412a384bc P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.001s
I20251025 07:52:40.835340 1166 raft_consensus.cc:359] T f97acc9a638b4d5a968dcc3412a384bc P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.836530 1166 raft_consensus.cc:385] T f97acc9a638b4d5a968dcc3412a384bc P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:40.836932 1166 raft_consensus.cc:740] T f97acc9a638b4d5a968dcc3412a384bc P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:40.838454 1612 consensus_queue.cc:1048] T 92aa605fabbc4257a6a204c087e7cc7f P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:40.838358 1166 consensus_queue.cc:260] T f97acc9a638b4d5a968dcc3412a384bc P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.839054 2146 tablet_replica.cc:333] T 896bd7870d87408e8adfdb7a78998111 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:40.839578 2146 raft_consensus.cc:2243] T 896bd7870d87408e8adfdb7a78998111 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:40.840184 2146 raft_consensus.cc:2272] T 896bd7870d87408e8adfdb7a78998111 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:40.841063 2146 ts_tablet_manager.cc:1916] T 896bd7870d87408e8adfdb7a78998111 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:40.844990 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 65) took 2327 ms. Trace:
I20251025 07:52:40.845470 2370 tablet.cc:1722] T b17d2a9ad6be49d29f2413666fba7e69 P 23eeffd3de254ce18db5febfde971c6b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251025 07:52:40.841399 1166 ts_tablet_manager.cc:1434] T f97acc9a638b4d5a968dcc3412a384bc P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.011s user 0.002s sys 0.001s
I20251025 07:52:40.847086 1166 tablet_bootstrap.cc:492] T 0325d462441349ef81841f1b0cab2a20 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:40.845155 2166 rpcz_store.cc:276] 1025 07:52:38.517243 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.517280 (+ 37us) service_pool.cc:225] Handling call
1025 07:52:40.279275 (+1761995us) ts_tablet_manager.cc:1151] Deleting tablet 7fbae8be73bf4e18b5a5c4ec4cdebc12
1025 07:52:40.279285 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:40.741150 (+461865us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.745104 (+ 3954us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.844939 (+ 99835us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1761873}
I20251025 07:52:40.851135 2166 tablet_replica.cc:333] T 551d07a9242740dcb9366bb2d707a4a8 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:40.851430 2166 raft_consensus.cc:2243] T 551d07a9242740dcb9366bb2d707a4a8 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:40.851652 2166 raft_consensus.cc:2272] T 551d07a9242740dcb9366bb2d707a4a8 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:40.852523 2166 ts_tablet_manager.cc:1916] T 551d07a9242740dcb9366bb2d707a4a8 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:40.854159 1166 tablet_bootstrap.cc:654] T 0325d462441349ef81841f1b0cab2a20 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:40.857791 2146 ts_tablet_manager.cc:1929] T 896bd7870d87408e8adfdb7a78998111 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251025 07:52:40.858672 982 consensus_peers.cc:597] T 3abfd52327a04ba88625f391a8658fbb P db42be7ad3c741a0aa427f27e245993b -> Peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Couldn't send request to peer 23eeffd3de254ce18db5febfde971c6b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3abfd52327a04ba88625f391a8658fbb. This is attempt 1: this message will repeat every 5th retry.
W20251025 07:52:40.858985 982 consensus_peers.cc:597] T 92aa605fabbc4257a6a204c087e7cc7f P db42be7ad3c741a0aa427f27e245993b -> Peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Couldn't send request to peer 23eeffd3de254ce18db5febfde971c6b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 92aa605fabbc4257a6a204c087e7cc7f. This is attempt 1: this message will repeat every 5th retry.
W20251025 07:52:40.859609 982 leader_election.cc:343] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: b947ecc352e54f4caa73c6ca97372256
I20251025 07:52:40.861018 1612 consensus_queue.cc:1048] T 74e2d78c024849b0af5c0bde1f48c36f P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:40.869258 1659 raft_consensus.cc:493] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3aea2869ff814cc7b142012faa274382)
I20251025 07:52:40.869451 1659 raft_consensus.cc:515] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:40.870008 1659 leader_election.cc:290] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:40.829244 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet b947ecc352e54f4caa73c6ca97372256 (table default.Table_503b86ee2ab545afa9829bebcc9ea11d [id=395c91146476423dad0455a12eb5a8af]) successfully deleted
W20251025 07:52:40.871083 710 leader_election.cc:343] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: db99a1e68daf4e948a3d66ad276cc5f9
I20251025 07:52:40.850915 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 7fbae8be73bf4e18b5a5c4ec4cdebc12 (table default.Table_6c5a24747ba14352b0285a41ec2985a4 [id=28bbaad474164a8c8c97a76bb8081791]) successfully deleted
W20251025 07:52:40.871757 712 leader_election.cc:343] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: db99a1e68daf4e948a3d66ad276cc5f9
I20251025 07:52:40.871865 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 4c155aa7fa5b4d68bbab1918b8432372 (table default.Table_6c5a24747ba14352b0285a41ec2985a4 [id=28bbaad474164a8c8c97a76bb8081791]). Expected version 2 got 0
I20251025 07:52:40.871883 712 leader_election.cc:304] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:40.872349 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7]). Expected version 1 got 0
I20251025 07:52:40.872476 1659 raft_consensus.cc:2749] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 07:52:40.872596 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 04d27d54fdb7483b869408af65ff85c9 (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]). Expected version 2 got 0
I20251025 07:52:40.872927 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet cb829649f73d4d3f8862d268c953bd86 (table default.Table_ba4b8cc7b930482690b7e192e0edff12 [id=7944225d3ce7439b881bfb9289d9f040]). Expected version 1 got 0
I20251025 07:52:40.873165 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 872cb7512d104827b1f1f186d33a4a44 (table default.Table_d5c65dce8b1b4214ba519d964012bfc0 [id=d4be274b36a845a99a4b96b8846248fd]). Expected version 1 got 0
I20251025 07:52:40.874220 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet aa9d9ab8073945d28d52fb02213f201a (table default.Table_4791672658f043f6b97b9320a531ebb5 [id=dfd20cf9c3e541c4abe23d59e0aa7f2a]). Expected version 1 got 0
I20251025 07:52:40.874531 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]). Expected version 1 got 0
I20251025 07:52:40.874784 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet ef9a2c9f2ccf432ab835396020872208 (table default.Table_2b9b772d4c234ac286bef3658482e490 [id=409fab60f9d744798e6f92b868f0f7c2]). Expected version 1 got 0
I20251025 07:52:40.875077 2348 raft_consensus.cc:493] T aa9d9ab8073945d28d52fb02213f201a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:40.875263 2348 raft_consensus.cc:515] T aa9d9ab8073945d28d52fb02213f201a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.876114 2348 leader_election.cc:290] T aa9d9ab8073945d28d52fb02213f201a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:40.859465 2105 raft_consensus.cc:493] T f97acc9a638b4d5a968dcc3412a384bc P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:40.877420 2105 raft_consensus.cc:515] T f97acc9a638b4d5a968dcc3412a384bc P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.878304 905 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa9d9ab8073945d28d52fb02213f201a" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:40.878525 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f97acc9a638b4d5a968dcc3412a384bc" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:40.878748 772 raft_consensus.cc:2468] T f97acc9a638b4d5a968dcc3412a384bc P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:40.879318 847 leader_election.cc:304] T f97acc9a638b4d5a968dcc3412a384bc P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:40.879693 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f97acc9a638b4d5a968dcc3412a384bc" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:40.880242 850 leader_election.cc:343] T f97acc9a638b4d5a968dcc3412a384bc P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:40.880436 2105 leader_election.cc:290] T f97acc9a638b4d5a968dcc3412a384bc P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:40.880439 2362 tablet.cc:1722] T 509c94f2a422493ea4390ecee576db72 P db42be7ad3c741a0aa427f27e245993b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 07:52:40.880726 2105 raft_consensus.cc:2804] T f97acc9a638b4d5a968dcc3412a384bc P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:40.880834 2105 raft_consensus.cc:493] T f97acc9a638b4d5a968dcc3412a384bc P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:40.880951 2105 raft_consensus.cc:3060] T f97acc9a638b4d5a968dcc3412a384bc P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:40.883298 2378 tablet.cc:1722] T 509c94f2a422493ea4390ecee576db72 P 3aea2869ff814cc7b142012faa274382: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 07:52:40.883641 1166 tablet_bootstrap.cc:492] T 0325d462441349ef81841f1b0cab2a20 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:40.883823 2105 raft_consensus.cc:515] T f97acc9a638b4d5a968dcc3412a384bc P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.884795 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f97acc9a638b4d5a968dcc3412a384bc" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:40.884899 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f97acc9a638b4d5a968dcc3412a384bc" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:40.884976 772 raft_consensus.cc:3060] T f97acc9a638b4d5a968dcc3412a384bc P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:40.885394 850 leader_election.cc:343] T f97acc9a638b4d5a968dcc3412a384bc P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:40.885569 2105 leader_election.cc:290] T f97acc9a638b4d5a968dcc3412a384bc P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:40.883935 1166 ts_tablet_manager.cc:1403] T 0325d462441349ef81841f1b0cab2a20 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.005s
I20251025 07:52:40.886420 1166 raft_consensus.cc:359] T 0325d462441349ef81841f1b0cab2a20 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:40.886600 1166 raft_consensus.cc:385] T 0325d462441349ef81841f1b0cab2a20 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:40.886665 1166 raft_consensus.cc:740] T 0325d462441349ef81841f1b0cab2a20 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:40.887301 1166 consensus_queue.cc:260] T 0325d462441349ef81841f1b0cab2a20 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:40.887974 772 raft_consensus.cc:2468] T f97acc9a638b4d5a968dcc3412a384bc P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:40.888343 847 leader_election.cc:304] T f97acc9a638b4d5a968dcc3412a384bc P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:40.889032 2105 raft_consensus.cc:2804] T f97acc9a638b4d5a968dcc3412a384bc P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:40.889263 2105 raft_consensus.cc:697] T f97acc9a638b4d5a968dcc3412a384bc P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:40.889516 2105 consensus_queue.cc:237] T f97acc9a638b4d5a968dcc3412a384bc P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.891342 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]). Expected version 1 got 0
I20251025 07:52:40.892578 2314 raft_consensus.cc:493] T ebf3a2a926754305b5bd95fcebb25d29 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:40.893849 2314 raft_consensus.cc:515] T ebf3a2a926754305b5bd95fcebb25d29 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.894888 2314 leader_election.cc:290] T ebf3a2a926754305b5bd95fcebb25d29 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:40.895380 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_a6c869949b744bc98efb735b0e73e7aa [id=bf77b5ff6f1048dd996d948437f9ff5e]). Expected version 2 got 0
I20251025 07:52:40.895923 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet ef1854d35a3047e3a9fbe0166ab1295a (table default.Table_088aad2a46b041bfa29ed8e6929760b1 [id=834669c54dae449790818684fd8659bc]). Expected version 2 got 0
I20251025 07:52:40.896301 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet bc2d16b33ca8426abbd7bd38339c9f1d (table default.Table_4ea985df4c5b415c9570c64506a98cc4 [id=98d2b7797e33485e8872054aa2eb8ec9]). Expected version 1 got 0
I20251025 07:52:40.898546 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebf3a2a926754305b5bd95fcebb25d29" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:40.898779 1049 raft_consensus.cc:2468] T ebf3a2a926754305b5bd95fcebb25d29 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
W20251025 07:52:40.899262 983 consensus_peers.cc:597] T 896bd7870d87408e8adfdb7a78998111 P db42be7ad3c741a0aa427f27e245993b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:40.899753 850 leader_election.cc:304] T ebf3a2a926754305b5bd95fcebb25d29 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:40.900046 2314 raft_consensus.cc:2804] T ebf3a2a926754305b5bd95fcebb25d29 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:40.900117 2314 raft_consensus.cc:493] T ebf3a2a926754305b5bd95fcebb25d29 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:40.900173 2314 raft_consensus.cc:3060] T ebf3a2a926754305b5bd95fcebb25d29 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:40.900570 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e]). Expected version 1 got 0
I20251025 07:52:40.901661 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet ab42909afc524f2fa3c1041495d13a59 (table default.Table_7eb8424d5192485da1a2d67bfdb2bbbf [id=40c29df99590408cbd5c583d4fc741eb]). Expected version 2 got 0
I20251025 07:52:40.903002 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 509c94f2a422493ea4390ecee576db72 (table default.Table_3f081b7bb4e1420c8121ea4115439f3d [id=696a1285fbfe4038a8b1b38708c8ea49]). Expected version 1 got 0
I20251025 07:52:40.903432 2314 raft_consensus.cc:515] T ebf3a2a926754305b5bd95fcebb25d29 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.904218 2314 leader_election.cc:290] T ebf3a2a926754305b5bd95fcebb25d29 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:40.904618 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebf3a2a926754305b5bd95fcebb25d29" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:40.904793 1044 raft_consensus.cc:3060] T ebf3a2a926754305b5bd95fcebb25d29 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:40.904927 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebf3a2a926754305b5bd95fcebb25d29" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:40.905084 772 raft_consensus.cc:3060] T ebf3a2a926754305b5bd95fcebb25d29 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:40.905458 1166 ts_tablet_manager.cc:1434] T 0325d462441349ef81841f1b0cab2a20 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.020s user 0.001s sys 0.000s
I20251025 07:52:40.905740 1166 tablet_bootstrap.cc:492] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:40.907410 1166 tablet_bootstrap.cc:654] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:40.907564 1044 raft_consensus.cc:2468] T ebf3a2a926754305b5bd95fcebb25d29 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:40.907836 772 raft_consensus.cc:2468] T ebf3a2a926754305b5bd95fcebb25d29 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:40.870321 2361 tablet.cc:1722] T ef1854d35a3047e3a9fbe0166ab1295a P 3aea2869ff814cc7b142012faa274382: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251025 07:52:40.908406 847 leader_election.cc:304] T ebf3a2a926754305b5bd95fcebb25d29 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:40.908603 1838 raft_consensus.cc:2804] T ebf3a2a926754305b5bd95fcebb25d29 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:40.908725 1838 raft_consensus.cc:697] T ebf3a2a926754305b5bd95fcebb25d29 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:40.908957 1838 consensus_queue.cc:237] T ebf3a2a926754305b5bd95fcebb25d29 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.913043 2314 raft_consensus.cc:493] T 88eb51447e0f4080baa49dd902deb24f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:40.913190 2314 raft_consensus.cc:515] T 88eb51447e0f4080baa49dd902deb24f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:40.914341 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88eb51447e0f4080baa49dd902deb24f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:40.914898 850 leader_election.cc:343] T 88eb51447e0f4080baa49dd902deb24f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:40.915072 2314 leader_election.cc:290] T 88eb51447e0f4080baa49dd902deb24f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:40.916326 1659 raft_consensus.cc:493] T 66f0d398e59a4175aa9fe90715124d1a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:40.916466 1659 raft_consensus.cc:515] T 66f0d398e59a4175aa9fe90715124d1a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:40.917127 1659 leader_election.cc:290] T 66f0d398e59a4175aa9fe90715124d1a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:40.918257 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66f0d398e59a4175aa9fe90715124d1a" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:40.918836 712 leader_election.cc:343] T 66f0d398e59a4175aa9fe90715124d1a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:40.919132 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66f0d398e59a4175aa9fe90715124d1a" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:40.919658 2146 log.cc:1199] T 896bd7870d87408e8adfdb7a78998111 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/896bd7870d87408e8adfdb7a78998111
W20251025 07:52:40.919724 710 leader_election.cc:343] T 66f0d398e59a4175aa9fe90715124d1a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:40.919833 710 leader_election.cc:304] T 66f0d398e59a4175aa9fe90715124d1a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:40.920281 2146 ts_tablet_manager.cc:1950] T 896bd7870d87408e8adfdb7a78998111 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:40.920256 1659 raft_consensus.cc:2749] T 66f0d398e59a4175aa9fe90715124d1a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:40.921172 1659 raft_consensus.cc:493] T 4c155aa7fa5b4d68bbab1918b8432372 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:40.921305 1659 raft_consensus.cc:515] T 4c155aa7fa5b4d68bbab1918b8432372 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.921869 1659 leader_election.cc:290] T 4c155aa7fa5b4d68bbab1918b8432372 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:40.922566 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 97) took 2466 ms. Trace:
I20251025 07:52:40.922693 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c155aa7fa5b4d68bbab1918b8432372" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:40.923143 712 leader_election.cc:343] T 4c155aa7fa5b4d68bbab1918b8432372 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:40.924105 2166 ts_tablet_manager.cc:1929] T 551d07a9242740dcb9366bb2d707a4a8 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:40.924536 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525]). Expected version 1 got 0
I20251025 07:52:40.925985 2166 log.cc:1199] T 551d07a9242740dcb9366bb2d707a4a8 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/551d07a9242740dcb9366bb2d707a4a8
I20251025 07:52:40.927196 1167 tablet_bootstrap.cc:492] T ab42909afc524f2fa3c1041495d13a59 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:40.927386 1167 ts_tablet_manager.cc:1403] T ab42909afc524f2fa3c1041495d13a59 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.442s user 0.006s sys 0.006s
I20251025 07:52:40.928134 1167 raft_consensus.cc:359] T ab42909afc524f2fa3c1041495d13a59 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:40.928325 1167 raft_consensus.cc:385] T ab42909afc524f2fa3c1041495d13a59 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:40.928411 1167 raft_consensus.cc:740] T ab42909afc524f2fa3c1041495d13a59 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:40.928632 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88eb51447e0f4080baa49dd902deb24f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:40.925776 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84]). Expected version 1 got 0
I20251025 07:52:40.929486 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 5dab439bc50d4e869548154e1d222ad6 (table default.Table_bd82d871f84943f4baa967f5c0ec74a3 [id=45325c188eac4d34bb016d01de091f87]). Expected version 2 got 0
I20251025 07:52:40.923393 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c155aa7fa5b4d68bbab1918b8432372" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:40.930716 2146 rpcz_store.cc:276] 1025 07:52:38.456685 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.459404 (+ 2719us) service_pool.cc:225] Handling call
1025 07:52:40.839004 (+2379600us) ts_tablet_manager.cc:1151] Deleting tablet 896bd7870d87408e8adfdb7a78998111
1025 07:52:40.839016 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:40.854850 (+ 15834us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.857702 (+ 2852us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.922549 (+ 64847us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2379422}
I20251025 07:52:40.931006 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_87d38832f85e4965bfab1a077990114f [id=ad812862698d421184ebf501ce79b7be]). Expected version 2 got 0
I20251025 07:52:40.931365 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 98) took 2471 ms. Trace:
I20251025 07:52:40.931454 2146 rpcz_store.cc:276] 1025 07:52:38.460092 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.460134 (+ 42us) service_pool.cc:225] Handling call
1025 07:52:40.931232 (+2471098us) ts_tablet_manager.cc:1151] Deleting tablet 95efd553493e4966bc48a01cac6e1e10
1025 07:52:40.931242 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:40.931349 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2470949}
I20251025 07:52:40.931260 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 896bd7870d87408e8adfdb7a78998111 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]) successfully deleted
I20251025 07:52:40.931855 2146 tablet_replica.cc:333] T 467324e2432f4caca148951329caa292 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
W20251025 07:52:40.931937 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 95efd553493e4966bc48a01cac6e1e10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95efd553493e4966bc48a01cac6e1e10 already in progress: creating tablet
I20251025 07:52:40.932055 2146 raft_consensus.cc:2243] T 467324e2432f4caca148951329caa292 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:40.932430 2146 raft_consensus.cc:2272] T 467324e2432f4caca148951329caa292 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:40.932924 982 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:46231, user_credentials={real_user=slave}} blocked reactor thread for 71947.3us
I20251025 07:52:40.932966 2146 ts_tablet_manager.cc:1916] T 467324e2432f4caca148951329caa292 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:40.933007 1166 tablet_bootstrap.cc:492] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:40.933163 1166 ts_tablet_manager.cc:1403] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20251025 07:52:40.928637 1167 consensus_queue.cc:260] T ab42909afc524f2fa3c1041495d13a59 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:40.933461 772 raft_consensus.cc:2468] T 88eb51447e0f4080baa49dd902deb24f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:40.933820 1167 ts_tablet_manager.cc:1434] T ab42909afc524f2fa3c1041495d13a59 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251025 07:52:40.934274 1167 tablet_bootstrap.cc:492] T bc2d16b33ca8426abbd7bd38339c9f1d P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:40.934338 1166 raft_consensus.cc:359] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:40.934587 1166 raft_consensus.cc:385] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:40.934711 1166 raft_consensus.cc:740] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:40.935106 1166 consensus_queue.cc:260] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:40.935716 847 leader_election.cc:304] T 88eb51447e0f4080baa49dd902deb24f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:40.935896 1166 ts_tablet_manager.cc:1434] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251025 07:52:40.936507 2106 raft_consensus.cc:2804] T 88eb51447e0f4080baa49dd902deb24f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:40.936587 2106 raft_consensus.cc:493] T 88eb51447e0f4080baa49dd902deb24f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:40.936666 2106 raft_consensus.cc:3060] T 88eb51447e0f4080baa49dd902deb24f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:40.936591 1166 tablet_bootstrap.cc:492] T 724d0b7e1b65445a9a1f056852983bdb P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:40.937042 771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa9d9ab8073945d28d52fb02213f201a" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:40.937222 771 raft_consensus.cc:2393] T aa9d9ab8073945d28d52fb02213f201a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:40.938920 1167 tablet_bootstrap.cc:654] T bc2d16b33ca8426abbd7bd38339c9f1d P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:40.929091 2166 ts_tablet_manager.cc:1950] T 551d07a9242740dcb9366bb2d707a4a8 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
W20251025 07:52:40.930573 710 leader_election.cc:343] T 4c155aa7fa5b4d68bbab1918b8432372 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:40.940279 710 leader_election.cc:304] T 4c155aa7fa5b4d68bbab1918b8432372 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:40.940666 1659 raft_consensus.cc:2749] T 4c155aa7fa5b4d68bbab1918b8432372 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:40.942278 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 66) took 2424 ms. Trace:
I20251025 07:52:40.942394 2166 rpcz_store.cc:276] 1025 07:52:38.517504 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.517547 (+ 43us) service_pool.cc:225] Handling call
1025 07:52:40.851069 (+2333522us) ts_tablet_manager.cc:1151] Deleting tablet 551d07a9242740dcb9366bb2d707a4a8
1025 07:52:40.851085 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:40.888571 (+ 37486us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.923994 (+ 35423us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.942264 (+ 18270us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2333067}
I20251025 07:52:40.942988 2166 tablet_replica.cc:333] T 6ec421eb820b4f40afcff5a5bf80c508 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
W20251025 07:52:40.943145 710 consensus_peers.cc:597] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 27651312cc2e42a8bfcd245f5d301679. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:40.943207 2166 raft_consensus.cc:2243] T 6ec421eb820b4f40afcff5a5bf80c508 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:40.943369 2166 raft_consensus.cc:2272] T 6ec421eb820b4f40afcff5a5bf80c508 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:40.944053 2166 ts_tablet_manager.cc:1916] T 6ec421eb820b4f40afcff5a5bf80c508 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:40.942858 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 551d07a9242740dcb9366bb2d707a4a8 (table default.Table_2c4afce2b14c41e29c5034dc68d31ead [id=dc6d03745bdc42d0ad8bdb9f81971b81]) successfully deleted
I20251025 07:52:40.932484 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet f70091024c034c118fb86578db19a31c (table default.Table_332b308220f64c6dbb5f29711d8463b3 [id=e041946c093b4c789ce175c736cc230b]). Expected version 1 got 0
I20251025 07:52:40.945595 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5]). Expected version 1 got 0
I20251025 07:52:40.945948 1169 tablet_bootstrap.cc:492] T b94634ac5077439fadfbbce569f60e8b P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:40.946144 1169 ts_tablet_manager.cc:1403] T b94634ac5077439fadfbbce569f60e8b P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.287s user 0.004s sys 0.000s
I20251025 07:52:40.929071 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebf3a2a926754305b5bd95fcebb25d29" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:40.953879 773 raft_consensus.cc:2376] T ebf3a2a926754305b5bd95fcebb25d29 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1. Re-sending same reply.
I20251025 07:52:40.957676 1166 tablet_bootstrap.cc:654] T 724d0b7e1b65445a9a1f056852983bdb P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:40.960601 982 consensus_peers.cc:597] T d66ad049d5484228a268b0facb00ecf1 P db42be7ad3c741a0aa427f27e245993b -> Peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Couldn't send request to peer 23eeffd3de254ce18db5febfde971c6b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d66ad049d5484228a268b0facb00ecf1. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:40.961086 2146 ts_tablet_manager.cc:1929] T 467324e2432f4caca148951329caa292 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:40.972397 2146 log.cc:1199] T 467324e2432f4caca148951329caa292 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/467324e2432f4caca148951329caa292
I20251025 07:52:40.973940 2146 ts_tablet_manager.cc:1950] T 467324e2432f4caca148951329caa292 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
W20251025 07:52:40.974637 710 consensus_peers.cc:597] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bef0ef23d90d480d9c99aef21d7271fd. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:40.975785 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 99) took 2515 ms. Trace:
I20251025 07:52:40.975908 2146 rpcz_store.cc:276] 1025 07:52:38.460404 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.460444 (+ 40us) service_pool.cc:225] Handling call
1025 07:52:40.931807 (+2471363us) ts_tablet_manager.cc:1151] Deleting tablet 467324e2432f4caca148951329caa292
1025 07:52:40.931815 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:40.936133 (+ 4318us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.960967 (+ 24834us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:40.975769 (+ 14802us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2471179}
I20251025 07:52:40.976377 2146 tablet_replica.cc:333] T 1d261e432cf5403ea162129706bc4d4d P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:40.976594 2146 raft_consensus.cc:2243] T 1d261e432cf5403ea162129706bc4d4d P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:40.976611 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 467324e2432f4caca148951329caa292 (table default.Table_4cf617a8064c4be99402936ea4cf514b [id=11ce21ee2d434ce38aadc6152c02e2bb]) successfully deleted
I20251025 07:52:40.977005 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]). Expected version 3 got 0
I20251025 07:52:40.977406 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 755f0d6c87044e20a8020f8410940123 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 4 got 0
I20251025 07:52:40.977838 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 22f7db660616457b83b24e947a0b0bae (table default.Table_4a86191163f2408f9bcc795cd8d1bbf6 [id=bb316a7e000748ca902a9bfb494802d7]). Expected version 1 got 0
I20251025 07:52:40.978709 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 0325d462441349ef81841f1b0cab2a20 (table default.Table_750ecb905839434fa9a1aa72f87394be [id=9ce3ba77c56a4ef09cb3b40ea0e488f6]). Expected version 1 got 0
I20251025 07:52:40.982442 2146 raft_consensus.cc:2272] T 1d261e432cf5403ea162129706bc4d4d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:40.983096 2146 ts_tablet_manager.cc:1916] T 1d261e432cf5403ea162129706bc4d4d P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:40.983070 982 leader_election.cc:304] T aa9d9ab8073945d28d52fb02213f201a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:40.983431 2050 raft_consensus.cc:3060] T aa9d9ab8073945d28d52fb02213f201a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:40.984649 1167 tablet_bootstrap.cc:492] T bc2d16b33ca8426abbd7bd38339c9f1d P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:40.984834 1167 ts_tablet_manager.cc:1403] T bc2d16b33ca8426abbd7bd38339c9f1d P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.000s
I20251025 07:52:40.985790 1167 raft_consensus.cc:359] T bc2d16b33ca8426abbd7bd38339c9f1d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:40.985980 1167 raft_consensus.cc:385] T bc2d16b33ca8426abbd7bd38339c9f1d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:40.986063 1167 raft_consensus.cc:740] T bc2d16b33ca8426abbd7bd38339c9f1d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:40.986243 2050 raft_consensus.cc:2749] T aa9d9ab8073945d28d52fb02213f201a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:40.986390 1167 consensus_queue.cc:260] T bc2d16b33ca8426abbd7bd38339c9f1d P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:40.987016 1167 ts_tablet_manager.cc:1434] T bc2d16b33ca8426abbd7bd38339c9f1d P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:40.961182 2348 consensus_queue.cc:1048] T 6a58931e7cd7454fb583f1c70cd2ad15 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:40.979625 2300 consensus_queue.cc:1048] T 88bfc23acd274041a9457079c1709d82 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:40.987336 1167 tablet_bootstrap.cc:492] T 050a5d3b5f934dd19bf2e221f2fe0780 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:40.991360 1167 tablet_bootstrap.cc:654] T 050a5d3b5f934dd19bf2e221f2fe0780 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:40.979763 1169 raft_consensus.cc:359] T b94634ac5077439fadfbbce569f60e8b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.991545 1169 raft_consensus.cc:385] T b94634ac5077439fadfbbce569f60e8b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:40.991623 1169 raft_consensus.cc:740] T b94634ac5077439fadfbbce569f60e8b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:40.991926 1169 consensus_queue.cc:260] T b94634ac5077439fadfbbce569f60e8b P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:40.979936 2169 raft_consensus.cc:2272] T 887e7c7a5d0e4aaf88f26246306df19c P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:40.992482 1169 ts_tablet_manager.cc:1434] T b94634ac5077439fadfbbce569f60e8b P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.046s user 0.002s sys 0.000s
I20251025 07:52:40.992777 1169 tablet_bootstrap.cc:492] T e9f7cd765d614d59a7c190b744b0844d P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:40.992938 2169 ts_tablet_manager.cc:1916] T 887e7c7a5d0e4aaf88f26246306df19c P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:40.995669 982 consensus_peers.cc:597] T 80ee75935d7e461d9d4066fac9819bef P db42be7ad3c741a0aa427f27e245993b -> Peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Couldn't send request to peer 23eeffd3de254ce18db5febfde971c6b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80ee75935d7e461d9d4066fac9819bef. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:40.996088 2348 consensus_queue.cc:1048] T fe5033e5ff404a5ebc6c8443d7548327 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:40.996590 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]). Expected version 1 got 0
I20251025 07:52:40.996958 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_e90d7f5ff13b40878e5b625c97792c7d [id=fc8180c09d7b4e74b2e161f7acb8fe0e]). Expected version 1 got 0
I20251025 07:52:40.997306 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 830fdbc6812e416690d0db42a73af503 (table default.Table_92824b3f2a774564b3aea04aaa25d204 [id=fb21a45b86ab446e8c3dae4512a9598f]). Expected version 1 got 0
I20251025 07:52:40.997746 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c]). Expected version 1 got 0
I20251025 07:52:41.001618 2370 tablet.cc:1722] T 509c94f2a422493ea4390ecee576db72 P 23eeffd3de254ce18db5febfde971c6b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 07:52:41.003634 1166 tablet_bootstrap.cc:492] T 724d0b7e1b65445a9a1f056852983bdb P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:41.004839 2146 ts_tablet_manager.cc:1929] T 1d261e432cf5403ea162129706bc4d4d P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:41.007264 1166 ts_tablet_manager.cc:1403] T 724d0b7e1b65445a9a1f056852983bdb P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.071s user 0.007s sys 0.002s
I20251025 07:52:41.010270 2146 log.cc:1199] T 1d261e432cf5403ea162129706bc4d4d P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/1d261e432cf5403ea162129706bc4d4d
I20251025 07:52:41.011521 2146 ts_tablet_manager.cc:1950] T 1d261e432cf5403ea162129706bc4d4d P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:41.011567 1166 raft_consensus.cc:359] T 724d0b7e1b65445a9a1f056852983bdb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:41.015027 1166 raft_consensus.cc:385] T 724d0b7e1b65445a9a1f056852983bdb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:41.015149 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 100) took 2554 ms. Trace:
I20251025 07:52:41.015172 1166 raft_consensus.cc:740] T 724d0b7e1b65445a9a1f056852983bdb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:41.015784 2146 rpcz_store.cc:276] 1025 07:52:38.460765 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.460807 (+ 42us) service_pool.cc:225] Handling call
1025 07:52:40.976331 (+2515524us) ts_tablet_manager.cc:1151] Deleting tablet 1d261e432cf5403ea162129706bc4d4d
1025 07:52:40.976340 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.001394 (+ 25054us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.004758 (+ 3364us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.015134 (+ 10376us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2515354}
I20251025 07:52:41.016176 1166 consensus_queue.cc:260] T 724d0b7e1b65445a9a1f056852983bdb P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:41.016538 2146 tablet_replica.cc:333] T b94634ac5077439fadfbbce569f60e8b P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:41.016727 2146 raft_consensus.cc:2243] T b94634ac5077439fadfbbce569f60e8b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:41.016878 2146 raft_consensus.cc:2272] T b94634ac5077439fadfbbce569f60e8b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:41.017048 1166 ts_tablet_manager.cc:1434] T 724d0b7e1b65445a9a1f056852983bdb P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251025 07:52:41.017427 2146 ts_tablet_manager.cc:1916] T b94634ac5077439fadfbbce569f60e8b P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:41.016312 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 1d261e432cf5403ea162129706bc4d4d (table default.Table_590cc281d8b042b19ef12c7f6436752a [id=b1eee5c0243a48c4bbdecc0b4f38ab2d]) successfully deleted
I20251025 07:52:41.017508 1166 tablet_bootstrap.cc:492] T 4c155aa7fa5b4d68bbab1918b8432372 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:41.022954 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236]). Expected version 1 got 0
I20251025 07:52:41.008335 2106 raft_consensus.cc:515] T 88eb51447e0f4080baa49dd902deb24f P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:41.024820 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88eb51447e0f4080baa49dd902deb24f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:41.025198 1166 tablet_bootstrap.cc:654] T 4c155aa7fa5b4d68bbab1918b8432372 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:41.025287 850 leader_election.cc:343] T 88eb51447e0f4080baa49dd902deb24f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:41.030897 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3]). Expected version 2 got 0
I20251025 07:52:41.031574 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 8f2c8582fed34c848d9903b8ad8b4057 (table default.Table_438f67f3dc62465999a917986922b3ed [id=7c7fcd265e624d7fab74661db505d9c3]). Expected version 1 got 0
I20251025 07:52:41.033378 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 36262ab8c6964dd684c01b4db6219a4f (table default.Table_76158f8cf82e4cdb9a90580e46aaea48 [id=f249f30497bc450c887d659721469de8]). Expected version 1 got 0
I20251025 07:52:41.033509 1659 raft_consensus.cc:493] T bc2d16b33ca8426abbd7bd38339c9f1d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:41.033644 1659 raft_consensus.cc:515] T bc2d16b33ca8426abbd7bd38339c9f1d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:41.034253 1659 leader_election.cc:290] T bc2d16b33ca8426abbd7bd38339c9f1d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:41.036435 2166 ts_tablet_manager.cc:1929] T 6ec421eb820b4f40afcff5a5bf80c508 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:41.036454 2146 ts_tablet_manager.cc:1929] T b94634ac5077439fadfbbce569f60e8b P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:41.035017 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 2c50ec95e48e49b59a76fac4fc862d3d (table default.Table_32b54e82cbc84d39a0020826f8737d62 [id=c28b0b6acd61490caecfdaab803a9d92]). Expected version 2 got 0
I20251025 07:52:41.036862 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3]). Expected version 1 got 0
I20251025 07:52:41.037635 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet b17d2a9ad6be49d29f2413666fba7e69 (table default.Table_fcddb31989ee4236a4981c12779b3f5d [id=f3794fa081364666b944c412415bdba0]). Expected version 3 got 0
I20251025 07:52:41.039188 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc2d16b33ca8426abbd7bd38339c9f1d" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:41.043236 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 292fe04055954afd8c08ac2636b68191 (table default.Table_9fa69b10b5e842639fc0b0a58cd9b6a5 [id=a253ce0253bd4da4814741c482193d8f]). Expected version 1 got 0
W20251025 07:52:41.039776 712 leader_election.cc:343] T bc2d16b33ca8426abbd7bd38339c9f1d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:41.041652 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88eb51447e0f4080baa49dd902deb24f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:41.043666 774 raft_consensus.cc:3060] T 88eb51447e0f4080baa49dd902deb24f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:41.047330 2370 tablet.cc:1722] T ef1854d35a3047e3a9fbe0166ab1295a P 23eeffd3de254ce18db5febfde971c6b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251025 07:52:41.047327 774 raft_consensus.cc:2468] T 88eb51447e0f4080baa49dd902deb24f P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:41.048924 2166 log.cc:1199] T 6ec421eb820b4f40afcff5a5bf80c508 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/6ec421eb820b4f40afcff5a5bf80c508
I20251025 07:52:41.050114 2166 ts_tablet_manager.cc:1950] T 6ec421eb820b4f40afcff5a5bf80c508 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:41.053339 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 67) took 2535 ms. Trace:
I20251025 07:52:41.053771 2169 ts_tablet_manager.cc:1929] T 887e7c7a5d0e4aaf88f26246306df19c P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:41.054639 2166 rpcz_store.cc:276] 1025 07:52:38.517771 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.517875 (+ 104us) service_pool.cc:225] Handling call
1025 07:52:40.942942 (+2425067us) ts_tablet_manager.cc:1151] Deleting tablet 6ec421eb820b4f40afcff5a5bf80c508
1025 07:52:40.942953 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.000643 (+ 57690us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.036318 (+ 35675us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.053321 (+ 17003us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2424911}
I20251025 07:52:41.054442 1612 raft_consensus.cc:493] T b94634ac5077439fadfbbce569f60e8b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:41.054989 1612 raft_consensus.cc:515] T b94634ac5077439fadfbbce569f60e8b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.055574 1612 leader_election.cc:290] T b94634ac5077439fadfbbce569f60e8b P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:41.056351 2169 log.cc:1199] T 887e7c7a5d0e4aaf88f26246306df19c P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/887e7c7a5d0e4aaf88f26246306df19c
I20251025 07:52:41.057543 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 6ec421eb820b4f40afcff5a5bf80c508 (table default.Table_5487882c2d0c4af4ad17df240fa90277 [id=d73352841fda408baf49087af00b4250]) successfully deleted
I20251025 07:52:41.058449 2166 tablet_replica.cc:333] T 467324e2432f4caca148951329caa292 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:41.059044 2166 raft_consensus.cc:2243] T 467324e2432f4caca148951329caa292 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:41.059274 2169 ts_tablet_manager.cc:1950] T 887e7c7a5d0e4aaf88f26246306df19c P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:41.059934 2166 raft_consensus.cc:2272] T 467324e2432f4caca148951329caa292 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:41.060442 1169 tablet_bootstrap.cc:654] T e9f7cd765d614d59a7c190b744b0844d P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:41.061009 2166 ts_tablet_manager.cc:1916] T 467324e2432f4caca148951329caa292 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:41.062397 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 76) took 2357 ms. Trace:
I20251025 07:52:41.062613 2169 rpcz_store.cc:276] 1025 07:52:38.705710 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.705816 (+ 106us) service_pool.cc:225] Handling call
1025 07:52:40.593864 (+1888048us) ts_tablet_manager.cc:1151] Deleting tablet 887e7c7a5d0e4aaf88f26246306df19c
1025 07:52:40.593874 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.020880 (+427006us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.053630 (+ 32750us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.062373 (+ 8743us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26252,"tablet-delete.queue_time_us":1887801}
I20251025 07:52:41.038282 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc2d16b33ca8426abbd7bd38339c9f1d" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:41.065666 2146 log.cc:1199] T b94634ac5077439fadfbbce569f60e8b P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/b94634ac5077439fadfbbce569f60e8b
I20251025 07:52:41.066045 2146 ts_tablet_manager.cc:1950] T b94634ac5077439fadfbbce569f60e8b P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:41.066437 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 101) took 2605 ms. Trace:
I20251025 07:52:41.041826 2105 consensus_queue.cc:1048] T aa9d9ab8073945d28d52fb02213f201a P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:41.049091 847 leader_election.cc:304] T 88eb51447e0f4080baa49dd902deb24f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:41.054911 960 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
I20251025 07:52:41.056191 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b94634ac5077439fadfbbce569f60e8b" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:41.067049 2146 rpcz_store.cc:276] 1025 07:52:38.461357 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.461905 (+ 548us) service_pool.cc:225] Handling call
1025 07:52:41.016491 (+2554586us) ts_tablet_manager.cc:1151] Deleting tablet b94634ac5077439fadfbbce569f60e8b
1025 07:52:41.016501 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.032687 (+ 16186us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.036155 (+ 3468us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.066419 (+ 30264us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2553486}
I20251025 07:52:41.041054 2106 leader_election.cc:290] T 88eb51447e0f4080baa49dd902deb24f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:41.067601 2106 raft_consensus.cc:2804] T 88eb51447e0f4080baa49dd902deb24f P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
W20251025 07:52:41.067795 710 leader_election.cc:343] T bc2d16b33ca8426abbd7bd38339c9f1d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:41.067843 2106 raft_consensus.cc:697] T 88eb51447e0f4080baa49dd902deb24f P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:41.067909 710 leader_election.cc:304] T bc2d16b33ca8426abbd7bd38339c9f1d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:41.068079 2106 consensus_queue.cc:237] T 88eb51447e0f4080baa49dd902deb24f P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:41.068619 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 104) took 2600 ms. Trace:
I20251025 07:52:41.068691 2146 rpcz_store.cc:276] 1025 07:52:38.468251 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.468304 (+ 53us) service_pool.cc:225] Handling call
1025 07:52:41.067363 (+2599059us) ts_tablet_manager.cc:1151] Deleting tablet 3775a2e08d4e40bd9f57e4cda759d535
1025 07:52:41.068517 (+ 1154us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.068604 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2595393}
I20251025 07:52:41.068943 2146 tablet_replica.cc:333] T 2784bbfb97d5466f9ec856add7d0b29a P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:41.069113 2146 raft_consensus.cc:2243] T 2784bbfb97d5466f9ec856add7d0b29a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:41.069250 2146 raft_consensus.cc:2272] T 2784bbfb97d5466f9ec856add7d0b29a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:41.069759 2146 ts_tablet_manager.cc:1916] T 2784bbfb97d5466f9ec856add7d0b29a P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:41.077518 2124 raft_consensus.cc:2749] T bc2d16b33ca8426abbd7bd38339c9f1d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:41.078352 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet b94634ac5077439fadfbbce569f60e8b (table default.Table_1ff54f4df7a2444489ae4e4060084156 [id=b88c37a678344c6b8e8017553e486cb2]) successfully deleted
W20251025 07:52:41.078677 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 3775a2e08d4e40bd9f57e4cda759d535 because the tablet was not found. No further retry: Not found: Tablet not found: 3775a2e08d4e40bd9f57e4cda759d535
I20251025 07:52:41.082294 2105 consensus_queue.cc:1048] T e54fe7cbcffa40c4b04144b47bee2b48 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:41.089556 1659 raft_consensus.cc:493] T 04449cc97f404a19b6a1b20dea074f95 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:41.089733 1659 raft_consensus.cc:515] T 04449cc97f404a19b6a1b20dea074f95 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:41.091611 1659 leader_election.cc:290] T 04449cc97f404a19b6a1b20dea074f95 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:41.092195 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04449cc97f404a19b6a1b20dea074f95" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:41.092759 710 leader_election.cc:343] T 04449cc97f404a19b6a1b20dea074f95 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:41.094002 1167 tablet_bootstrap.cc:492] T 050a5d3b5f934dd19bf2e221f2fe0780 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:41.094192 1167 ts_tablet_manager.cc:1403] T 050a5d3b5f934dd19bf2e221f2fe0780 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.107s user 0.005s sys 0.002s
I20251025 07:52:41.095144 1167 raft_consensus.cc:359] T 050a5d3b5f934dd19bf2e221f2fe0780 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:41.095456 1166 tablet_bootstrap.cc:492] T 4c155aa7fa5b4d68bbab1918b8432372 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:41.095726 1166 ts_tablet_manager.cc:1403] T 4c155aa7fa5b4d68bbab1918b8432372 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.078s user 0.002s sys 0.016s
I20251025 07:52:41.096151 1167 raft_consensus.cc:385] T 050a5d3b5f934dd19bf2e221f2fe0780 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:41.096374 1167 raft_consensus.cc:740] T 050a5d3b5f934dd19bf2e221f2fe0780 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:41.096768 1167 consensus_queue.cc:260] T 050a5d3b5f934dd19bf2e221f2fe0780 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:41.096992 1166 raft_consensus.cc:359] T 4c155aa7fa5b4d68bbab1918b8432372 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.097246 1166 raft_consensus.cc:385] T 4c155aa7fa5b4d68bbab1918b8432372 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:41.097323 1166 raft_consensus.cc:740] T 4c155aa7fa5b4d68bbab1918b8432372 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:41.097705 1166 consensus_queue.cc:260] T 4c155aa7fa5b4d68bbab1918b8432372 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.098352 1167 ts_tablet_manager.cc:1434] T 050a5d3b5f934dd19bf2e221f2fe0780 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251025 07:52:41.098618 1166 ts_tablet_manager.cc:1434] T 4c155aa7fa5b4d68bbab1918b8432372 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251025 07:52:41.103945 1167 tablet_bootstrap.cc:492] T aaef24ccdfba45fe8204327723df834e P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:41.063035 2316 consensus_queue.cc:1048] T 88bfc23acd274041a9457079c1709d82 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251025 07:52:41.070528 983 leader_election.cc:343] T b94634ac5077439fadfbbce569f60e8b P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: Tablet not RUNNING: SHUTDOWN
I20251025 07:52:41.107488 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04449cc97f404a19b6a1b20dea074f95" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:41.095645 2348 consensus_queue.cc:1048] T ef9a2c9f2ccf432ab835396020872208 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:41.099196 1169 tablet_bootstrap.cc:492] T e9f7cd765d614d59a7c190b744b0844d P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:41.108857 1169 ts_tablet_manager.cc:1403] T e9f7cd765d614d59a7c190b744b0844d P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.116s user 0.003s sys 0.000s
I20251025 07:52:41.109699 1169 raft_consensus.cc:359] T e9f7cd765d614d59a7c190b744b0844d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.109880 1169 raft_consensus.cc:385] T e9f7cd765d614d59a7c190b744b0844d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:41.109970 1169 raft_consensus.cc:740] T e9f7cd765d614d59a7c190b744b0844d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:41.110252 1169 consensus_queue.cc:260] T e9f7cd765d614d59a7c190b744b0844d P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.110834 1169 ts_tablet_manager.cc:1434] T e9f7cd765d614d59a7c190b744b0844d P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:41.111240 1169 tablet_bootstrap.cc:492] T f97acc9a638b4d5a968dcc3412a384bc P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:41.113003 1169 tablet_bootstrap.cc:654] T f97acc9a638b4d5a968dcc3412a384bc P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:41.115617 1166 tablet_bootstrap.cc:492] T c2713942584e4734acbc0775ead772fa P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:41.117051 2314 consensus_queue.cc:1048] T aa9d9ab8073945d28d52fb02213f201a P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:41.117260 1166 tablet_bootstrap.cc:654] T c2713942584e4734acbc0775ead772fa P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:41.118059 2146 ts_tablet_manager.cc:1929] T 2784bbfb97d5466f9ec856add7d0b29a P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:41.118191 2146 log.cc:1199] T 2784bbfb97d5466f9ec856add7d0b29a P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/2784bbfb97d5466f9ec856add7d0b29a
I20251025 07:52:41.115636 2105 consensus_queue.cc:1048] T e54fe7cbcffa40c4b04144b47bee2b48 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20251025 07:52:41.118526 2146 ts_tablet_manager.cc:1950] T 2784bbfb97d5466f9ec856add7d0b29a P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:41.119370 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 102) took 2651 ms. Trace:
I20251025 07:52:41.119473 2146 rpcz_store.cc:276] 1025 07:52:38.468005 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.468621 (+ 616us) service_pool.cc:225] Handling call
1025 07:52:41.068903 (+2600282us) ts_tablet_manager.cc:1151] Deleting tablet 2784bbfb97d5466f9ec856add7d0b29a
1025 07:52:41.068912 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.079458 (+ 10546us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.117941 (+ 38483us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.119349 (+ 1408us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2596552}
I20251025 07:52:41.120003 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 2784bbfb97d5466f9ec856add7d0b29a (table default.Table_5aff13b5fcbc4899800760718fb3cc90 [id=ab3900039f5c491bbc1b96925deb1f1b]) successfully deleted
W20251025 07:52:41.122612 982 leader_election.cc:343] T b94634ac5077439fadfbbce569f60e8b P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: b94634ac5077439fadfbbce569f60e8b
I20251025 07:52:41.123323 982 leader_election.cc:304] T b94634ac5077439fadfbbce569f60e8b P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:41.123564 2348 raft_consensus.cc:2749] T b94634ac5077439fadfbbce569f60e8b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:41.134059 1096 heartbeater.cc:344] Connected to a master server at 127.0.104.61:42203
I20251025 07:52:41.135788 2146 tablet_replica.cc:333] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:41.136087 2146 raft_consensus.cc:2243] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:41.136642 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 77) took 2429 ms. Trace:
I20251025 07:52:41.136682 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 887e7c7a5d0e4aaf88f26246306df19c (table default.Table_2c1cdea0079240b889637e5d8d13da95 [id=8e07d0ad2db84d1b8a3a3a3efa2d7f69]) successfully deleted
I20251025 07:52:41.136744 2169 rpcz_store.cc:276] 1025 07:52:38.707467 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.707999 (+ 532us) service_pool.cc:225] Handling call
1025 07:52:41.136524 (+2428525us) ts_tablet_manager.cc:1151] Deleting tablet c11f674385184ae6b235f1db8bf1fb91
1025 07:52:41.136534 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.136626 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2428285}
I20251025 07:52:41.137099 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 78) took 2428 ms. Trace:
W20251025 07:52:41.137257 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet c11f674385184ae6b235f1db8bf1fb91 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c11f674385184ae6b235f1db8bf1fb91 already in progress: creating tablet
I20251025 07:52:41.137192 2169 rpcz_store.cc:276] 1025 07:52:38.709372 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.709448 (+ 76us) service_pool.cc:225] Handling call
1025 07:52:41.136998 (+2427550us) ts_tablet_manager.cc:1151] Deleting tablet d77704b4fe6c4456a2ba40ffec0c60fc
1025 07:52:41.137009 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.137085 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2427209}
I20251025 07:52:41.137501 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 79) took 2427 ms. Trace:
I20251025 07:52:41.137580 2169 rpcz_store.cc:276] 1025 07:52:38.710554 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.710944 (+ 390us) service_pool.cc:225] Handling call
1025 07:52:41.137416 (+2426472us) ts_tablet_manager.cc:1151] Deleting tablet e4da89f863cc43f3bf6df908bb9dc6bf
1025 07:52:41.137424 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.137485 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2426184}
W20251025 07:52:41.137686 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet d77704b4fe6c4456a2ba40ffec0c60fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d77704b4fe6c4456a2ba40ffec0c60fc already in progress: creating tablet
I20251025 07:52:41.137828 2169 tablet_replica.cc:333] T 4037dee385d04e85bb78c8e77cb00379 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:41.138024 2169 raft_consensus.cc:2243] T 4037dee385d04e85bb78c8e77cb00379 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
W20251025 07:52:41.138159 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet e4da89f863cc43f3bf6df908bb9dc6bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4da89f863cc43f3bf6df908bb9dc6bf already in progress: creating tablet
I20251025 07:52:41.138314 2169 raft_consensus.cc:2272] T 4037dee385d04e85bb78c8e77cb00379 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:41.138949 2169 ts_tablet_manager.cc:1916] T 4037dee385d04e85bb78c8e77cb00379 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:41.108361 712 leader_election.cc:343] T 04449cc97f404a19b6a1b20dea074f95 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:41.140884 712 leader_election.cc:304] T 04449cc97f404a19b6a1b20dea074f95 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:41.141156 2124 raft_consensus.cc:2749] T 04449cc97f404a19b6a1b20dea074f95 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:41.151064 1167 tablet_bootstrap.cc:654] T aaef24ccdfba45fe8204327723df834e P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:41.156220 1659 consensus_queue.cc:1048] T 943ec703e65f4fd3b0c690e08c0bd226 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20251025 07:52:41.136479 2146 raft_consensus.cc:2272] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:41.170845 2146 ts_tablet_manager.cc:1916] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:41.177191 2316 consensus_queue.cc:1048] T 22c97b3ee92140c68a7e1b93033cb299 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251025 07:52:41.177340 710 consensus_peers.cc:597] T cb824bd780524832b2b60f5618a39da5 P 23eeffd3de254ce18db5febfde971c6b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cb824bd780524832b2b60f5618a39da5. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:41.180843 2314 raft_consensus.cc:493] T 0325d462441349ef81841f1b0cab2a20 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:41.181001 2314 raft_consensus.cc:515] T 0325d462441349ef81841f1b0cab2a20 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:41.181591 2314 leader_election.cc:290] T 0325d462441349ef81841f1b0cab2a20 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:41.182363 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325d462441349ef81841f1b0cab2a20" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:41.182916 850 leader_election.cc:343] T 0325d462441349ef81841f1b0cab2a20 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:41.182917 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325d462441349ef81841f1b0cab2a20" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:41.183197 775 raft_consensus.cc:2468] T 0325d462441349ef81841f1b0cab2a20 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:41.183643 847 leader_election.cc:304] T 0325d462441349ef81841f1b0cab2a20 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:41.184229 2314 raft_consensus.cc:2804] T 0325d462441349ef81841f1b0cab2a20 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:41.184314 2314 raft_consensus.cc:493] T 0325d462441349ef81841f1b0cab2a20 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:41.184398 2314 raft_consensus.cc:3060] T 0325d462441349ef81841f1b0cab2a20 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:41.184478 2166 ts_tablet_manager.cc:1929] T 467324e2432f4caca148951329caa292 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:41.184589 2166 log.cc:1199] T 467324e2432f4caca148951329caa292 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/467324e2432f4caca148951329caa292
I20251025 07:52:41.184952 2166 ts_tablet_manager.cc:1950] T 467324e2432f4caca148951329caa292 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:41.185269 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 68) took 2667 ms. Trace:
I20251025 07:52:41.185340 2166 rpcz_store.cc:276] 1025 07:52:38.517846 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.518063 (+ 217us) service_pool.cc:225] Handling call
1025 07:52:41.058412 (+2540349us) ts_tablet_manager.cc:1151] Deleting tablet 467324e2432f4caca148951329caa292
1025 07:52:41.058420 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.176078 (+117658us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.184384 (+ 8306us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.185254 (+ 870us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2540201}
I20251025 07:52:41.185673 2166 tablet_replica.cc:333] T 1d261e432cf5403ea162129706bc4d4d P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:41.185878 2166 raft_consensus.cc:2243] T 1d261e432cf5403ea162129706bc4d4d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:41.186048 2166 raft_consensus.cc:2272] T 1d261e432cf5403ea162129706bc4d4d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:41.187559 2314 raft_consensus.cc:515] T 0325d462441349ef81841f1b0cab2a20 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:41.189065 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325d462441349ef81841f1b0cab2a20" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:41.189208 2314 leader_election.cc:290] T 0325d462441349ef81841f1b0cab2a20 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:41.189503 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0325d462441349ef81841f1b0cab2a20" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:41.189656 768 raft_consensus.cc:3060] T 0325d462441349ef81841f1b0cab2a20 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:41.192472 768 raft_consensus.cc:2468] T 0325d462441349ef81841f1b0cab2a20 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:41.192916 847 leader_election.cc:304] T 0325d462441349ef81841f1b0cab2a20 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
W20251025 07:52:41.193120 850 leader_election.cc:343] T 0325d462441349ef81841f1b0cab2a20 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:41.193838 2169 ts_tablet_manager.cc:1929] T 4037dee385d04e85bb78c8e77cb00379 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:41.193674 2106 raft_consensus.cc:2804] T 0325d462441349ef81841f1b0cab2a20 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:41.194309 2106 raft_consensus.cc:697] T 0325d462441349ef81841f1b0cab2a20 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:41.196440 2106 consensus_queue.cc:237] T 0325d462441349ef81841f1b0cab2a20 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:41.198506 1167 tablet_bootstrap.cc:492] T aaef24ccdfba45fe8204327723df834e P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:41.198683 1167 ts_tablet_manager.cc:1403] T aaef24ccdfba45fe8204327723df834e P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.095s user 0.003s sys 0.003s
I20251025 07:52:41.199517 1167 raft_consensus.cc:359] T aaef24ccdfba45fe8204327723df834e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.199714 1167 raft_consensus.cc:385] T aaef24ccdfba45fe8204327723df834e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:41.199798 1167 raft_consensus.cc:740] T aaef24ccdfba45fe8204327723df834e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:41.200003 1167 consensus_queue.cc:260] T aaef24ccdfba45fe8204327723df834e P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.200717 1167 ts_tablet_manager.cc:1434] T aaef24ccdfba45fe8204327723df834e P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251025 07:52:41.201066 1167 tablet_bootstrap.cc:492] T 26c849d376d143c6bef0c8e0e1d7d03a P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:41.202757 2124 consensus_queue.cc:1048] T 943ec703e65f4fd3b0c690e08c0bd226 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:41.203068 1167 tablet_bootstrap.cc:654] T 26c849d376d143c6bef0c8e0e1d7d03a P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:41.204558 2166 ts_tablet_manager.cc:1916] T 1d261e432cf5403ea162129706bc4d4d P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:41.206148 2314 raft_consensus.cc:493] T 7c7a9d7a4fb3449bb57eafa83230a839 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:41.206295 2314 raft_consensus.cc:515] T 7c7a9d7a4fb3449bb57eafa83230a839 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.207460 2314 leader_election.cc:290] T 7c7a9d7a4fb3449bb57eafa83230a839 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:41.208349 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c7a9d7a4fb3449bb57eafa83230a839" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:41.208585 775 raft_consensus.cc:2468] T 7c7a9d7a4fb3449bb57eafa83230a839 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:41.209116 847 leader_election.cc:304] T 7c7a9d7a4fb3449bb57eafa83230a839 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:41.209394 2314 raft_consensus.cc:2804] T 7c7a9d7a4fb3449bb57eafa83230a839 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:41.209532 2314 raft_consensus.cc:493] T 7c7a9d7a4fb3449bb57eafa83230a839 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:41.209658 2314 raft_consensus.cc:3060] T 7c7a9d7a4fb3449bb57eafa83230a839 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:41.214105 1167 tablet_bootstrap.cc:492] T 26c849d376d143c6bef0c8e0e1d7d03a P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:41.214269 1167 ts_tablet_manager.cc:1403] T 26c849d376d143c6bef0c8e0e1d7d03a P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.001s
I20251025 07:52:41.216187 1167 raft_consensus.cc:359] T 26c849d376d143c6bef0c8e0e1d7d03a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:41.216338 1167 raft_consensus.cc:385] T 26c849d376d143c6bef0c8e0e1d7d03a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:41.216390 1167 raft_consensus.cc:740] T 26c849d376d143c6bef0c8e0e1d7d03a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:41.205348 1169 tablet_bootstrap.cc:492] T f97acc9a638b4d5a968dcc3412a384bc P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:41.207315 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c7a9d7a4fb3449bb57eafa83230a839" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:41.218945 1169 ts_tablet_manager.cc:1403] T f97acc9a638b4d5a968dcc3412a384bc P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.108s user 0.008s sys 0.001s
W20251025 07:52:41.219357 850 leader_election.cc:343] T 7c7a9d7a4fb3449bb57eafa83230a839 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:41.208043 2348 consensus_queue.cc:1048] T ef9a2c9f2ccf432ab835396020872208 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:41.219734 1169 raft_consensus.cc:359] T f97acc9a638b4d5a968dcc3412a384bc P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.219890 1169 raft_consensus.cc:385] T f97acc9a638b4d5a968dcc3412a384bc P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:41.219941 1169 raft_consensus.cc:740] T f97acc9a638b4d5a968dcc3412a384bc P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:41.220198 1169 consensus_queue.cc:260] T f97acc9a638b4d5a968dcc3412a384bc P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.220914 1169 ts_tablet_manager.cc:1434] T f97acc9a638b4d5a968dcc3412a384bc P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:41.221091 2169 log.cc:1199] T 4037dee385d04e85bb78c8e77cb00379 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/4037dee385d04e85bb78c8e77cb00379
I20251025 07:52:41.221623 2169 ts_tablet_manager.cc:1950] T 4037dee385d04e85bb78c8e77cb00379 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:41.218569 1167 consensus_queue.cc:260] T 26c849d376d143c6bef0c8e0e1d7d03a P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:41.222411 1167 ts_tablet_manager.cc:1434] T 26c849d376d143c6bef0c8e0e1d7d03a P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20251025 07:52:41.223479 1167 tablet_bootstrap.cc:492] T 04d27d54fdb7483b869408af65ff85c9 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:41.217904 2314 raft_consensus.cc:515] T 7c7a9d7a4fb3449bb57eafa83230a839 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.226017 1167 tablet_bootstrap.cc:654] T 04d27d54fdb7483b869408af65ff85c9 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:41.229686 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 467324e2432f4caca148951329caa292 (table default.Table_4cf617a8064c4be99402936ea4cf514b [id=11ce21ee2d434ce38aadc6152c02e2bb]) successfully deleted
I20251025 07:52:41.230082 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c7a9d7a4fb3449bb57eafa83230a839" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:41.233490 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c7a9d7a4fb3449bb57eafa83230a839" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:41.232031 773 raft_consensus.cc:3060] T 7c7a9d7a4fb3449bb57eafa83230a839 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:41.233176 1166 tablet_bootstrap.cc:492] T c2713942584e4734acbc0775ead772fa P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:41.234457 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 80) took 2522 ms. Trace:
I20251025 07:52:41.234593 2169 rpcz_store.cc:276] 1025 07:52:38.712501 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.712759 (+ 258us) service_pool.cc:225] Handling call
1025 07:52:41.137790 (+2425031us) ts_tablet_manager.cc:1151] Deleting tablet 4037dee385d04e85bb78c8e77cb00379
1025 07:52:41.137798 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.181431 (+ 43633us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.193739 (+ 12308us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.234438 (+ 40699us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2424782}
W20251025 07:52:41.234387 850 leader_election.cc:343] T 7c7a9d7a4fb3449bb57eafa83230a839 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:41.235036 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 4037dee385d04e85bb78c8e77cb00379 (table default.Table_892da226f7054e61b5ffc882b1d5c35a [id=c13e40cbb3624b3997e907d4f2c1f706]) successfully deleted
I20251025 07:52:41.233533 2314 leader_election.cc:290] T 7c7a9d7a4fb3449bb57eafa83230a839 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:41.235168 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 81) took 2521 ms. Trace:
I20251025 07:52:41.235235 2169 rpcz_store.cc:276] 1025 07:52:38.713448 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.713578 (+ 130us) service_pool.cc:225] Handling call
1025 07:52:41.235053 (+2521475us) ts_tablet_manager.cc:1151] Deleting tablet e47c4f741f354a19b49777bd06e54764
1025 07:52:41.235063 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.235155 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2521276}
I20251025 07:52:41.235838 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 82) took 2522 ms. Trace:
W20251025 07:52:41.235890 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet e47c4f741f354a19b49777bd06e54764 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e47c4f741f354a19b49777bd06e54764 already in progress: creating tablet
I20251025 07:52:41.236008 2169 rpcz_store.cc:276] 1025 07:52:38.713526 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.715152 (+ 1626us) service_pool.cc:225] Handling call
1025 07:52:41.235750 (+2520598us) ts_tablet_manager.cc:1151] Deleting tablet 72dc084f8efc45478771c247e07a16f4
1025 07:52:41.235759 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.235825 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2520439}
I20251025 07:52:41.235862 1169 tablet_bootstrap.cc:492] T 7c7a9d7a4fb3449bb57eafa83230a839 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
W20251025 07:52:41.236421 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 72dc084f8efc45478771c247e07a16f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72dc084f8efc45478771c247e07a16f4 already in progress: creating tablet
I20251025 07:52:41.236466 2169 tablet_replica.cc:333] T be5ab0db6ac34c2c9a370445e251eb46 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:41.236656 2169 raft_consensus.cc:2243] T be5ab0db6ac34c2c9a370445e251eb46 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:41.236217 1166 ts_tablet_manager.cc:1403] T c2713942584e4734acbc0775ead772fa P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.120s user 0.011s sys 0.015s
I20251025 07:52:41.236966 2169 raft_consensus.cc:2272] T be5ab0db6ac34c2c9a370445e251eb46 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:41.237696 2169 ts_tablet_manager.cc:1916] T be5ab0db6ac34c2c9a370445e251eb46 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:41.241366 773 raft_consensus.cc:2468] T 7c7a9d7a4fb3449bb57eafa83230a839 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:41.243177 1166 raft_consensus.cc:359] T c2713942584e4734acbc0775ead772fa P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:41.243400 2146 ts_tablet_manager.cc:1929] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 07:52:41.244693 2146 log.cc:1199] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/627a10a0714f4be8bdbced503fd6388d
I20251025 07:52:41.245123 2146 ts_tablet_manager.cc:1950] T 627a10a0714f4be8bdbced503fd6388d P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:41.245690 1166 raft_consensus.cc:385] T c2713942584e4734acbc0775ead772fa P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:41.245563 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 105) took 2777 ms. Trace:
I20251025 07:52:41.246177 1166 raft_consensus.cc:740] T c2713942584e4734acbc0775ead772fa P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:41.253687 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_a6c869949b744bc98efb735b0e73e7aa [id=bf77b5ff6f1048dd996d948437f9ff5e]). Expected version 2 got 0
I20251025 07:52:41.254390 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3]). Expected version 1 got 0
I20251025 07:52:41.255344 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5]). Expected version 1 got 0
I20251025 07:52:41.257355 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f]). Expected version 1 got 0
I20251025 07:52:41.258314 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7]). Expected version 1 got 0
I20251025 07:52:41.259029 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 872cb7512d104827b1f1f186d33a4a44 (table default.Table_d5c65dce8b1b4214ba519d964012bfc0 [id=d4be274b36a845a99a4b96b8846248fd]). Expected version 1 got 0
I20251025 07:52:41.259967 2169 ts_tablet_manager.cc:1929] T be5ab0db6ac34c2c9a370445e251eb46 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:41.260493 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 04d27d54fdb7483b869408af65ff85c9 (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]). Expected version 2 got 0
I20251025 07:52:41.246399 2146 rpcz_store.cc:276] 1025 07:52:38.468587 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.469054 (+ 467us) service_pool.cc:225] Handling call
1025 07:52:41.127255 (+2658201us) ts_tablet_manager.cc:1151] Deleting tablet 627a10a0714f4be8bdbced503fd6388d
1025 07:52:41.135726 (+ 8471us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.184262 (+ 48536us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.235745 (+ 51483us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.245547 (+ 9802us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2654844}
I20251025 07:52:41.261579 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 627a10a0714f4be8bdbced503fd6388d (table default.Table_8773c967a71644baba838946c154aee2 [id=7fc07b0c33f74604be526c4a0df949e9]) successfully deleted
I20251025 07:52:41.260975 1166 consensus_queue.cc:260] T c2713942584e4734acbc0775ead772fa P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:41.263033 2169 log.cc:1199] T be5ab0db6ac34c2c9a370445e251eb46 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/be5ab0db6ac34c2c9a370445e251eb46
I20251025 07:52:41.262930 2146 tablet_replica.cc:333] T 35ba9d23262248cda6fdf03c51be11a4 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:41.263401 2169 ts_tablet_manager.cc:1950] T be5ab0db6ac34c2c9a370445e251eb46 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:41.263666 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 83) took 2548 ms. Trace:
I20251025 07:52:41.262766 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 4c155aa7fa5b4d68bbab1918b8432372 (table default.Table_6c5a24747ba14352b0285a41ec2985a4 [id=28bbaad474164a8c8c97a76bb8081791]). Expected version 2 got 0
I20251025 07:52:41.264420 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet cb829649f73d4d3f8862d268c953bd86 (table default.Table_ba4b8cc7b930482690b7e192e0edff12 [id=7944225d3ce7439b881bfb9289d9f040]). Expected version 1 got 0
I20251025 07:52:41.264736 2146 raft_consensus.cc:2243] T 35ba9d23262248cda6fdf03c51be11a4 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:41.265662 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 292fe04055954afd8c08ac2636b68191 (table default.Table_9fa69b10b5e842639fc0b0a58cd9b6a5 [id=a253ce0253bd4da4814741c482193d8f]). Expected version 1 got 0
I20251025 07:52:41.260011 1169 tablet_bootstrap.cc:654] T 7c7a9d7a4fb3449bb57eafa83230a839 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:41.271541 2169 rpcz_store.cc:276] 1025 07:52:38.715503 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.715574 (+ 71us) service_pool.cc:225] Handling call
1025 07:52:41.236424 (+2520850us) ts_tablet_manager.cc:1151] Deleting tablet be5ab0db6ac34c2c9a370445e251eb46
1025 07:52:41.236434 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.251451 (+ 15017us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.259865 (+ 8414us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.263654 (+ 3789us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2520687}
I20251025 07:52:41.274654 2146 raft_consensus.cc:2272] T 35ba9d23262248cda6fdf03c51be11a4 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:41.274876 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 84) took 2558 ms. Trace:
I20251025 07:52:41.274999 2169 rpcz_store.cc:276] 1025 07:52:38.716235 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.716362 (+ 127us) service_pool.cc:225] Handling call
1025 07:52:41.274756 (+2558394us) ts_tablet_manager.cc:1151] Deleting tablet 4f7c15529be54a11961f385a953b0b03
1025 07:52:41.274764 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.274857 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2558255}
I20251025 07:52:41.272562 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet be5ab0db6ac34c2c9a370445e251eb46 (table default.Table_8faecb0cb37949a39f4ef5d9d2740c3d [id=f03dd08b39184812884db77faf4bb39e]) successfully deleted
I20251025 07:52:41.275480 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet aa9d9ab8073945d28d52fb02213f201a (table default.Table_4791672658f043f6b97b9320a531ebb5 [id=dfd20cf9c3e541c4abe23d59e0aa7f2a]). Expected version 1 got 0
I20251025 07:52:41.275839 2169 tablet_replica.cc:333] T abf94c32ff2945119b710905c075d63c P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
W20251025 07:52:41.275820 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 4f7c15529be54a11961f385a953b0b03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f7c15529be54a11961f385a953b0b03 already in progress: creating tablet
I20251025 07:52:41.276105 2169 raft_consensus.cc:2243] T abf94c32ff2945119b710905c075d63c P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:41.276480 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet b17d2a9ad6be49d29f2413666fba7e69 (table default.Table_fcddb31989ee4236a4981c12779b3f5d [id=f3794fa081364666b944c412415bdba0]). Expected version 3 got 0
I20251025 07:52:41.276814 2169 raft_consensus.cc:2272] T abf94c32ff2945119b710905c075d63c P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:41.277163 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]). Expected version 1 got 0
I20251025 07:52:41.277710 2169 ts_tablet_manager.cc:1916] T abf94c32ff2945119b710905c075d63c P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:41.277911 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet ef9a2c9f2ccf432ab835396020872208 (table default.Table_2b9b772d4c234ac286bef3658482e490 [id=409fab60f9d744798e6f92b868f0f7c2]). Expected version 1 got 0
I20251025 07:52:41.278174 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]). Expected version 1 got 0
I20251025 07:52:41.279975 2348 consensus_queue.cc:1048] T 22c97b3ee92140c68a7e1b93033cb299 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:41.291735 2146 ts_tablet_manager.cc:1916] T 35ba9d23262248cda6fdf03c51be11a4 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:41.274634 1167 tablet_bootstrap.cc:492] T 04d27d54fdb7483b869408af65ff85c9 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:41.292776 1167 ts_tablet_manager.cc:1403] T 04d27d54fdb7483b869408af65ff85c9 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.069s user 0.008s sys 0.002s
I20251025 07:52:41.293637 1167 raft_consensus.cc:359] T 04d27d54fdb7483b869408af65ff85c9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.292397 847 leader_election.cc:304] T 7c7a9d7a4fb3449bb57eafa83230a839 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:41.294484 2106 raft_consensus.cc:2804] T 7c7a9d7a4fb3449bb57eafa83230a839 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:41.294564 1166 ts_tablet_manager.cc:1434] T c2713942584e4734acbc0775ead772fa P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.052s user 0.002s sys 0.000s
I20251025 07:52:41.294607 2106 raft_consensus.cc:697] T 7c7a9d7a4fb3449bb57eafa83230a839 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:41.294833 1166 tablet_bootstrap.cc:492] T aaef24ccdfba45fe8204327723df834e P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:41.295266 1167 raft_consensus.cc:385] T 04d27d54fdb7483b869408af65ff85c9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:41.295502 1167 raft_consensus.cc:740] T 04d27d54fdb7483b869408af65ff85c9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:41.295886 2106 consensus_queue.cc:237] T 7c7a9d7a4fb3449bb57eafa83230a839 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.295859 1167 consensus_queue.cc:260] T 04d27d54fdb7483b869408af65ff85c9 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.296854 1167 ts_tablet_manager.cc:1434] T 04d27d54fdb7483b869408af65ff85c9 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251025 07:52:41.297420 1167 tablet_bootstrap.cc:492] T 30418858bf95438686bdfe64ac733144 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:41.306319 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84]). Expected version 1 got 0
I20251025 07:52:41.309201 1167 tablet_bootstrap.cc:654] T 30418858bf95438686bdfe64ac733144 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:41.309759 1169 tablet_bootstrap.cc:492] T 7c7a9d7a4fb3449bb57eafa83230a839 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:41.314093 1169 ts_tablet_manager.cc:1403] T 7c7a9d7a4fb3449bb57eafa83230a839 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.081s user 0.002s sys 0.006s
I20251025 07:52:41.316586 1166 tablet_bootstrap.cc:654] T aaef24ccdfba45fe8204327723df834e P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:41.322459 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 5dab439bc50d4e869548154e1d222ad6 (table default.Table_bd82d871f84943f4baa967f5c0ec74a3 [id=45325c188eac4d34bb016d01de091f87]). Expected version 2 got 0
I20251025 07:52:41.322715 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_87d38832f85e4965bfab1a077990114f [id=ad812862698d421184ebf501ce79b7be]). Expected version 2 got 0
I20251025 07:52:41.323463 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet f70091024c034c118fb86578db19a31c (table default.Table_332b308220f64c6dbb5f29711d8463b3 [id=e041946c093b4c789ce175c736cc230b]). Expected version 1 got 0
I20251025 07:52:41.322966 1169 raft_consensus.cc:359] T 7c7a9d7a4fb3449bb57eafa83230a839 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.323961 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]). Expected version 3 got 0
I20251025 07:52:41.323989 1169 raft_consensus.cc:385] T 7c7a9d7a4fb3449bb57eafa83230a839 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:41.324337 1169 raft_consensus.cc:740] T 7c7a9d7a4fb3449bb57eafa83230a839 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:41.324453 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 22f7db660616457b83b24e947a0b0bae (table default.Table_4a86191163f2408f9bcc795cd8d1bbf6 [id=bb316a7e000748ca902a9bfb494802d7]). Expected version 1 got 0
I20251025 07:52:41.325011 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 0325d462441349ef81841f1b0cab2a20 (table default.Table_750ecb905839434fa9a1aa72f87394be [id=9ce3ba77c56a4ef09cb3b40ea0e488f6]). Expected version 1 got 0
I20251025 07:52:41.325372 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 830fdbc6812e416690d0db42a73af503 (table default.Table_92824b3f2a774564b3aea04aaa25d204 [id=fb21a45b86ab446e8c3dae4512a9598f]). Expected version 1 got 0
I20251025 07:52:41.325507 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 755f0d6c87044e20a8020f8410940123 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 4 got 0
I20251025 07:52:41.325192 1169 consensus_queue.cc:260] T 7c7a9d7a4fb3449bb57eafa83230a839 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.325829 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]). Expected version 1 got 0
I20251025 07:52:41.331115 1169 ts_tablet_manager.cc:1434] T 7c7a9d7a4fb3449bb57eafa83230a839 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20251025 07:52:41.332150 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_e90d7f5ff13b40878e5b625c97792c7d [id=fc8180c09d7b4e74b2e161f7acb8fe0e]). Expected version 1 got 0
I20251025 07:52:41.332578 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236]). Expected version 1 got 0
I20251025 07:52:41.332828 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c]). Expected version 1 got 0
I20251025 07:52:41.333246 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3]). Expected version 2 got 0
I20251025 07:52:41.336107 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 8f2c8582fed34c848d9903b8ad8b4057 (table default.Table_438f67f3dc62465999a917986922b3ed [id=7c7fcd265e624d7fab74661db505d9c3]). Expected version 1 got 0
I20251025 07:52:41.336424 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525]). Expected version 1 got 0
I20251025 07:52:41.336884 2314 raft_consensus.cc:493] T c2713942584e4734acbc0775ead772fa P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:41.337009 2314 raft_consensus.cc:515] T c2713942584e4734acbc0775ead772fa P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:41.337579 2314 leader_election.cc:290] T c2713942584e4734acbc0775ead772fa P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:41.338338 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2713942584e4734acbc0775ead772fa" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:41.338563 774 raft_consensus.cc:2468] T c2713942584e4734acbc0775ead772fa P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:41.338976 847 leader_election.cc:304] T c2713942584e4734acbc0775ead772fa P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:41.339279 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2713942584e4734acbc0775ead772fa" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:41.339526 2106 raft_consensus.cc:2804] T c2713942584e4734acbc0775ead772fa P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:41.339635 2106 raft_consensus.cc:493] T c2713942584e4734acbc0775ead772fa P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:41.339730 2106 raft_consensus.cc:3060] T c2713942584e4734acbc0775ead772fa P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:41.339763 850 leader_election.cc:343] T c2713942584e4734acbc0775ead772fa P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:41.336714 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 36262ab8c6964dd684c01b4db6219a4f (table default.Table_76158f8cf82e4cdb9a90580e46aaea48 [id=f249f30497bc450c887d659721469de8]). Expected version 1 got 0
W20251025 07:52:41.340106 983 consensus_peers.cc:597] T fab67b56eaa44b54886c17181d8c52b2 P db42be7ad3c741a0aa427f27e245993b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fab67b56eaa44b54886c17181d8c52b2. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:41.340656 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 2c50ec95e48e49b59a76fac4fc862d3d (table default.Table_32b54e82cbc84d39a0020826f8737d62 [id=c28b0b6acd61490caecfdaab803a9d92]). Expected version 2 got 0
I20251025 07:52:41.342216 1167 tablet_bootstrap.cc:492] T 30418858bf95438686bdfe64ac733144 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:41.342340 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 509c94f2a422493ea4390ecee576db72 (table default.Table_3f081b7bb4e1420c8121ea4115439f3d [id=696a1285fbfe4038a8b1b38708c8ea49]). Expected version 1 got 0
I20251025 07:52:41.342521 1167 ts_tablet_manager.cc:1403] T 30418858bf95438686bdfe64ac733144 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.045s user 0.006s sys 0.000s
I20251025 07:52:41.342527 2106 raft_consensus.cc:515] T c2713942584e4734acbc0775ead772fa P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:41.342725 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet ab42909afc524f2fa3c1041495d13a59 (table default.Table_7eb8424d5192485da1a2d67bfdb2bbbf [id=40c29df99590408cbd5c583d4fc741eb]). Expected version 2 got 0
I20251025 07:52:41.343000 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e]). Expected version 1 got 0
I20251025 07:52:41.343216 2106 leader_election.cc:290] T c2713942584e4734acbc0775ead772fa P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:41.343852 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2713942584e4734acbc0775ead772fa" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:41.344120 1167 raft_consensus.cc:359] T 30418858bf95438686bdfe64ac733144 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
W20251025 07:52:41.344374 850 leader_election.cc:343] T c2713942584e4734acbc0775ead772fa P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:41.344470 1167 raft_consensus.cc:385] T 30418858bf95438686bdfe64ac733144 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:41.344630 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2713942584e4734acbc0775ead772fa" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:41.344832 774 raft_consensus.cc:3060] T c2713942584e4734acbc0775ead772fa P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:41.347641 774 raft_consensus.cc:2468] T c2713942584e4734acbc0775ead772fa P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:41.348042 847 leader_election.cc:304] T c2713942584e4734acbc0775ead772fa P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:41.344623 1167 raft_consensus.cc:740] T 30418858bf95438686bdfe64ac733144 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:41.348707 1167 consensus_queue.cc:260] T 30418858bf95438686bdfe64ac733144 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:41.349323 1167 ts_tablet_manager.cc:1434] T 30418858bf95438686bdfe64ac733144 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251025 07:52:41.349757 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet bc2d16b33ca8426abbd7bd38339c9f1d (table default.Table_4ea985df4c5b415c9570c64506a98cc4 [id=98d2b7797e33485e8872054aa2eb8ec9]). Expected version 1 got 0
I20251025 07:52:41.350114 452 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet ef1854d35a3047e3a9fbe0166ab1295a (table default.Table_088aad2a46b041bfa29ed8e6929760b1 [id=834669c54dae449790818684fd8659bc]). Expected version 2 got 0
I20251025 07:52:41.350572 453 catalog_manager.cc:6194] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: default.Table_350383f0e98947eba77186e74ad88781 [id=96b0e47c666c4d8f982741aea7b17ae7] alter complete (version 2)
I20251025 07:52:41.352010 2169 ts_tablet_manager.cc:1929] T abf94c32ff2945119b710905c075d63c P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:41.352279 2169 log.cc:1199] T abf94c32ff2945119b710905c075d63c P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/abf94c32ff2945119b710905c075d63c
I20251025 07:52:41.353374 2169 ts_tablet_manager.cc:1950] T abf94c32ff2945119b710905c075d63c P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:41.357452 2106 raft_consensus.cc:2804] T c2713942584e4734acbc0775ead772fa P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:41.357597 2106 raft_consensus.cc:697] T c2713942584e4734acbc0775ead772fa P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:41.357831 2106 consensus_queue.cc:237] T c2713942584e4734acbc0775ead772fa P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:41.358952 2159 log.cc:927] Time spent T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Append to log took a long time: real 0.053s user 0.003s sys 0.000s
I20251025 07:52:41.350615 447 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet fa998b9ca6724c4cb1eb68a3eecb0b11
I20251025 07:52:41.362040 447 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:47610 (request call id 55) took 3117 ms. Trace:
I20251025 07:52:41.362298 1169 tablet_bootstrap.cc:492] T 80fe3255ed614270bd8352a2853fbc43 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:41.362583 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 85) took 2645 ms. Trace:
I20251025 07:52:41.362190 447 rpcz_store.cc:276] 1025 07:52:38.244318 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:40.481623 (+2237305us) service_pool.cc:225] Handling call
1025 07:52:40.482100 (+ 477us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:40.482281 (+ 181us) catalog_manager.cc:2815] Modifying in-memory table state
1025 07:52:40.482351 (+ 70us) catalog_manager.cc:2824] Locking tablets
1025 07:52:40.497550 (+ 15199us) catalog_manager.cc:2838] Removing table and tablets from system table
1025 07:52:40.505931 (+ 8381us) catalog_manager.cc:2856] Removing table from by-name map
1025 07:52:40.519694 (+ 13763us) catalog_manager.cc:2872] Committing in-memory state
1025 07:52:41.350542 (+830848us) catalog_manager.cc:2876] Aborting table tasks
1025 07:52:41.361998 (+ 11456us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:40.498665 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:40.498771 (+ 106us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:40.498777 (+ 6us) write_op.cc:435] Acquired schema lock
1025 07:52:40.498779 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:40.498881 (+ 102us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:40.498894 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:40.498899 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:40.498999 (+ 100us) tablet.cc:699] Row locks acquired
1025 07:52:40.499001 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:40.499034 (+ 33us) write_op.cc:270] Start()
1025 07:52:40.499061 (+ 27us) write_op.cc:276] Timestamp: P: 1761378760499016 usec, L: 0
1025 07:52:40.499065 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:40.499250 (+ 185us) log.cc:844] Serialized 759 byte log entry
1025 07:52:40.504252 (+ 5002us) op_driver.cc:464] REPLICATION: finished
1025 07:52:40.504377 (+ 125us) write_op.cc:301] APPLY: starting
1025 07:52:40.504391 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:40.504453 (+ 62us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:40.504488 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:40.504557 (+ 69us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:40.504594 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1025 07:52:40.504602 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:40.504670 (+ 68us) log.cc:844] Serialized 29 byte log entry
1025 07:52:40.504747 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:40.504768 (+ 21us) write_op.cc:454] Released schema lock
1025 07:52:40.504774 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:40.504851 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:40.504858 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":71,"apply.run_cpu_time_us":462,"apply.run_wall_time_us":462,"num_ops":2,"prepare.queue_time_us":321,"prepare.run_cpu_time_us":685,"prepare.run_wall_time_us":684,"raft.queue_time_us":341,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":193,"replication_time_us":5154}]]}
I20251025 07:52:41.362664 2169 rpcz_store.cc:276] 1025 07:52:38.719649 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.719692 (+ 43us) service_pool.cc:225] Handling call
1025 07:52:41.275792 (+2556100us) ts_tablet_manager.cc:1151] Deleting tablet abf94c32ff2945119b710905c075d63c
1025 07:52:41.275802 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.332261 (+ 56459us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.351645 (+ 19384us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.362562 (+ 10917us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2555965}
I20251025 07:52:41.363427 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 86) took 2643 ms. Trace:
I20251025 07:52:41.363544 2169 rpcz_store.cc:276] 1025 07:52:38.719987 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.720043 (+ 56us) service_pool.cc:225] Handling call
1025 07:52:41.363223 (+2643180us) ts_tablet_manager.cc:1151] Deleting tablet 1912e59548874e0e8e989050ec0064c5
1025 07:52:41.363234 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.363388 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2643016}
I20251025 07:52:41.364002 1169 tablet_bootstrap.cc:654] T 80fe3255ed614270bd8352a2853fbc43 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:41.364215 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 87) took 2643 ms. Trace:
I20251025 07:52:41.364485 2146 ts_tablet_manager.cc:1929] T 35ba9d23262248cda6fdf03c51be11a4 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:41.364652 2146 log.cc:1199] T 35ba9d23262248cda6fdf03c51be11a4 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/35ba9d23262248cda6fdf03c51be11a4
I20251025 07:52:41.364341 2169 rpcz_store.cc:276] 1025 07:52:38.720399 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.720482 (+ 83us) service_pool.cc:225] Handling call
1025 07:52:41.364031 (+2643549us) ts_tablet_manager.cc:1151] Deleting tablet 2b594d4718344aafbdaec34040695589
1025 07:52:41.364039 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.364168 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2643369}
I20251025 07:52:41.364980 2146 ts_tablet_manager.cc:1950] T 35ba9d23262248cda6fdf03c51be11a4 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:41.365410 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 88) took 2644 ms. Trace:
I20251025 07:52:41.365530 2169 rpcz_store.cc:276] 1025 07:52:38.720980 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.721040 (+ 60us) service_pool.cc:225] Handling call
1025 07:52:41.365269 (+2644229us) ts_tablet_manager.cc:1151] Deleting tablet ee1c8ea694284d80a30f9c05be4cbf49
1025 07:52:41.365276 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.365395 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2644032}
I20251025 07:52:41.365744 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 106) took 2896 ms. Trace:
I20251025 07:52:41.365820 2146 rpcz_store.cc:276] 1025 07:52:38.469268 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.469540 (+ 272us) service_pool.cc:225] Handling call
1025 07:52:41.262561 (+2793021us) ts_tablet_manager.cc:1151] Deleting tablet 35ba9d23262248cda6fdf03c51be11a4
1025 07:52:41.262574 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.314095 (+ 51521us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.364369 (+ 50274us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.365724 (+ 1355us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2789896}
I20251025 07:52:41.366585 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet abf94c32ff2945119b710905c075d63c (table default.Table_4b63840ca3ac4296b23104bf46030bb4 [id=0010b5f16dd14d64b8c93d941803cc75]) successfully deleted
I20251025 07:52:41.366655 745 tablet_service.cc:1552] Processing DeleteTablet for tablet fa998b9ca6724c4cb1eb68a3eecb0b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:58412
W20251025 07:52:41.366884 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 1912e59548874e0e8e989050ec0064c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1912e59548874e0e8e989050ec0064c5 already in progress: creating tablet
I20251025 07:52:41.366740 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet fa998b9ca6724c4cb1eb68a3eecb0b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.366115 2169 tablet_replica.cc:333] T 7f29b64704e147aa9b0b5f6cda860f09 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
W20251025 07:52:41.367153 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 2b594d4718344aafbdaec34040695589 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b594d4718344aafbdaec34040695589 already in progress: creating tablet
W20251025 07:52:41.367370 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet ee1c8ea694284d80a30f9c05be4cbf49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee1c8ea694284d80a30f9c05be4cbf49 already in progress: creating tablet
I20251025 07:52:41.367812 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 35ba9d23262248cda6fdf03c51be11a4 (table default.Table_072e0092adb64ef0ac6834dc18688ccf [id=1d4cbe4a4f38445297ceeebb8ffdaa8e]) successfully deleted
I20251025 07:52:41.369406 875 tablet_service.cc:1552] Processing DeleteTablet for tablet fa998b9ca6724c4cb1eb68a3eecb0b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:59616
W20251025 07:52:41.370471 712 consensus_peers.cc:597] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: be5ab0db6ac34c2c9a370445e251eb46. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:41.370599 2169 raft_consensus.cc:2243] T 7f29b64704e147aa9b0b5f6cda860f09 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:41.371243 2169 raft_consensus.cc:2272] T 7f29b64704e147aa9b0b5f6cda860f09 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:41.373070 2169 ts_tablet_manager.cc:1916] T 7f29b64704e147aa9b0b5f6cda860f09 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:41.373277 1166 tablet_bootstrap.cc:492] T aaef24ccdfba45fe8204327723df834e P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:41.373440 1166 ts_tablet_manager.cc:1403] T aaef24ccdfba45fe8204327723df834e P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.079s user 0.001s sys 0.003s
I20251025 07:52:41.374248 1166 raft_consensus.cc:359] T aaef24ccdfba45fe8204327723df834e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.374452 1166 raft_consensus.cc:385] T aaef24ccdfba45fe8204327723df834e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:41.374540 1166 raft_consensus.cc:740] T aaef24ccdfba45fe8204327723df834e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:41.374814 1166 consensus_queue.cc:260] T aaef24ccdfba45fe8204327723df834e P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.375337 1166 ts_tablet_manager.cc:1434] T aaef24ccdfba45fe8204327723df834e P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:41.375649 1166 tablet_bootstrap.cc:492] T a4e4b99be0434baaa75d988281797846 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:41.379156 1166 tablet_bootstrap.cc:654] T a4e4b99be0434baaa75d988281797846 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:41.381794 2166 ts_tablet_manager.cc:1929] T 1d261e432cf5403ea162129706bc4d4d P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:41.381958 2166 log.cc:1199] T 1d261e432cf5403ea162129706bc4d4d P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/1d261e432cf5403ea162129706bc4d4d
I20251025 07:52:41.382347 2166 ts_tablet_manager.cc:1950] T 1d261e432cf5403ea162129706bc4d4d P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:41.384908 1167 tablet_bootstrap.cc:492] T c9f7ba3709c745d4a625449d6e1c8ba1 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:41.386373 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 107) took 2916 ms. Trace:
I20251025 07:52:41.387176 1167 tablet_bootstrap.cc:654] T c9f7ba3709c745d4a625449d6e1c8ba1 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:41.387285 2146 rpcz_store.cc:276] 1025 07:52:38.470383 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.470666 (+ 283us) service_pool.cc:225] Handling call
1025 07:52:41.386253 (+2915587us) ts_tablet_manager.cc:1151] Deleting tablet 2209089667dd4937abd96615f6929449
1025 07:52:41.386266 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.386352 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2913771}
W20251025 07:52:41.387828 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 2209089667dd4937abd96615f6929449 because the tablet was not found. No further retry: Not found: Tablet not found: 2209089667dd4937abd96615f6929449
I20251025 07:52:41.388419 2146 tablet_replica.cc:333] T 74e2d78c024849b0af5c0bde1f48c36f P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:41.388721 2146 raft_consensus.cc:2243] T 74e2d78c024849b0af5c0bde1f48c36f P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:41.390882 2146 raft_consensus.cc:2272] T 74e2d78c024849b0af5c0bde1f48c36f P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:41.392000 2146 ts_tablet_manager.cc:1916] T 74e2d78c024849b0af5c0bde1f48c36f P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:41.396076 453 catalog_manager.cc:6194] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: default.Table_99fd36369f9a4e0fb14f234c98635203 [id=c8059ab6cec146efa631d06799d969a9] alter complete (version 1)
I20251025 07:52:41.392999 1166 tablet_bootstrap.cc:492] T a4e4b99be0434baaa75d988281797846 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:41.398716 1166 ts_tablet_manager.cc:1403] T a4e4b99be0434baaa75d988281797846 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.001s
I20251025 07:52:41.402611 2314 raft_consensus.cc:493] T 80fe3255ed614270bd8352a2853fbc43 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:41.402882 2314 raft_consensus.cc:515] T 80fe3255ed614270bd8352a2853fbc43 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.403568 2314 leader_election.cc:290] T 80fe3255ed614270bd8352a2853fbc43 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:41.404577 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "80fe3255ed614270bd8352a2853fbc43" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:41.405123 850 leader_election.cc:343] T 80fe3255ed614270bd8352a2853fbc43 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:41.405411 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "80fe3255ed614270bd8352a2853fbc43" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:41.405611 774 raft_consensus.cc:2468] T 80fe3255ed614270bd8352a2853fbc43 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:41.405969 847 leader_election.cc:304] T 80fe3255ed614270bd8352a2853fbc43 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:41.406385 2314 raft_consensus.cc:2804] T 80fe3255ed614270bd8352a2853fbc43 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:41.406459 2314 raft_consensus.cc:493] T 80fe3255ed614270bd8352a2853fbc43 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:41.406520 2314 raft_consensus.cc:3060] T 80fe3255ed614270bd8352a2853fbc43 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:41.408767 2169 ts_tablet_manager.cc:1929] T 7f29b64704e147aa9b0b5f6cda860f09 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:41.409126 2314 raft_consensus.cc:515] T 80fe3255ed614270bd8352a2853fbc43 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.409466 2169 log.cc:1199] T 7f29b64704e147aa9b0b5f6cda860f09 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/7f29b64704e147aa9b0b5f6cda860f09
I20251025 07:52:41.410084 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "80fe3255ed614270bd8352a2853fbc43" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:41.410283 773 raft_consensus.cc:3060] T 80fe3255ed614270bd8352a2853fbc43 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:41.411705 2169 ts_tablet_manager.cc:1950] T 7f29b64704e147aa9b0b5f6cda860f09 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:41.413194 773 raft_consensus.cc:2468] T 80fe3255ed614270bd8352a2853fbc43 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:41.413733 847 leader_election.cc:304] T 80fe3255ed614270bd8352a2853fbc43 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:41.413995 2105 raft_consensus.cc:2804] T 80fe3255ed614270bd8352a2853fbc43 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:41.414103 2105 raft_consensus.cc:697] T 80fe3255ed614270bd8352a2853fbc43 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:41.414383 2105 consensus_queue.cc:237] T 80fe3255ed614270bd8352a2853fbc43 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
W20251025 07:52:41.415373 448 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 872cb7512d104827b1f1f186d33a4a44 (table default.Table_d5c65dce8b1b4214ba519d964012bfc0 [id=d4be274b36a845a99a4b96b8846248fd], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 872cb7512d104827b1f1f186d33a4a44 consensus state has no leader
I20251025 07:52:41.415714 875 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f0c4f0e8f014e59be9e46d2afe0be9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.415961 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e6544202d4f489d82f22d6073ef240d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.416064 875 tablet_service.cc:1552] Processing DeleteTablet for tablet ffe7c8e1d6c44872b7b65e512764a56a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.416257 875 tablet_service.cc:1552] Processing DeleteTablet for tablet 80fe3255ed614270bd8352a2853fbc43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.416431 875 tablet_service.cc:1552] Processing DeleteTablet for tablet ce1ebb1ec14b4801930a72765bca1a7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.416604 880 tablet_service.cc:1552] Processing DeleteTablet for tablet ebc824ec4c60411dab1a2ea4a385148e with delete_type TABLET_DATA_DELETED (replaced by tablet 29dbaf9149134f548a974a5779f7da67) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.416693 875 tablet_service.cc:1552] Processing DeleteTablet for tablet 050a5d3b5f934dd19bf2e221f2fe0780 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.416762 880 tablet_service.cc:1552] Processing DeleteTablet for tablet 04449cc97f404a19b6a1b20dea074f95 with delete_type TABLET_DATA_DELETED (replaced by tablet f70091024c034c118fb86578db19a31c) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.416908 880 tablet_service.cc:1552] Processing DeleteTablet for tablet f43ef51ee45d47a99ec732b49d50d3b4 with delete_type TABLET_DATA_DELETED (replaced by tablet e429b94326cc4827a03968fd92de38e0) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.416967 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 80ee75935d7e461d9d4066fac9819bef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.417158 878 tablet_service.cc:1552] Processing DeleteTablet for tablet dda0b26e4f1a44928bc4b84f333901c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.416921 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 89) took 2695 ms. Trace:
I20251025 07:52:41.417361 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 91728987303b41fd840eb56f3a311c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.417534 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b6383c99e9e4c90a374da9ccd830a6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:59616
W20251025 07:52:41.418489 710 consensus_peers.cc:597] T 4f25c9caa05b4da4bd56a8a21fde423c P 23eeffd3de254ce18db5febfde971c6b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f25c9caa05b4da4bd56a8a21fde423c. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:41.418964 885 tablet_service.cc:1552] Processing DeleteTablet for tablet d37758f606344127b3db74c87babe69c with delete_type TABLET_DATA_DELETED (replaced by tablet 07898860aa594a05a211fc89768cb76d) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.417338 2169 rpcz_store.cc:276] 1025 07:52:38.722124 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.722338 (+ 214us) service_pool.cc:225] Handling call
1025 07:52:41.365995 (+2643657us) ts_tablet_manager.cc:1151] Deleting tablet 7f29b64704e147aa9b0b5f6cda860f09
1025 07:52:41.366051 (+ 56us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.381385 (+ 15334us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.408284 (+ 26899us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.416735 (+ 8451us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2643484}
I20251025 07:52:41.419431 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 0f5f6ca6c3cd4d369690e425387a4344 with delete_type TABLET_DATA_DELETED (replaced by tablet 22fc90c704f64d2f9f4b8c54dee45576) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.419631 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b940fa5ae26427dafbcfd26372754b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.419901 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 7f29b64704e147aa9b0b5f6cda860f09 (table default.Table_2253f6700c2f4a3a9847023ba98b80ea [id=17c206c8e24647ab9f483d9944dd3191]) successfully deleted
I20251025 07:52:41.420585 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "80fe3255ed614270bd8352a2853fbc43" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:41.421077 850 leader_election.cc:343] T 80fe3255ed614270bd8352a2853fbc43 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:41.421790 880 tablet_service.cc:1552] Processing DeleteTablet for tablet b3a0af893a8947848e12093d0a7f0229 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.421976 878 tablet_service.cc:1552] Processing DeleteTablet for tablet f97acc9a638b4d5a968dcc3412a384bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.421973 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 90) took 2699 ms. Trace:
I20251025 07:52:41.422147 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 410ef150a7014c6c867a021b544a2f4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.422297 894 tablet_service.cc:1552] Processing DeleteTablet for tablet aaef24ccdfba45fe8204327723df834e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.422492 885 tablet_service.cc:1552] Processing DeleteTablet for tablet d9bfd7c0c3ad4e5b98b6708793d61d41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.422233 2169 rpcz_store.cc:276] 1025 07:52:38.723059 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.723251 (+ 192us) service_pool.cc:225] Handling call
1025 07:52:41.421574 (+2698323us) ts_tablet_manager.cc:1151] Deleting tablet 6dcab53a020847a1aa27930265ba0205
1025 07:52:41.421590 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.421867 (+ 277us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2698061}
I20251025 07:52:41.423472 880 tablet_service.cc:1552] Processing DeleteTablet for tablet f656945454fe4921ab819c772a089eb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.423583 2169 tablet_replica.cc:333] T 4f25c9caa05b4da4bd56a8a21fde423c P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:41.423668 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 48ff4ce7846b4e56a37eee9bcb17bd5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.423732 875 tablet_service.cc:1552] Processing DeleteTablet for tablet e54fe7cbcffa40c4b04144b47bee2b48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.424042 875 tablet_service.cc:1552] Processing DeleteTablet for tablet b947ecc352e54f4caa73c6ca97372256 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.424117 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f26667f053a4f16b5da330c0b5e40f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 34315f64b89148efbf1f20a43d5c5efb) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.424047 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 26c849d376d143c6bef0c8e0e1d7d03a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.424448 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 88eb51447e0f4080baa49dd902deb24f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.424463 2169 raft_consensus.cc:2243] T 4f25c9caa05b4da4bd56a8a21fde423c P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
W20251025 07:52:41.424784 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 6dcab53a020847a1aa27930265ba0205 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6dcab53a020847a1aa27930265ba0205 already in progress: creating tablet
I20251025 07:52:41.424894 885 tablet_service.cc:1552] Processing DeleteTablet for tablet f661d5909657455581dcb5e6eadd3c0b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.425252 2169 raft_consensus.cc:2272] T 4f25c9caa05b4da4bd56a8a21fde423c P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:41.426905 894 tablet_service.cc:1552] Processing DeleteTablet for tablet b94634ac5077439fadfbbce569f60e8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.427625 2316 raft_consensus.cc:493] T e9f7cd765d614d59a7c190b744b0844d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:41.427716 878 tablet_service.cc:1552] Processing DeleteTablet for tablet c11f674385184ae6b235f1db8bf1fb91 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.427822 2316 raft_consensus.cc:515] T e9f7cd765d614d59a7c190b744b0844d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.428450 2316 leader_election.cc:290] T e9f7cd765d614d59a7c190b744b0844d P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:41.429059 894 tablet_service.cc:1552] Processing DeleteTablet for tablet d77704b4fe6c4456a2ba40ffec0c60fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.429141 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9f7cd765d614d59a7c190b744b0844d" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:41.429368 775 raft_consensus.cc:2393] T e9f7cd765d614d59a7c190b744b0844d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:41.429755 894 tablet_service.cc:1552] Processing DeleteTablet for tablet e4da89f863cc43f3bf6df908bb9dc6bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.427551 2169 ts_tablet_manager.cc:1916] T 4f25c9caa05b4da4bd56a8a21fde423c P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:41.430317 878 tablet_service.cc:1552] Processing DeleteTablet for tablet fd265cdbf6a8440393a2b5d8240d1872 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
W20251025 07:52:41.430322 448 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet f70091024c034c118fb86578db19a31c (table default.Table_332b308220f64c6dbb5f29711d8463b3 [id=e041946c093b4c789ce175c736cc230b], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet f70091024c034c118fb86578db19a31c consensus state has no leader
I20251025 07:52:41.430547 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f7c15529be54a11961f385a953b0b03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.431097 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 72dc084f8efc45478771c247e07a16f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.431470 878 tablet_service.cc:1552] Processing DeleteTablet for tablet be5ab0db6ac34c2c9a370445e251eb46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.431844 2314 leader_election.cc:290] T 80fe3255ed614270bd8352a2853fbc43 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:41.432160 878 tablet_service.cc:1552] Processing DeleteTablet for tablet e47c4f741f354a19b49777bd06e54764 with delete_type TABLET_DATA_DELETED (replaced by tablet 755f0d6c87044e20a8020f8410940123) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.432832 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 7f29b64704e147aa9b0b5f6cda860f09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.433097 454 catalog_manager.cc:6194] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: default.Table_ca479cb5c1184c5391b7e1c9d8a08937 [id=d82b3e1317434cf08a0d51e8a9427224] alter complete (version 3)
I20251025 07:52:41.433357 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 95efd553493e4966bc48a01cac6e1e10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.433818 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 896bd7870d87408e8adfdb7a78998111 with delete_type TABLET_DATA_DELETED (replaced by tablet e8402fd33ba842e981c6c68787c7c548) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.434324 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 27651312cc2e42a8bfcd245f5d301679 with delete_type TABLET_DATA_DELETED (replaced by tablet a4e4b99be0434baaa75d988281797846) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.436275 878 tablet_service.cc:1552] Processing DeleteTablet for tablet fab67b56eaa44b54886c17181d8c52b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.436785 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e098c9c4613459295b6043048cba587 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.437275 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f25c9caa05b4da4bd56a8a21fde423c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.437737 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 887e7c7a5d0e4aaf88f26246306df19c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.438247 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 6dcab53a020847a1aa27930265ba0205 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.438817 878 tablet_service.cc:1552] Processing DeleteTablet for tablet abf94c32ff2945119b710905c075d63c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.439318 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 1912e59548874e0e8e989050ec0064c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.439917 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 1b5ac7e845a24b2eb5b2429d47361232 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
W20251025 07:52:41.440593 982 consensus_peers.cc:597] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b -> Peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Couldn't send request to peer 23eeffd3de254ce18db5febfde971c6b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b947ecc352e54f4caa73c6ca97372256. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:41.441044 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9f7cd765d614d59a7c190b744b0844d" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:41.443621 983 leader_election.cc:304] T e9f7cd765d614d59a7c190b744b0844d P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:41.444067 2348 raft_consensus.cc:3060] T e9f7cd765d614d59a7c190b744b0844d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:41.444115 982 consensus_peers.cc:597] T 551d07a9242740dcb9366bb2d707a4a8 P db42be7ad3c741a0aa427f27e245993b -> Peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Couldn't send request to peer 23eeffd3de254ce18db5febfde971c6b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 551d07a9242740dcb9366bb2d707a4a8. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:41.446673 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet dda0b26e4f1a44928bc4b84f333901c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.446898 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 80ee75935d7e461d9d4066fac9819bef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.447055 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e6544202d4f489d82f22d6073ef240d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.447296 1010 tablet_service.cc:1552] Processing DeleteTablet for tablet ffe7c8e1d6c44872b7b65e512764a56a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.447337 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet 80fe3255ed614270bd8352a2853fbc43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.447508 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b6383c99e9e4c90a374da9ccd830a6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.447535 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f0c4f0e8f014e59be9e46d2afe0be9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.447700 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet db99a1e68daf4e948a3d66ad276cc5f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.447691 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 716fb61312e94a4d9eb8ab8336ca140c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.447919 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet bef0ef23d90d480d9c99aef21d7271fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.448959 2348 raft_consensus.cc:2749] T e9f7cd765d614d59a7c190b744b0844d P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:41.449935 1838 raft_consensus.cc:493] T 4c155aa7fa5b4d68bbab1918b8432372 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 07:52:41.450155 439 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 872cb7512d104827b1f1f186d33a4a44 (table default.Table_d5c65dce8b1b4214ba519d964012bfc0 [id=d4be274b36a845a99a4b96b8846248fd], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 872cb7512d104827b1f1f186d33a4a44 consensus state has no leader
I20251025 07:52:41.450111 1838 raft_consensus.cc:515] T 4c155aa7fa5b4d68bbab1918b8432372 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.450740 1838 leader_election.cc:290] T 4c155aa7fa5b4d68bbab1918b8432372 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:41.451481 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c155aa7fa5b4d68bbab1918b8432372" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:41.451858 774 raft_consensus.cc:2468] T 4c155aa7fa5b4d68bbab1918b8432372 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:41.451965 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 91728987303b41fd840eb56f3a311c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.452117 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet aaef24ccdfba45fe8204327723df834e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.452183 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 050a5d3b5f934dd19bf2e221f2fe0780 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.452360 847 leader_election.cc:304] T 4c155aa7fa5b4d68bbab1918b8432372 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:41.452486 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 0f5f6ca6c3cd4d369690e425387a4344 with delete_type TABLET_DATA_DELETED (replaced by tablet 22fc90c704f64d2f9f4b8c54dee45576) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.452543 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet ce1ebb1ec14b4801930a72765bca1a7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.452710 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet ebc824ec4c60411dab1a2ea4a385148e with delete_type TABLET_DATA_DELETED (replaced by tablet 29dbaf9149134f548a974a5779f7da67) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.452895 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet d9bfd7c0c3ad4e5b98b6708793d61d41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.452884 2314 raft_consensus.cc:2804] T 4c155aa7fa5b4d68bbab1918b8432372 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:41.453262 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 04449cc97f404a19b6a1b20dea074f95 with delete_type TABLET_DATA_DELETED (replaced by tablet f70091024c034c118fb86578db19a31c) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.453104 2314 raft_consensus.cc:493] T 4c155aa7fa5b4d68bbab1918b8432372 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:41.453482 2314 raft_consensus.cc:3060] T 4c155aa7fa5b4d68bbab1918b8432372 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:41.453962 2422 raft_consensus.cc:493] T ab42909afc524f2fa3c1041495d13a59 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:41.454075 2422 raft_consensus.cc:515] T ab42909afc524f2fa3c1041495d13a59 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:41.454648 2422 leader_election.cc:290] T ab42909afc524f2fa3c1041495d13a59 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:41.455404 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ab42909afc524f2fa3c1041495d13a59" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:41.455641 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ab42909afc524f2fa3c1041495d13a59" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:41.455933 712 leader_election.cc:343] T ab42909afc524f2fa3c1041495d13a59 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:41.456089 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c155aa7fa5b4d68bbab1918b8432372" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:41.456197 710 leader_election.cc:343] T ab42909afc524f2fa3c1041495d13a59 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:41.456317 710 leader_election.cc:304] T ab42909afc524f2fa3c1041495d13a59 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:41.456705 850 leader_election.cc:343] T 4c155aa7fa5b4d68bbab1918b8432372 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:41.457046 2124 raft_consensus.cc:2749] T ab42909afc524f2fa3c1041495d13a59 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:41.457703 983 consensus_peers.cc:597] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b947ecc352e54f4caa73c6ca97372256. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:41.458570 878 tablet_service.cc:1552] Processing DeleteTablet for tablet db99a1e68daf4e948a3d66ad276cc5f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.459864 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet f43ef51ee45d47a99ec732b49d50d3b4 with delete_type TABLET_DATA_DELETED (replaced by tablet e429b94326cc4827a03968fd92de38e0) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.460263 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet d37758f606344127b3db74c87babe69c with delete_type TABLET_DATA_DELETED (replaced by tablet 07898860aa594a05a211fc89768cb76d) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.460574 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b940fa5ae26427dafbcfd26372754b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.460886 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet b3a0af893a8947848e12093d0a7f0229 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.461207 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet f661d5909657455581dcb5e6eadd3c0b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.461519 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 88eb51447e0f4080baa49dd902deb24f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.461803 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f26667f053a4f16b5da330c0b5e40f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 34315f64b89148efbf1f20a43d5c5efb) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.461045 2314 raft_consensus.cc:515] T 4c155aa7fa5b4d68bbab1918b8432372 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.462231 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet f97acc9a638b4d5a968dcc3412a384bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.462556 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 410ef150a7014c6c867a021b544a2f4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.462857 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 26c849d376d143c6bef0c8e0e1d7d03a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.463178 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 48ff4ce7846b4e56a37eee9bcb17bd5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.463676 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet b947ecc352e54f4caa73c6ca97372256 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.463928 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet f656945454fe4921ab819c772a089eb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.464273 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet e54fe7cbcffa40c4b04144b47bee2b48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.464591 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet b94634ac5077439fadfbbce569f60e8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.464939 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 92aa605fabbc4257a6a204c087e7cc7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.465337 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b28215f61574cf48c447dd01955ac77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.465728 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c155aa7fa5b4d68bbab1918b8432372" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:41.465910 774 raft_consensus.cc:3060] T 4c155aa7fa5b4d68bbab1918b8432372 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:41.466694 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c76d1b509f04ab1a10b83cff675c11e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.467161 878 tablet_service.cc:1552] Processing DeleteTablet for tablet b772025852794a49b072f37cccbd3291 with delete_type TABLET_DATA_DELETED (replaced by tablet f97acc9a638b4d5a968dcc3412a384bc) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.467566 878 tablet_service.cc:1552] Processing DeleteTablet for tablet fa77085e81c7440d9e255c44d1e80c26 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.467967 878 tablet_service.cc:1552] Processing DeleteTablet for tablet a6ddefa1738f4ff98586900c344359d2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f0c4f0e8f014e59be9e46d2afe0be9d) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.468396 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b21b7da1e2f47829c3f6cdaee689e8a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.468849 878 tablet_service.cc:1552] Processing DeleteTablet for tablet f0bdaa8cfd764f1b83f47881353d26b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.468999 774 raft_consensus.cc:2468] T 4c155aa7fa5b4d68bbab1918b8432372 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:41.469379 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 8956ad28b35d45f28c683ac055bba441 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.469659 847 leader_election.cc:304] T 4c155aa7fa5b4d68bbab1918b8432372 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:41.469949 1838 raft_consensus.cc:2804] T 4c155aa7fa5b4d68bbab1918b8432372 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:41.471477 878 tablet_service.cc:1552] Processing DeleteTablet for tablet cc407c7eb12e4eadb41fb21385611c94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.471004 2314 leader_election.cc:290] T 4c155aa7fa5b4d68bbab1918b8432372 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:41.473142 453 catalog_manager.cc:6194] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: default.Table_10e57b42f97544208675708b1e028846 [id=f5c1be203e2b4831a025871cf8a2ef17] alter complete (version 2)
I20251025 07:52:41.475055 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 2784bbfb97d5466f9ec856add7d0b29a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.475728 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d261e432cf5403ea162129706bc4d4d with delete_type TABLET_DATA_DELETED (replaced by tablet e54fe7cbcffa40c4b04144b47bee2b48) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.476593 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 467324e2432f4caca148951329caa292 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.477749 2169 ts_tablet_manager.cc:1929] T 4f25c9caa05b4da4bd56a8a21fde423c P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:41.478120 2169 log.cc:1199] T 4f25c9caa05b4da4bd56a8a21fde423c P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/4f25c9caa05b4da4bd56a8a21fde423c
I20251025 07:52:41.478896 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c76d1b509f04ab1a10b83cff675c11e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.479817 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet b772025852794a49b072f37cccbd3291 with delete_type TABLET_DATA_DELETED (replaced by tablet f97acc9a638b4d5a968dcc3412a384bc) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.480012 2169 ts_tablet_manager.cc:1950] T 4f25c9caa05b4da4bd56a8a21fde423c P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:41.480487 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet a6ddefa1738f4ff98586900c344359d2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f0c4f0e8f014e59be9e46d2afe0be9d) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.480886 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b6383c99e9e4c90a374da9ccd830a6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.481000 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 80fe3255ed614270bd8352a2853fbc43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.481365 754 tablet_service.cc:1552] Processing DeleteTablet for tablet ebc824ec4c60411dab1a2ea4a385148e with delete_type TABLET_DATA_DELETED (replaced by tablet 29dbaf9149134f548a974a5779f7da67) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.481828 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 91728987303b41fd840eb56f3a311c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.482167 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 0f5f6ca6c3cd4d369690e425387a4344 with delete_type TABLET_DATA_DELETED (replaced by tablet 22fc90c704f64d2f9f4b8c54dee45576) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.482491 754 tablet_service.cc:1552] Processing DeleteTablet for tablet aaef24ccdfba45fe8204327723df834e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.483220 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c155aa7fa5b4d68bbab1918b8432372" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:41.483340 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 91) took 2759 ms. Trace:
W20251025 07:52:41.483744 850 leader_election.cc:343] T 4c155aa7fa5b4d68bbab1918b8432372 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:41.483675 2169 rpcz_store.cc:276] 1025 07:52:38.724112 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.724249 (+ 137us) service_pool.cc:225] Handling call
1025 07:52:41.423326 (+2699077us) ts_tablet_manager.cc:1151] Deleting tablet 4f25c9caa05b4da4bd56a8a21fde423c
1025 07:52:41.423336 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.446319 (+ 22983us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.477359 (+ 31040us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.483236 (+ 5877us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2698911}
I20251025 07:52:41.484513 745 tablet_service.cc:1552] Processing DeleteTablet for tablet ce1ebb1ec14b4801930a72765bca1a7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.485049 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 92) took 2760 ms. Trace:
I20251025 07:52:41.485327 2169 rpcz_store.cc:276] 1025 07:52:38.742247 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.742377 (+ 130us) service_pool.cc:225] Handling call
1025 07:52:41.484628 (+2742251us) ts_tablet_manager.cc:1151] Deleting tablet cc407c7eb12e4eadb41fb21385611c94
1025 07:52:41.484642 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.485025 (+ 383us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2741974}
I20251025 07:52:41.486511 2422 raft_consensus.cc:493] T 2784bbfb97d5466f9ec856add7d0b29a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db42be7ad3c741a0aa427f27e245993b)
I20251025 07:52:41.486653 2422 raft_consensus.cc:515] T 2784bbfb97d5466f9ec856add7d0b29a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:41.486804 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 93) took 2744 ms. Trace:
I20251025 07:52:41.487306 2422 leader_election.cc:290] T 2784bbfb97d5466f9ec856add7d0b29a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:41.487066 2169 rpcz_store.cc:276] 1025 07:52:38.742714 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.742759 (+ 45us) service_pool.cc:225] Handling call
1025 07:52:41.486329 (+2743570us) ts_tablet_manager.cc:1151] Deleting tablet 1b5ac7e845a24b2eb5b2429d47361232
1025 07:52:41.486511 (+ 182us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.486789 (+ 278us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2743215}
W20251025 07:52:41.488323 710 leader_election.cc:343] T 2784bbfb97d5466f9ec856add7d0b29a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 2784bbfb97d5466f9ec856add7d0b29a
I20251025 07:52:41.488217 2169 tablet_replica.cc:333] T 21e52928580c4762b0ae7152490ef955 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:41.488987 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 050a5d3b5f934dd19bf2e221f2fe0780 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.489051 2169 raft_consensus.cc:2243] T 21e52928580c4762b0ae7152490ef955 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:41.489610 756 tablet_service.cc:1552] Processing DeleteTablet for tablet f43ef51ee45d47a99ec732b49d50d3b4 with delete_type TABLET_DATA_DELETED (replaced by tablet e429b94326cc4827a03968fd92de38e0) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.489998 756 tablet_service.cc:1552] Processing DeleteTablet for tablet d37758f606344127b3db74c87babe69c with delete_type TABLET_DATA_DELETED (replaced by tablet 07898860aa594a05a211fc89768cb76d) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.490188 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b940fa5ae26427dafbcfd26372754b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.490468 745 tablet_service.cc:1552] Processing DeleteTablet for tablet f97acc9a638b4d5a968dcc3412a384bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.490772 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 410ef150a7014c6c867a021b544a2f4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.490710 2169 raft_consensus.cc:2272] T 21e52928580c4762b0ae7152490ef955 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:41.491319 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f26667f053a4f16b5da330c0b5e40f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 34315f64b89148efbf1f20a43d5c5efb) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.491600 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 26c849d376d143c6bef0c8e0e1d7d03a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.491905 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 04449cc97f404a19b6a1b20dea074f95 with delete_type TABLET_DATA_DELETED (replaced by tablet f70091024c034c118fb86578db19a31c) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.492058 745 tablet_service.cc:1552] Processing DeleteTablet for tablet b3a0af893a8947848e12093d0a7f0229 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.492341 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 48ff4ce7846b4e56a37eee9bcb17bd5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.492681 745 tablet_service.cc:1552] Processing DeleteTablet for tablet b947ecc352e54f4caa73c6ca97372256 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.492998 2169 ts_tablet_manager.cc:1916] T 21e52928580c4762b0ae7152490ef955 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:41.493172 745 tablet_service.cc:1552] Processing DeleteTablet for tablet d9bfd7c0c3ad4e5b98b6708793d61d41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.493352 754 tablet_service.cc:1552] Processing DeleteTablet for tablet a4e4b99be0434baaa75d988281797846 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.493647 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 80ee75935d7e461d9d4066fac9819bef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.493937 754 tablet_service.cc:1552] Processing DeleteTablet for tablet b94634ac5077439fadfbbce569f60e8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.494266 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 92aa605fabbc4257a6a204c087e7cc7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.494576 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 50ae9bca6bed4f749d54191ac1543410 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.494897 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 3abfd52327a04ba88625f391a8658fbb with delete_type TABLET_DATA_DELETED (replaced by tablet 04d27d54fdb7483b869408af65ff85c9) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.495234 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 34315f64b89148efbf1f20a43d5c5efb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.495580 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 88eb51447e0f4080baa49dd902deb24f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.495992 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 08da9d2687704725aacd11cd92c6a9f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.496358 754 tablet_service.cc:1552] Processing DeleteTablet for tablet d66ad049d5484228a268b0facb00ecf1 with delete_type TABLET_DATA_DELETED (replaced by tablet 2e098c9c4613459295b6043048cba587) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.496758 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b28215f61574cf48c447dd01955ac77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:58412
W20251025 07:52:41.498721 712 leader_election.cc:343] T 2784bbfb97d5466f9ec856add7d0b29a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 2784bbfb97d5466f9ec856add7d0b29a
I20251025 07:52:41.498870 712 leader_election.cc:304] T 2784bbfb97d5466f9ec856add7d0b29a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:41.499789 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 7fbae8be73bf4e18b5a5c4ec4cdebc12 with delete_type TABLET_DATA_DELETED (replaced by tablet 4c155aa7fa5b4d68bbab1918b8432372) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.499989 2422 raft_consensus.cc:2749] T 2784bbfb97d5466f9ec856add7d0b29a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 07:52:41.504431 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 551d07a9242740dcb9366bb2d707a4a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.504801 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ec421eb820b4f40afcff5a5bf80c508 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.505173 754 tablet_service.cc:1552] Processing DeleteTablet for tablet af743ee344e543a0aa6da3cd7a30e467 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.506276 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 4f25c9caa05b4da4bd56a8a21fde423c (table default.Table_60bfcc345b8649e9bc73fe3d077f46c7 [id=ed06dbbb96a24ab39619f4186ba93191]) successfully deleted
I20251025 07:52:41.506405 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 467324e2432f4caca148951329caa292 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:58412
W20251025 07:52:41.506651 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet cc407c7eb12e4eadb41fb21385611c94 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc407c7eb12e4eadb41fb21385611c94 already in progress: creating tablet
W20251025 07:52:41.506958 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 1b5ac7e845a24b2eb5b2429d47361232 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b5ac7e845a24b2eb5b2429d47361232 already in progress: creating tablet
I20251025 07:52:41.509897 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet fa77085e81c7440d9e255c44d1e80c26 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.511008 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b21b7da1e2f47829c3f6cdaee689e8a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.511620 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 716fb61312e94a4d9eb8ab8336ca140c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.512398 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet fd265cdbf6a8440393a2b5d8240d1872 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.513456 894 tablet_service.cc:1552] Processing DeleteTablet for tablet bef0ef23d90d480d9c99aef21d7271fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.514096 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 887e7c7a5d0e4aaf88f26246306df19c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.514756 894 tablet_service.cc:1552] Processing DeleteTablet for tablet a7ab3150a35b4caca9119a1374d3f2f7 with delete_type TABLET_DATA_DELETED (replaced by tablet c19c97f881cb45218d4a6f6f231aa96e) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.515480 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet c11f674385184ae6b235f1db8bf1fb91 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.516023 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet d77704b4fe6c4456a2ba40ffec0c60fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.516685 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet e4da89f863cc43f3bf6df908bb9dc6bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.517009 894 tablet_service.cc:1552] Processing DeleteTablet for tablet cb824bd780524832b2b60f5618a39da5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.517573 894 tablet_service.cc:1552] Processing DeleteTablet for tablet ee1c8ea694284d80a30f9c05be4cbf49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.518213 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 4037dee385d04e85bb78c8e77cb00379 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.518718 2169 ts_tablet_manager.cc:1929] T 21e52928580c4762b0ae7152490ef955 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:41.519232 2169 log.cc:1199] T 21e52928580c4762b0ae7152490ef955 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/21e52928580c4762b0ae7152490ef955
I20251025 07:52:41.519618 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 4037dee385d04e85bb78c8e77cb00379 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.519924 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet e47c4f741f354a19b49777bd06e54764 with delete_type TABLET_DATA_DELETED (replaced by tablet 755f0d6c87044e20a8020f8410940123) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.520483 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 72dc084f8efc45478771c247e07a16f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.520980 2169 ts_tablet_manager.cc:1950] T 21e52928580c4762b0ae7152490ef955 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:41.521196 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet be5ab0db6ac34c2c9a370445e251eb46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.524752 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 467324e2432f4caca148951329caa292 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.525133 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d261e432cf5403ea162129706bc4d4d with delete_type TABLET_DATA_DELETED (replaced by tablet e54fe7cbcffa40c4b04144b47bee2b48) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.525480 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 2784bbfb97d5466f9ec856add7d0b29a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.525882 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 94) took 2782 ms. Trace:
I20251025 07:52:41.526234 2169 rpcz_store.cc:276] 1025 07:52:38.743057 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.743126 (+ 69us) service_pool.cc:225] Handling call
1025 07:52:41.488074 (+2744948us) ts_tablet_manager.cc:1151] Deleting tablet 21e52928580c4762b0ae7152490ef955
1025 07:52:41.488085 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.505598 (+ 17513us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.518312 (+ 12714us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.525773 (+ 7461us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2744712}
I20251025 07:52:41.527375 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 627a10a0714f4be8bdbced503fd6388d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:33 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.527967 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 97) took 2775 ms. Trace:
I20251025 07:52:41.528071 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 35ba9d23262248cda6fdf03c51be11a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 01f82980559b4662bdc09786ed429ddc) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:41.528318 2169 rpcz_store.cc:276] 1025 07:52:38.752797 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.752862 (+ 65us) service_pool.cc:225] Handling call
1025 07:52:41.527416 (+2774554us) ts_tablet_manager.cc:1151] Deleting tablet 2e098c9c4613459295b6043048cba587
1025 07:52:41.527428 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.527856 (+ 428us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2774106}
I20251025 07:52:41.529302 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f7c15529be54a11961f385a953b0b03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.529539 2169 tablet_replica.cc:333] T ac0ea298bc114c56a8093e12def21ce1 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:41.529663 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 7f29b64704e147aa9b0b5f6cda860f09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.530468 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet abf94c32ff2945119b710905c075d63c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.530471 2169 raft_consensus.cc:2243] T ac0ea298bc114c56a8093e12def21ce1 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:41.530941 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 1912e59548874e0e8e989050ec0064c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.531243 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b594d4718344aafbdaec34040695589 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.531647 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet ee1c8ea694284d80a30f9c05be4cbf49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.531512 2169 raft_consensus.cc:2272] T ac0ea298bc114c56a8093e12def21ce1 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:41.532270 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 69) took 3013 ms. Trace:
I20251025 07:52:41.532455 2166 rpcz_store.cc:276] 1025 07:52:38.518327 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.518366 (+ 39us) service_pool.cc:225] Handling call
1025 07:52:41.185622 (+2667256us) ts_tablet_manager.cc:1151] Deleting tablet 1d261e432cf5403ea162129706bc4d4d
1025 07:52:41.185634 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.216338 (+ 30704us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.381694 (+165356us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.532235 (+150541us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2667107}
I20251025 07:52:41.533317 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 70) took 3014 ms. Trace:
I20251025 07:52:41.533674 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 1d261e432cf5403ea162129706bc4d4d (table default.Table_590cc281d8b042b19ef12c7f6436752a [id=b1eee5c0243a48c4bbdecc0b4f38ab2d]) successfully deleted
I20251025 07:52:41.534376 2169 ts_tablet_manager.cc:1916] T ac0ea298bc114c56a8093e12def21ce1 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:41.538161 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 6dcab53a020847a1aa27930265ba0205 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.539108 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f25c9caa05b4da4bd56a8a21fde423c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.540115 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet cc407c7eb12e4eadb41fb21385611c94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.540711 1166 raft_consensus.cc:359] T a4e4b99be0434baaa75d988281797846 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.541131 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 1b5ac7e845a24b2eb5b2429d47361232 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
W20251025 07:52:41.542237 448 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 755f0d6c87044e20a8020f8410940123 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439], current schema version=4): Not found: failed to reset TS proxy: No leader found: Tablet 755f0d6c87044e20a8020f8410940123 consensus state has no leader
I20251025 07:52:41.542542 1838 raft_consensus.cc:697] T 4c155aa7fa5b4d68bbab1918b8432372 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:41.542932 1838 consensus_queue.cc:237] T 4c155aa7fa5b4d68bbab1918b8432372 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.545032 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 21e52928580c4762b0ae7152490ef955 (table default.Table_e7d32e91b9c64ce1b3fcff3d806e959f [id=bab869e5001c4d999daa22083fca128a]) successfully deleted
I20251025 07:52:41.545768 1166 raft_consensus.cc:385] T a4e4b99be0434baaa75d988281797846 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:41.545898 1166 raft_consensus.cc:740] T a4e4b99be0434baaa75d988281797846 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:41.546216 1166 consensus_queue.cc:260] T a4e4b99be0434baaa75d988281797846 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.546787 1166 ts_tablet_manager.cc:1434] T a4e4b99be0434baaa75d988281797846 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
W20251025 07:52:41.547099 2159 log.cc:927] Time spent T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Append to log took a long time: real 0.161s user 0.000s sys 0.003s
W20251025 07:52:41.548527 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 2e098c9c4613459295b6043048cba587 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e098c9c4613459295b6043048cba587 already in progress: creating tablet
W20251025 07:52:41.553296 446 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:38361 (request call id 209) took 2813 ms (client timeout 1000 ms). Trace:
I20251025 07:52:41.536271 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d261e432cf5403ea162129706bc4d4d with delete_type TABLET_DATA_DELETED (replaced by tablet e54fe7cbcffa40c4b04144b47bee2b48) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.554306 2166 rpcz_store.cc:276] 1025 07:52:38.518617 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.518658 (+ 41us) service_pool.cc:225] Handling call
1025 07:52:41.533196 (+3014538us) ts_tablet_manager.cc:1151] Deleting tablet 716fb61312e94a4d9eb8ab8336ca140c
1025 07:52:41.533205 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.533299 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3014100}
I20251025 07:52:41.554652 2166 tablet_replica.cc:333] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:41.554868 2166 raft_consensus.cc:2243] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:41.555404 2166 raft_consensus.cc:2272] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:41.556037 2166 ts_tablet_manager.cc:1916] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:41.556329 1167 tablet_bootstrap.cc:492] T c9f7ba3709c745d4a625449d6e1c8ba1 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:41.556495 1167 ts_tablet_manager.cc:1403] T c9f7ba3709c745d4a625449d6e1c8ba1 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.172s user 0.000s sys 0.004s
I20251025 07:52:41.556527 1169 tablet_bootstrap.cc:492] T 80fe3255ed614270bd8352a2853fbc43 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:41.557385 1167 raft_consensus.cc:359] T c9f7ba3709c745d4a625449d6e1c8ba1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:41.557827 1169 ts_tablet_manager.cc:1403] T 80fe3255ed614270bd8352a2853fbc43 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.196s user 0.055s sys 0.063s
I20251025 07:52:41.558679 1169 raft_consensus.cc:359] T 80fe3255ed614270bd8352a2853fbc43 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.558887 1169 raft_consensus.cc:385] T 80fe3255ed614270bd8352a2853fbc43 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:41.558972 1169 raft_consensus.cc:740] T 80fe3255ed614270bd8352a2853fbc43 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:41.559229 1169 consensus_queue.cc:260] T 80fe3255ed614270bd8352a2853fbc43 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.559864 1169 ts_tablet_manager.cc:1434] T 80fe3255ed614270bd8352a2853fbc43 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251025 07:52:41.565972 1166 tablet_bootstrap.cc:492] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:41.568022 2169 ts_tablet_manager.cc:1929] T ac0ea298bc114c56a8093e12def21ce1 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:41.568146 1166 tablet_bootstrap.cc:654] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:41.568504 2169 log.cc:1199] T ac0ea298bc114c56a8093e12def21ce1 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/ac0ea298bc114c56a8093e12def21ce1
I20251025 07:52:41.570009 2169 ts_tablet_manager.cc:1950] T ac0ea298bc114c56a8093e12def21ce1 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:41.572397 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 21e52928580c4762b0ae7152490ef955 with delete_type TABLET_DATA_DELETED (replaced by tablet 9d6a9937d10a4cc2813671dd1bfa1664) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.573297 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 96) took 2820 ms. Trace:
I20251025 07:52:41.574056 2169 rpcz_store.cc:276] 1025 07:52:38.752600 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.753277 (+ 677us) service_pool.cc:225] Handling call
1025 07:52:41.529304 (+2776027us) ts_tablet_manager.cc:1151] Deleting tablet ac0ea298bc114c56a8093e12def21ce1
1025 07:52:41.529315 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.551776 (+ 22461us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.567637 (+ 15861us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.573190 (+ 5553us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2775709}
I20251025 07:52:41.575414 1169 tablet_bootstrap.cc:492] T 34315f64b89148efbf1f20a43d5c5efb P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:41.576862 2169 tablet_replica.cc:333] T 27651312cc2e42a8bfcd245f5d301679 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:41.577801 2169 raft_consensus.cc:2243] T 27651312cc2e42a8bfcd245f5d301679 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:41.578788 2169 raft_consensus.cc:2272] T 27651312cc2e42a8bfcd245f5d301679 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:41.579623 1167 raft_consensus.cc:385] T c9f7ba3709c745d4a625449d6e1c8ba1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:41.582448 2169 ts_tablet_manager.cc:1916] T 27651312cc2e42a8bfcd245f5d301679 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:41.583604 1169 tablet_bootstrap.cc:654] T 34315f64b89148efbf1f20a43d5c5efb P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:41.586149 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 5361f08f50b745eb98bf02e979d7c71e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:41.590137 1167 raft_consensus.cc:740] T c9f7ba3709c745d4a625449d6e1c8ba1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:41.590778 1167 consensus_queue.cc:260] T c9f7ba3709c745d4a625449d6e1c8ba1 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:41.597148 1167 ts_tablet_manager.cc:1434] T c9f7ba3709c745d4a625449d6e1c8ba1 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.040s user 0.002s sys 0.000s
I20251025 07:52:41.597534 1167 tablet_bootstrap.cc:492] T 09898735796541b0a8f3be22cdb6cc30 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:41.599677 1167 tablet_bootstrap.cc:654] T 09898735796541b0a8f3be22cdb6cc30 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:41.608687 983 consensus_peers.cc:597] T 35ba9d23262248cda6fdf03c51be11a4 P db42be7ad3c741a0aa427f27e245993b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 35ba9d23262248cda6fdf03c51be11a4. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:41.609591 2146 ts_tablet_manager.cc:1929] T 74e2d78c024849b0af5c0bde1f48c36f P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:41.609777 2146 log.cc:1199] T 74e2d78c024849b0af5c0bde1f48c36f P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/74e2d78c024849b0af5c0bde1f48c36f
W20251025 07:52:41.554971 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 716fb61312e94a4d9eb8ab8336ca140c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 716fb61312e94a4d9eb8ab8336ca140c already in progress: creating tablet
W20251025 07:52:41.612737 439 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:46231, user_credentials={real_user=slave}} blocked reactor thread for 57801.1us
W20251025 07:52:41.563501 455 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:35777 (request call id 190) took 4083 ms (client timeout 1000 ms). Trace:
W20251025 07:52:41.613082 455 rpcz_store.cc:269] 1025 07:52:37.479711 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.024450 (+544739us) service_pool.cc:225] Handling call
1025 07:52:41.563456 (+3539006us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":43578,"reported_tablets":245,"spinlock_wait_cycles":896}
I20251025 07:52:41.614974 1838 consensus_queue.cc:1048] T e9f7cd765d614d59a7c190b744b0844d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251025 07:52:41.618521 446 rpcz_store.cc:269] 1025 07:52:38.739735 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.954083 (+214348us) service_pool.cc:225] Handling call
1025 07:52:40.232529 (+1278446us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:40.232534 (+ 5us) write_op.cc:454] Released schema lock
1025 07:52:40.601150 (+368616us) spinlock_profiling.cc:243] Waited 132 ms on lock 0x61b00032efe8. stack: 00007fb90f7779f6 00007fb90f7777b0 00007fb90e7533ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fb9232e3398 00007fb9232de751 00007fb9232d9091 00007fb92347fc37 00007fb919f96ca6 00007fb919f969d5 00007fb91671327e 00007fb91671215e 00007fb916718395 00007fb916718cbf
1025 07:52:41.553222 (+952072us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:40.225502 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:40.225629 (+ 127us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:40.225636 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:40.225638 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:40.225762 (+ 124us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:40.225780 (+ 18us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:40.225785 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:40.225902 (+ 117us) tablet.cc:699] Row locks acquired
1025 07:52:40.225905 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:40.225944 (+ 39us) write_op.cc:270] Start()
1025 07:52:40.225983 (+ 39us) write_op.cc:276] Timestamp: P: 1761378760225929 usec, L: 0
1025 07:52:40.225987 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:40.226215 (+ 228us) log.cc:844] Serialized 901 byte log entry
1025 07:52:40.229783 (+ 3568us) op_driver.cc:464] REPLICATION: finished
1025 07:52:40.229908 (+ 125us) write_op.cc:301] APPLY: starting
1025 07:52:40.229925 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:40.229975 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:40.230000 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:40.230070 (+ 70us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:40.230104 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1025 07:52:40.230110 (+ 6us) write_op.cc:312] APPLY: finished
1025 07:52:40.230207 (+ 97us) log.cc:844] Serialized 29 byte log entry
1025 07:52:40.230305 (+ 98us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:40.230328 (+ 23us) write_op.cc:454] Released schema lock
1025 07:52:40.230335 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":312575,"reported_tablets":258,"spinlock_wait_cycles":303091072,"child_traces":[["op",{"apply.queue_time_us":74,"apply.run_cpu_time_us":481,"apply.run_wall_time_us":480,"num_ops":2,"prepare.queue_time_us":50,"prepare.run_cpu_time_us":831,"prepare.run_wall_time_us":830,"raft.queue_time_us":431,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":267,"replication_time_us":3768}]]}
W20251025 07:52:41.575076 453 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:38361 (request call id 210) took 1801 ms (client timeout 1000 ms). Trace:
I20251025 07:52:41.623203 1166 tablet_bootstrap.cc:492] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:41.623425 1166 ts_tablet_manager.cc:1403] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.058s user 0.003s sys 0.001s
I20251025 07:52:41.624513 1166 raft_consensus.cc:359] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:41.624771 1166 raft_consensus.cc:385] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:41.624861 1166 raft_consensus.cc:740] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:41.625156 1166 consensus_queue.cc:260] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:41.625322 2166 ts_tablet_manager.cc:1929] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251025 07:52:41.579906 448 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:42387 (request call id 211) took 2907 ms (client timeout 1000 ms). Trace:
W20251025 07:52:41.580570 450 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:42387 (request call id 210) took 4045 ms (client timeout 1000 ms). Trace:
I20251025 07:52:41.581619 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet ac0ea298bc114c56a8093e12def21ce1 (table default.Table_4228cd02e10a4b65ad0134b261130561 [id=c68c2e10aae74b6ea22df9c3715eac49]) successfully deleted
I20251025 07:52:41.625810 1166 ts_tablet_manager.cc:1434] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20251025 07:52:41.589473 451 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:35777 (request call id 191) took 2947 ms (client timeout 1000 ms). Trace:
W20251025 07:52:41.606026 449 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:38361 (request call id 208) took 4195 ms (client timeout 1000 ms). Trace:
I20251025 07:52:41.626068 2443 raft_consensus.cc:493] T aaef24ccdfba45fe8204327723df834e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:41.626204 2443 raft_consensus.cc:515] T aaef24ccdfba45fe8204327723df834e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.626832 2443 leader_election.cc:290] T aaef24ccdfba45fe8204327723df834e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:41.628006 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aaef24ccdfba45fe8204327723df834e" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:41.628734 850 leader_election.cc:343] T aaef24ccdfba45fe8204327723df834e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:41.633252 451 rpcz_store.cc:269] 1025 07:52:38.642218 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.781807 (+139589us) service_pool.cc:225] Handling call
1025 07:52:40.443366 (+1661559us) spinlock_profiling.cc:243] Waited 6.39 ms on lock 0x61b000311ae8. stack: 00007fb90f7779f6 00007fb90f7777b0 00007fb90e7533ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fb9232e3398 00007fb9232de751 00007fb9232d9091 00007fb92347fc37 00007fb919f96ca6 00007fb919f969d5 00007fb91671327e 00007fb91671215e 00007fb916718395 00007fb916718cbf
1025 07:52:40.604133 (+160767us) spinlock_profiling.cc:243] Waited 94.1 ms on lock 0x61b00032efe8. stack: 00007fb90f7779f6 00007fb90f7777b0 00007fb90e7533ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fb9232e3398 00007fb9232de751 00007fb9232d9091 00007fb92347fc37 00007fb919f96ca6 00007fb919f969d5 00007fb91671327e 00007fb91671215e 00007fb916718395 00007fb916718cbf
1025 07:52:41.589385 (+985252us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":246231,"reported_tablets":266,"spinlock_wait_cycles":231063552}
W20251025 07:52:41.635361 449 rpcz_store.cc:269] 1025 07:52:37.410706 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:37.952175 (+541469us) service_pool.cc:225] Handling call
1025 07:52:40.442371 (+2490196us) spinlock_profiling.cc:243] Waited 6.7 ms on lock 0x61b000311ae8. stack: 00007fb90f7779f6 00007fb90f7777b0 00007fb90e7533ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fb9232e3398 00007fb9232de751 00007fb9232d9091 00007fb92347fc37 00007fb919f96ca6 00007fb919f969d5 00007fb91671327e 00007fb91671215e 00007fb916718395 00007fb916718cbf
1025 07:52:40.598264 (+155893us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x61b00032efe8. stack: 00007fb90f7779f6 00007fb90f7777b0 00007fb90e7533ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fb9232e3398 00007fb9232de751 00007fb9232d9091 00007fb92347fc37 00007fb919f96ca6 00007fb919f969d5 00007fb91671327e 00007fb91671215e 00007fb916718395 00007fb916718cbf
1025 07:52:41.605961 (+1007697us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:38.829634 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:38.829782 (+ 148us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:38.829792 (+ 10us) write_op.cc:435] Acquired schema lock
1025 07:52:38.829796 (+ 4us) tablet.cc:660] Decoding operations
1025 07:52:38.829915 (+ 119us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:38.829930 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:38.829936 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:38.830015 (+ 79us) tablet.cc:699] Row locks acquired
1025 07:52:38.830017 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:38.830053 (+ 36us) write_op.cc:270] Start()
1025 07:52:38.830089 (+ 36us) write_op.cc:276] Timestamp: P: 1761378758830035 usec, L: 0
1025 07:52:38.830092 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:38.830369 (+ 277us) log.cc:844] Serialized 527 byte log entry
1025 07:52:38.834600 (+ 4231us) op_driver.cc:464] REPLICATION: finished
1025 07:52:38.834758 (+ 158us) write_op.cc:301] APPLY: starting
1025 07:52:38.834775 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:38.834840 (+ 65us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:38.834913 (+ 73us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:38.834970 (+ 57us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:38.835003 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:38.835010 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:38.835083 (+ 73us) log.cc:844] Serialized 23 byte log entry
1025 07:52:38.835193 (+ 110us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.835214 (+ 21us) write_op.cc:454] Released schema lock
1025 07:52:38.835221 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:38.835298 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:38.835303 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":200680,"reported_tablets":241,"spinlock_wait_cycles":316050688,"child_traces":[["op",{"apply.queue_time_us":90,"apply.run_cpu_time_us":531,"apply.run_wall_time_us":530,"num_ops":1,"prepare.queue_time_us":57,"prepare.run_cpu_time_us":860,"prepare.run_wall_time_us":860,"raft.queue_time_us":503,"raft.run_cpu_time_us":298,"raft.run_wall_time_us":299,"replication_time_us":4457}]]}
I20251025 07:52:41.638392 2146 ts_tablet_manager.cc:1950] T 74e2d78c024849b0af5c0bde1f48c36f P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
W20251025 07:52:41.637514 453 rpcz_store.cc:269] 1025 07:52:39.773900 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:40.140165 (+366265us) service_pool.cc:225] Handling call
1025 07:52:41.007845 (+867680us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:41.007855 (+ 10us) write_op.cc:454] Released schema lock
1025 07:52:41.575030 (+567175us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:40.483729 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:40.483908 (+ 179us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:40.483947 (+ 39us) write_op.cc:435] Acquired schema lock
1025 07:52:40.483950 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:40.484079 (+ 129us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:40.484091 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:40.484095 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:40.484231 (+ 136us) tablet.cc:699] Row locks acquired
1025 07:52:40.484235 (+ 4us) write_op.cc:260] PREPARE: finished
1025 07:52:40.484281 (+ 46us) write_op.cc:270] Start()
1025 07:52:40.484322 (+ 41us) write_op.cc:276] Timestamp: P: 1761378760484259 usec, L: 0
1025 07:52:40.484328 (+ 6us) op_driver.cc:348] REPLICATION: starting
1025 07:52:40.484562 (+ 234us) log.cc:844] Serialized 901 byte log entry
1025 07:52:40.488006 (+ 3444us) op_driver.cc:464] REPLICATION: finished
1025 07:52:40.488124 (+ 118us) write_op.cc:301] APPLY: starting
1025 07:52:40.488139 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:40.488193 (+ 54us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:40.488216 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:40.488286 (+ 70us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:40.488317 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1025 07:52:40.488324 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:40.488403 (+ 79us) log.cc:844] Serialized 29 byte log entry
1025 07:52:40.488464 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:40.488488 (+ 24us) write_op.cc:454] Released schema lock
1025 07:52:40.488494 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:40.488561 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:40.488565 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
1025 07:52:40.970345 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:40.970450 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:40.970456 (+ 6us) write_op.cc:435] Acquired schema lock
1025 07:52:40.970458 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:40.970549 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:40.970561 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:40.970566 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:40.970651 (+ 85us) tablet.cc:699] Row locks acquired
1025 07:52:40.970654 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:40.970686 (+ 32us) write_op.cc:270] Start()
1025 07:52:40.970715 (+ 29us) write_op.cc:276] Timestamp: P: 1761378760970672 usec, L: 0
1025 07:52:40.970718 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:40.970888 (+ 170us) log.cc:844] Serialized 356 byte log entry
1025 07:52:40.974481 (+ 3593us) op_driver.cc:464] REPLICATION: finished
1025 07:52:40.974595 (+ 114us) write_op.cc:301] APPLY: starting
1025 07:52:40.974611 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:40.974654 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:40.974684 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:40.974740 (+ 56us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:40.974778 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:40.974787 (+ 9us) write_op.cc:312] APPLY: finished
1025 07:52:40.974865 (+ 78us) log.cc:844] Serialized 23 byte log entry
1025 07:52:40.974957 (+ 92us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:40.974979 (+ 22us) write_op.cc:454] Released schema lock
1025 07:52:40.974988 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
1025 07:52:41.371078 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:41.371238 (+ 160us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:41.371247 (+ 9us) write_op.cc:435] Acquired schema lock
1025 07:52:41.371249 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:41.371360 (+ 111us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:41.371374 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:41.371379 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:41.371447 (+ 68us) tablet.cc:699] Row locks acquired
1025 07:52:41.371451 (+ 4us) write_op.cc:260] PREPARE: finished
1025 07:52:41.371489 (+ 38us) write_op.cc:270] Start()
1025 07:52:41.371525 (+ 36us) write_op.cc:276] Timestamp: P: 1761378761371470 usec, L: 0
1025 07:52:41.371528 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:41.371725 (+ 197us) log.cc:844] Serialized 356 byte log entry
1025 07:52:41.378062 (+ 6337us) op_driver.cc:464] REPLICATION: finished
1025 07:52:41.378256 (+ 194us) write_op.cc:301] APPLY: starting
1025 07:52:41.378273 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:41.378320 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:41.378360 (+ 40us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:41.378421 (+ 61us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:41.378456 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:41.378464 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:41.378542 (+ 78us) log.cc:844] Serialized 23 byte log entry
1025 07:52:41.378605 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:41.378628 (+ 23us) write_op.cc:454] Released schema lock
1025 07:52:41.378635 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:41.378704 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:41.378708 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
1025 07:52:41.399090 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:41.399234 (+ 144us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:41.399242 (+ 8us) write_op.cc:435] Acquired schema lock
1025 07:52:41.399245 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:41.399349 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:41.399362 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:41.399367 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:41.399429 (+ 62us) tablet.cc:699] Row locks acquired
1025 07:52:41.399431 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:41.399466 (+ 35us) write_op.cc:270] Start()
1025 07:52:41.399498 (+ 32us) write_op.cc:276] Timestamp: P: 1761378761399447 usec, L: 0
1025 07:52:41.399502 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:41.399683 (+ 181us) log.cc:844] Serialized 356 byte log entry
1025 07:52:41.406010 (+ 6327us) op_driver.cc:464] REPLICATION: finished
1025 07:52:41.406198 (+ 188us) write_op.cc:301] APPLY: starting
1025 07:52:41.406221 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:41.406278 (+ 57us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:41.406313 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:41.406385 (+ 72us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:41.406430 (+ 45us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:41.406442 (+ 12us) write_op.cc:312] APPLY: finished
1025 07:52:41.406546 (+ 104us) log.cc:844] Serialized 23 byte log entry
1025 07:52:41.406621 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:41.406651 (+ 30us) write_op.cc:454] Released schema lock
1025 07:52:41.406660 (+ 9us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:41.406746 (+ 86us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:41.406753 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":109842,"reported_tablets":255,"child_traces":[["op",{"apply.queue_time_us":72,"apply.run_cpu_time_us":427,"apply.run_wall_time_us":427,"num_ops":2,"prepare.queue_time_us":49,"prepare.run_cpu_time_us":936,"prepare.run_wall_time_us":936,"raft.queue_time_us":393,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":246,"replication_time_us":3650}],["op",{"apply.queue_time_us":64,"apply.run_cpu_time_us":456,"apply.run_wall_time_us":455,"num_ops":1,"prepare.queue_time_us":50,"prepare.run_cpu_time_us":651,"prepare.run_wall_time_us":651,"raft.queue_time_us":374,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":270,"replication_time_us":3729}],["op",{"apply.queue_time_us":138,"apply.run_cpu_time_us":442,"apply.run_wall_time_us":441,"num_ops":1,"prepare.queue_time_us":64,"prepare.run_cpu_time_us":764,"prepare.run_wall_time_us":765,"raft.queue_time_us":45,"raft.run_cpu_time_us":286,"raft.run_wall_time_us":286,"replication_time_us":6500}],["op",{"apply.queue_time_us":105,"apply.run_cpu_time_us":539,"apply.run_wall_time_us":539,"num_ops":1,"prepare.queue_time_us":53,"prepare.run_cpu_time_us":695,"prepare.run_wall_time_us":695,"raft.queue_time_us":396,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":245,"replication_time_us":6466}]]}
I20251025 07:52:41.639742 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 108) took 3168 ms. Trace:
I20251025 07:52:41.639894 2146 rpcz_store.cc:276] 1025 07:52:38.471003 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.471553 (+ 550us) service_pool.cc:225] Handling call
1025 07:52:41.388367 (+2916814us) ts_tablet_manager.cc:1151] Deleting tablet 74e2d78c024849b0af5c0bde1f48c36f
1025 07:52:41.388376 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.578305 (+189929us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.609399 (+ 31094us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.639719 (+ 30320us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2915861}
I20251025 07:52:41.640406 2146 tablet_replica.cc:333] T 943ec703e65f4fd3b0c690e08c0bd226 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:41.640513 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 74e2d78c024849b0af5c0bde1f48c36f (table default.Table_434ef1ade0a14af987864563993afae8 [id=3a3af35e11944a6f97fc0144cb9d08d4]) successfully deleted
W20251025 07:52:41.640645 442 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:45417, user_credentials={real_user=slave}} blocked reactor thread for 44356.3us
I20251025 07:52:41.641356 451 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_bd82d871f84943f4baa967f5c0ec74a3" } new_table_name: "default.Table_fb9def2302784c999bf0f5707c60eb4c" modify_external_catalogs: true
I20251025 07:52:41.642478 2169 ts_tablet_manager.cc:1929] T 27651312cc2e42a8bfcd245f5d301679 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:41.642617 2169 log.cc:1199] T 27651312cc2e42a8bfcd245f5d301679 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/27651312cc2e42a8bfcd245f5d301679
I20251025 07:52:41.643007 2169 ts_tablet_manager.cc:1950] T 27651312cc2e42a8bfcd245f5d301679 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:41.643985 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 99) took 2890 ms. Trace:
I20251025 07:52:41.649775 449 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_974f1ee455d7488593e5c496cc012099" } new_table_name: "default.Table_27c62e973d74406fb00db8bcf324749c" modify_external_catalogs: true
I20251025 07:52:41.658226 2169 rpcz_store.cc:276] 1025 07:52:38.753942 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.753997 (+ 55us) service_pool.cc:225] Handling call
1025 07:52:41.576658 (+2822661us) ts_tablet_manager.cc:1151] Deleting tablet 27651312cc2e42a8bfcd245f5d301679
1025 07:52:41.576669 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.586125 (+ 9456us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.642336 (+ 56211us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.643960 (+ 1624us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2822257}
W20251025 07:52:41.658231 450 rpcz_store.cc:269] 1025 07:52:37.535113 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.037501 (+502388us) service_pool.cc:225] Handling call
1025 07:52:40.602706 (+2565205us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x61b00032efe8. stack: 00007fb90f7779f6 00007fb90f7777b0 00007fb90e7533ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fb9232e3398 00007fb9232de751 00007fb9232d9091 00007fb92347fc37 00007fb919f96ca6 00007fb919f969d5 00007fb91671327e 00007fb91671215e 00007fb916718395 00007fb916718cbf
1025 07:52:41.580531 (+977825us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:39.342434 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:39.342561 (+ 127us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:39.342568 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:39.342571 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:39.342686 (+ 115us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:39.342702 (+ 16us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:39.342707 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:39.342809 (+ 102us) tablet.cc:699] Row locks acquired
1025 07:52:39.342812 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:39.342866 (+ 54us) write_op.cc:270] Start()
1025 07:52:39.342898 (+ 32us) write_op.cc:276] Timestamp: P: 1761378759342831 usec, L: 0
1025 07:52:39.342901 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:39.343117 (+ 216us) log.cc:844] Serialized 901 byte log entry
1025 07:52:39.347890 (+ 4773us) op_driver.cc:464] REPLICATION: finished
1025 07:52:39.348130 (+ 240us) write_op.cc:301] APPLY: starting
1025 07:52:39.348155 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:39.348218 (+ 63us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:39.348255 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:39.348382 (+ 127us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:39.348432 (+ 50us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1025 07:52:39.348444 (+ 12us) write_op.cc:312] APPLY: finished
1025 07:52:39.348544 (+ 100us) log.cc:844] Serialized 29 byte log entry
1025 07:52:39.348620 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:39.348651 (+ 31us) write_op.cc:454] Released schema lock
1025 07:52:39.348662 (+ 11us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:39.348751 (+ 89us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:39.348757 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":137256,"reported_tablets":242,"spinlock_wait_cycles":282320256,"child_traces":[["op",{"apply.queue_time_us":158,"apply.run_cpu_time_us":618,"apply.run_wall_time_us":618,"num_ops":2,"prepare.queue_time_us":50,"prepare.run_cpu_time_us":787,"prepare.run_wall_time_us":787,"raft.queue_time_us":313,"raft.run_cpu_time_us":245,"raft.run_wall_time_us":245,"replication_time_us":4959}]]}
I20251025 07:52:41.659013 2169 tablet_replica.cc:333] T fab67b56eaa44b54886c17181d8c52b2 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:41.659225 2169 raft_consensus.cc:2243] T fab67b56eaa44b54886c17181d8c52b2 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:41.659538 2169 raft_consensus.cc:2272] T fab67b56eaa44b54886c17181d8c52b2 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:41.659911 2146 raft_consensus.cc:2243] T 943ec703e65f4fd3b0c690e08c0bd226 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:41.660081 2169 ts_tablet_manager.cc:1916] T fab67b56eaa44b54886c17181d8c52b2 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:41.660118 2146 raft_consensus.cc:2272] T 943ec703e65f4fd3b0c690e08c0bd226 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:41.661016 2146 ts_tablet_manager.cc:1916] T 943ec703e65f4fd3b0c690e08c0bd226 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:41.661121 450 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_ea269532aff34ca1803fc1c5d5ad3641" } new_table_name: "default.Table_6e242ba55b194c23a6701f4b064fdbe5" modify_external_catalogs: true
I20251025 07:52:41.662528 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 27651312cc2e42a8bfcd245f5d301679 (table default.Table_dac528b081844b7c86027169c6234c27 [id=b0caa78ee8c84aa8bea4818cf9feb70c]) successfully deleted
I20251025 07:52:41.669544 2316 raft_consensus.cc:493] T f97acc9a638b4d5a968dcc3412a384bc P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:41.669705 2316 raft_consensus.cc:515] T f97acc9a638b4d5a968dcc3412a384bc P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
W20251025 07:52:41.657999 448 rpcz_store.cc:269] 1025 07:52:38.672155 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.891493 (+219338us) service_pool.cc:225] Handling call
1025 07:52:40.168506 (+1277013us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:40.168514 (+ 8us) write_op.cc:454] Released schema lock
1025 07:52:40.597940 (+429426us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x61b00032efe8. stack: 00007fb90f7779f6 00007fb90f7777b0 00007fb90e7533ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fb9232e3398 00007fb9232de751 00007fb9232d9091 00007fb92347fc37 00007fb919f96ca6 00007fb919f969d5 00007fb91671327e 00007fb91671215e 00007fb916718395 00007fb916718cbf
1025 07:52:41.579864 (+981924us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:40.155273 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:40.155407 (+ 134us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:40.155414 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:40.155416 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:40.155564 (+ 148us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:40.155579 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:40.155584 (+ 5us) tablet.cc:683] Acquiring locks for 3 operations
1025 07:52:40.155687 (+ 103us) tablet.cc:699] Row locks acquired
1025 07:52:40.155690 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:40.155726 (+ 36us) write_op.cc:270] Start()
1025 07:52:40.155759 (+ 33us) write_op.cc:276] Timestamp: P: 1761378760155713 usec, L: 0
1025 07:52:40.155763 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:40.155935 (+ 172us) log.cc:844] Serialized 1275 byte log entry
1025 07:52:40.160102 (+ 4167us) op_driver.cc:464] REPLICATION: finished
1025 07:52:40.160219 (+ 117us) write_op.cc:301] APPLY: starting
1025 07:52:40.160245 (+ 26us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:40.160295 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:40.160327 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:40.160404 (+ 77us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:40.160438 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
1025 07:52:40.160445 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:40.160522 (+ 77us) log.cc:844] Serialized 35 byte log entry
1025 07:52:40.160579 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:40.160602 (+ 23us) write_op.cc:454] Released schema lock
1025 07:52:40.160609 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":168640,"reported_tablets":259,"spinlock_wait_cycles":289482624,"child_traces":[["op",{"apply.queue_time_us":69,"apply.run_cpu_time_us":439,"apply.run_wall_time_us":439,"num_ops":3,"prepare.queue_time_us":1225,"prepare.run_cpu_time_us":770,"prepare.run_wall_time_us":771,"raft.queue_time_us":402,"raft.run_cpu_time_us":277,"raft.run_wall_time_us":277,"replication_time_us":4308}]]}
I20251025 07:52:41.648291 1169 tablet_bootstrap.cc:492] T 34315f64b89148efbf1f20a43d5c5efb P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:41.672638 1169 ts_tablet_manager.cc:1403] T 34315f64b89148efbf1f20a43d5c5efb P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.097s user 0.005s sys 0.042s
I20251025 07:52:41.672875 448 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_332b308220f64c6dbb5f29711d8463b3" } new_table_name: "default.Table_b7da40b96d6c4359bed6f898e8cf6275" modify_external_catalogs: true
W20251025 07:52:41.673815 983 consensus_peers.cc:597] T 74e2d78c024849b0af5c0bde1f48c36f P db42be7ad3c741a0aa427f27e245993b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 74e2d78c024849b0af5c0bde1f48c36f. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:41.674516 2316 leader_election.cc:290] T f97acc9a638b4d5a968dcc3412a384bc P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:41.676203 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f97acc9a638b4d5a968dcc3412a384bc" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:41.677604 1169 raft_consensus.cc:359] T 34315f64b89148efbf1f20a43d5c5efb P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.678256 1169 raft_consensus.cc:385] T 34315f64b89148efbf1f20a43d5c5efb P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:41.678589 1169 raft_consensus.cc:740] T 34315f64b89148efbf1f20a43d5c5efb P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:41.679622 1169 consensus_queue.cc:260] T 34315f64b89148efbf1f20a43d5c5efb P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.690619 1169 ts_tablet_manager.cc:1434] T 34315f64b89148efbf1f20a43d5c5efb P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20251025 07:52:41.692363 2443 raft_consensus.cc:493] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:41.692502 2443 raft_consensus.cc:515] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:41.693081 2443 leader_election.cc:290] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:41.696382 2314 consensus_queue.cc:1048] T e9f7cd765d614d59a7c190b744b0844d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:41.698287 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8eaf6c97ab9b41e9ae746a9c5abe1735" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:41.700341 850 leader_election.cc:343] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:41.701123 453 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_7613dd8cbc914d8d922e21f8b4183d5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:41.701880 453 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7613dd8cbc914d8d922e21f8b4183d5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:41.702733 2169 ts_tablet_manager.cc:1929] T fab67b56eaa44b54886c17181d8c52b2 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:41.710194 2169 log.cc:1199] T fab67b56eaa44b54886c17181d8c52b2 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/fab67b56eaa44b54886c17181d8c52b2
I20251025 07:52:41.710739 2169 ts_tablet_manager.cc:1950] T fab67b56eaa44b54886c17181d8c52b2 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:41.711241 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 98) took 2957 ms. Trace:
I20251025 07:52:41.714326 2169 rpcz_store.cc:276] 1025 07:52:38.753746 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.754296 (+ 550us) service_pool.cc:225] Handling call
1025 07:52:41.658960 (+2904664us) ts_tablet_manager.cc:1151] Deleting tablet fab67b56eaa44b54886c17181d8c52b2
1025 07:52:41.658972 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.675467 (+ 16495us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.702613 (+ 27146us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.711224 (+ 8611us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2904441}
I20251025 07:52:41.718029 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet fab67b56eaa44b54886c17181d8c52b2 (table default.Table_6ca844d1e78c4f19939985aec32a8afd [id=1e933863244e4792b4c1b4d6d5c674b7]) successfully deleted
I20251025 07:52:41.718187 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 101) took 2962 ms. Trace:
I20251025 07:52:41.718287 2169 rpcz_store.cc:276] 1025 07:52:38.755601 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.755652 (+ 51us) service_pool.cc:225] Handling call
1025 07:52:41.718028 (+2962376us) ts_tablet_manager.cc:1151] Deleting tablet 95efd553493e4966bc48a01cac6e1e10
1025 07:52:41.718041 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.718161 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2962175}
W20251025 07:52:41.722378 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 95efd553493e4966bc48a01cac6e1e10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95efd553493e4966bc48a01cac6e1e10 already in progress: creating tablet
I20251025 07:52:41.723074 1166 tablet_bootstrap.cc:492] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:41.724023 2169 tablet_replica.cc:333] T 896bd7870d87408e8adfdb7a78998111 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:41.724476 2169 raft_consensus.cc:2243] T 896bd7870d87408e8adfdb7a78998111 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:41.725164 1166 tablet_bootstrap.cc:654] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:41.627422 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aaef24ccdfba45fe8204327723df834e" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:41.641302 1167 tablet_bootstrap.cc:492] T 09898735796541b0a8f3be22cdb6cc30 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:41.727994 774 raft_consensus.cc:2468] T aaef24ccdfba45fe8204327723df834e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:41.728050 1167 ts_tablet_manager.cc:1403] T 09898735796541b0a8f3be22cdb6cc30 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.131s user 0.004s sys 0.000s
I20251025 07:52:41.675222 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f97acc9a638b4d5a968dcc3412a384bc" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:41.728345 2166 log.cc:1199] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/bef0ef23d90d480d9c99aef21d7271fd
I20251025 07:52:41.695474 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8eaf6c97ab9b41e9ae746a9c5abe1735" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:41.728369 775 raft_consensus.cc:2393] T f97acc9a638b4d5a968dcc3412a384bc P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:41.728511 773 raft_consensus.cc:2468] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
W20251025 07:52:41.676975 712 consensus_peers.cc:597] T 7f29b64704e147aa9b0b5f6cda860f09 P 23eeffd3de254ce18db5febfde971c6b -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f29b64704e147aa9b0b5f6cda860f09. This is attempt 1: this message will repeat every 5th retry.
W20251025 07:52:41.728684 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:45417, user_credentials={real_user=slave}} blocked reactor thread for 51750.6us
I20251025 07:52:41.729085 2166 ts_tablet_manager.cc:1950] T bef0ef23d90d480d9c99aef21d7271fd P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:41.728964 1167 raft_consensus.cc:359] T 09898735796541b0a8f3be22cdb6cc30 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:41.729207 1167 raft_consensus.cc:385] T 09898735796541b0a8f3be22cdb6cc30 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:41.729264 1167 raft_consensus.cc:740] T 09898735796541b0a8f3be22cdb6cc30 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:41.729440 1167 consensus_queue.cc:260] T 09898735796541b0a8f3be22cdb6cc30 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:41.729969 1167 ts_tablet_manager.cc:1434] T 09898735796541b0a8f3be22cdb6cc30 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:41.730261 1167 tablet_bootstrap.cc:492] T b2044ca206634eb285197af4ba2073e7 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:41.731933 1167 tablet_bootstrap.cc:654] T b2044ca206634eb285197af4ba2073e7 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:41.691816 1169 tablet_bootstrap.cc:492] T 724d0b7e1b65445a9a1f056852983bdb P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:41.730062 982 leader_election.cc:304] T f97acc9a638b4d5a968dcc3412a384bc P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:41.734587 2169 raft_consensus.cc:2272] T 896bd7870d87408e8adfdb7a78998111 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:41.735030 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 71) took 3198 ms. Trace:
I20251025 07:52:41.735114 2348 raft_consensus.cc:3060] T f97acc9a638b4d5a968dcc3412a384bc P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:41.736183 2166 rpcz_store.cc:276] 1025 07:52:38.536869 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.536918 (+ 49us) service_pool.cc:225] Handling call
1025 07:52:41.554604 (+3017686us) ts_tablet_manager.cc:1151] Deleting tablet bef0ef23d90d480d9c99aef21d7271fd
1025 07:52:41.554613 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.622111 (+ 67498us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.625219 (+ 3108us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.735014 (+109795us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3017513}
I20251025 07:52:41.737884 2348 raft_consensus.cc:2749] T f97acc9a638b4d5a968dcc3412a384bc P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:41.737998 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet bef0ef23d90d480d9c99aef21d7271fd (table default.Table_e3b368ac698c46f4a8c947d02f9a2680 [id=ce5f22039d104e518ef6f40468b404cc]) successfully deleted
I20251025 07:52:41.738494 2169 ts_tablet_manager.cc:1916] T 896bd7870d87408e8adfdb7a78998111 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:41.738859 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 72) took 3184 ms. Trace:
I20251025 07:52:41.739131 2166 rpcz_store.cc:276] 1025 07:52:38.554398 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.554448 (+ 50us) service_pool.cc:225] Handling call
1025 07:52:41.738432 (+3183984us) ts_tablet_manager.cc:1151] Deleting tablet f661d5909657455581dcb5e6eadd3c0b
1025 07:52:41.738441 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.738543 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3183784}
W20251025 07:52:41.739876 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet f661d5909657455581dcb5e6eadd3c0b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f661d5909657455581dcb5e6eadd3c0b already in progress: creating tablet
I20251025 07:52:41.729596 847 leader_election.cc:304] T aaef24ccdfba45fe8204327723df834e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:41.740592 847 leader_election.cc:304] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:41.740813 2314 raft_consensus.cc:2804] T aaef24ccdfba45fe8204327723df834e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:41.740877 2314 raft_consensus.cc:493] T aaef24ccdfba45fe8204327723df834e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:41.740950 2314 raft_consensus.cc:3060] T aaef24ccdfba45fe8204327723df834e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:41.741730 2106 raft_consensus.cc:2804] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:41.741797 2106 raft_consensus.cc:493] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:41.741863 2106 raft_consensus.cc:3060] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:41.743624 2314 raft_consensus.cc:515] T aaef24ccdfba45fe8204327723df834e P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.744343 2106 raft_consensus.cc:515] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:41.744796 2314 leader_election.cc:290] T aaef24ccdfba45fe8204327723df834e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:41.744908 2106 leader_election.cc:290] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:41.745400 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8eaf6c97ab9b41e9ae746a9c5abe1735" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:41.745620 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aaef24ccdfba45fe8204327723df834e" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:41.745754 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8eaf6c97ab9b41e9ae746a9c5abe1735" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:41.745888 775 raft_consensus.cc:3060] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:41.745782 773 raft_consensus.cc:3060] T aaef24ccdfba45fe8204327723df834e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:41.745990 1169 tablet_bootstrap.cc:654] T 724d0b7e1b65445a9a1f056852983bdb P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:41.748639 773 raft_consensus.cc:2468] T aaef24ccdfba45fe8204327723df834e P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:41.748870 775 raft_consensus.cc:2468] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:41.749053 847 leader_election.cc:304] T aaef24ccdfba45fe8204327723df834e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
W20251025 07:52:41.749050 850 leader_election.cc:343] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:41.749553 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aaef24ccdfba45fe8204327723df834e" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:41.749716 847 leader_election.cc:304] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:41.749934 2314 raft_consensus.cc:2804] T aaef24ccdfba45fe8204327723df834e P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:41.750083 2105 raft_consensus.cc:2804] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:41.750999 2314 raft_consensus.cc:697] T aaef24ccdfba45fe8204327723df834e P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:41.751125 2105 raft_consensus.cc:697] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:41.751256 2314 consensus_queue.cc:237] T aaef24ccdfba45fe8204327723df834e P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.751288 2105 consensus_queue.cc:237] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:41.750437 850 leader_election.cc:343] T aaef24ccdfba45fe8204327723df834e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:41.761399 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 73) took 3206 ms. Trace:
I20251025 07:52:41.762032 2166 rpcz_store.cc:276] 1025 07:52:38.554817 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.554878 (+ 61us) service_pool.cc:225] Handling call
1025 07:52:41.761261 (+3206383us) ts_tablet_manager.cc:1151] Deleting tablet 4f0c4f0e8f014e59be9e46d2afe0be9d
1025 07:52:41.761271 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.761375 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3206209}
W20251025 07:52:41.763144 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 4f0c4f0e8f014e59be9e46d2afe0be9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f0c4f0e8f014e59be9e46d2afe0be9d already in progress: creating tablet
I20251025 07:52:41.763653 2166 tablet_replica.cc:333] T 2784bbfb97d5466f9ec856add7d0b29a P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:41.764331 2166 raft_consensus.cc:2243] T 2784bbfb97d5466f9ec856add7d0b29a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:41.765278 2166 raft_consensus.cc:2272] T 2784bbfb97d5466f9ec856add7d0b29a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:41.766825 2166 ts_tablet_manager.cc:1916] T 2784bbfb97d5466f9ec856add7d0b29a P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:41.767802 1166 tablet_bootstrap.cc:492] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:41.768630 1166 ts_tablet_manager.cc:1403] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.002s
I20251025 07:52:41.769902 1166 raft_consensus.cc:359] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:41.773020 1169 tablet_bootstrap.cc:492] T 724d0b7e1b65445a9a1f056852983bdb P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:41.773295 1169 ts_tablet_manager.cc:1403] T 724d0b7e1b65445a9a1f056852983bdb P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.082s user 0.005s sys 0.001s
I20251025 07:52:41.774461 1169 raft_consensus.cc:359] T 724d0b7e1b65445a9a1f056852983bdb P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:41.775207 1169 raft_consensus.cc:385] T 724d0b7e1b65445a9a1f056852983bdb P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:41.775357 1169 raft_consensus.cc:740] T 724d0b7e1b65445a9a1f056852983bdb P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:41.775764 1169 consensus_queue.cc:260] T 724d0b7e1b65445a9a1f056852983bdb P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:41.778270 1169 ts_tablet_manager.cc:1434] T 724d0b7e1b65445a9a1f056852983bdb P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20251025 07:52:41.779695 1169 tablet_bootstrap.cc:492] T 88eb51447e0f4080baa49dd902deb24f P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:41.785244 1166 raft_consensus.cc:385] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:41.785439 1166 raft_consensus.cc:740] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:41.785730 2314 consensus_queue.cc:1048] T f97acc9a638b4d5a968dcc3412a384bc P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:41.785988 1166 consensus_queue.cc:260] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:41.787323 1166 ts_tablet_manager.cc:1434] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20251025 07:52:41.788103 1166 tablet_bootstrap.cc:492] T 4b28215f61574cf48c447dd01955ac77 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:41.793226 1167 tablet_bootstrap.cc:492] T b2044ca206634eb285197af4ba2073e7 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:41.793435 1167 ts_tablet_manager.cc:1403] T b2044ca206634eb285197af4ba2073e7 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.063s user 0.004s sys 0.004s
I20251025 07:52:41.776917 2146 ts_tablet_manager.cc:1929] T 943ec703e65f4fd3b0c690e08c0bd226 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:41.795737 2146 log.cc:1199] T 943ec703e65f4fd3b0c690e08c0bd226 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/943ec703e65f4fd3b0c690e08c0bd226
I20251025 07:52:41.798638 2146 ts_tablet_manager.cc:1950] T 943ec703e65f4fd3b0c690e08c0bd226 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:41.795001 1167 raft_consensus.cc:359] T b2044ca206634eb285197af4ba2073e7 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:41.799063 1167 raft_consensus.cc:385] T b2044ca206634eb285197af4ba2073e7 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:41.799309 1167 raft_consensus.cc:740] T b2044ca206634eb285197af4ba2073e7 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:41.800345 1166 tablet_bootstrap.cc:654] T 4b28215f61574cf48c447dd01955ac77 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:41.802484 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 103) took 3334 ms. Trace:
I20251025 07:52:41.808617 453 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 60) took 3262 ms. Trace:
I20251025 07:52:41.808772 453 rpcz_store.cc:276] 1025 07:52:38.545909 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.700848 (+3154939us) service_pool.cc:225] Handling call
1025 07:52:41.702025 (+ 1177us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:41.702564 (+ 539us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:41.734185 (+ 31621us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:41.734194 (+ 9us) write_op.cc:454] Released schema lock
1025 07:52:41.734304 (+ 110us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:41.734423 (+ 119us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:41.808553 (+ 74130us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:41.703155 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:41.703274 (+ 119us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:41.703282 (+ 8us) write_op.cc:435] Acquired schema lock
1025 07:52:41.703285 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:41.703382 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:41.703401 (+ 19us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:41.703408 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:41.703493 (+ 85us) tablet.cc:699] Row locks acquired
1025 07:52:41.703495 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:41.703531 (+ 36us) write_op.cc:270] Start()
1025 07:52:41.703567 (+ 36us) write_op.cc:276] Timestamp: P: 1761378761703511 usec, L: 0
1025 07:52:41.703570 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:41.703758 (+ 188us) log.cc:844] Serialized 433 byte log entry
1025 07:52:41.707415 (+ 3657us) op_driver.cc:464] REPLICATION: finished
1025 07:52:41.707538 (+ 123us) write_op.cc:301] APPLY: starting
1025 07:52:41.707553 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:41.707602 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:41.707629 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:41.707782 (+ 153us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:41.707819 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:41.707828 (+ 9us) write_op.cc:312] APPLY: finished
1025 07:52:41.707927 (+ 99us) log.cc:844] Serialized 29 byte log entry
1025 07:52:41.708002 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:41.708024 (+ 22us) write_op.cc:454] Released schema lock
1025 07:52:41.708031 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":72,"apply.run_cpu_time_us":553,"apply.run_wall_time_us":552,"num_ops":2,"prepare.queue_time_us":57,"prepare.run_cpu_time_us":757,"prepare.run_wall_time_us":757,"raft.queue_time_us":115,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":255,"replication_time_us":3811}]]}
I20251025 07:52:41.809882 453 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_22e4370917454a7cb03920884a70a4f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:41.803496 1167 consensus_queue.cc:260] T b2044ca206634eb285197af4ba2073e7 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:41.814469 1169 tablet_bootstrap.cc:654] T 88eb51447e0f4080baa49dd902deb24f P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:41.822167 1167 ts_tablet_manager.cc:1434] T b2044ca206634eb285197af4ba2073e7 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.029s user 0.000s sys 0.003s
I20251025 07:52:41.822436 2146 rpcz_store.cc:276] 1025 07:52:38.468151 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.468727 (+ 576us) service_pool.cc:225] Handling call
1025 07:52:41.640343 (+3171616us) ts_tablet_manager.cc:1151] Deleting tablet 943ec703e65f4fd3b0c690e08c0bd226
1025 07:52:41.640358 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.735772 (+ 95414us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.776812 (+ 41040us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.802462 (+ 25650us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19251,"tablet-delete.queue_time_us":3167808}
W20251025 07:52:41.822753 453 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22e4370917454a7cb03920884a70a4f1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:41.823189 1167 tablet_bootstrap.cc:492] T 0f5f6ca6c3cd4d369690e425387a4344 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:41.827579 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 943ec703e65f4fd3b0c690e08c0bd226 (table default.Table_db98182d07a54725823623925f51a0cb [id=62cf364c073b47a1939f41dd69222582]) successfully deleted
I20251025 07:52:41.828697 2146 tablet_replica.cc:333] T 5fca357d7e24494fb6af4bd73d98833c P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:41.829165 2146 raft_consensus.cc:2243] T 5fca357d7e24494fb6af4bd73d98833c P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:41.830260 2146 raft_consensus.cc:2272] T 5fca357d7e24494fb6af4bd73d98833c P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:41.832070 2314 raft_consensus.cc:493] T 724d0b7e1b65445a9a1f056852983bdb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:41.836586 2314 raft_consensus.cc:515] T 724d0b7e1b65445a9a1f056852983bdb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:41.836706 1169 tablet_bootstrap.cc:492] T 88eb51447e0f4080baa49dd902deb24f P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:41.836984 1169 ts_tablet_manager.cc:1403] T 88eb51447e0f4080baa49dd902deb24f P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.008s
I20251025 07:52:41.837695 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724d0b7e1b65445a9a1f056852983bdb" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:41.837893 1049 raft_consensus.cc:2468] T 724d0b7e1b65445a9a1f056852983bdb P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:41.838686 850 leader_election.cc:304] T 724d0b7e1b65445a9a1f056852983bdb P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:41.838721 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724d0b7e1b65445a9a1f056852983bdb" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:41.838932 768 raft_consensus.cc:2468] T 724d0b7e1b65445a9a1f056852983bdb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:41.838996 2105 raft_consensus.cc:2804] T 724d0b7e1b65445a9a1f056852983bdb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:41.839071 2105 raft_consensus.cc:493] T 724d0b7e1b65445a9a1f056852983bdb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:41.839079 453 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 61) took 3284 ms. Trace:
I20251025 07:52:41.839147 2105 raft_consensus.cc:3060] T 724d0b7e1b65445a9a1f056852983bdb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:41.839222 453 rpcz_store.cc:276] 1025 07:52:38.554773 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.809637 (+3254864us) service_pool.cc:225] Handling call
1025 07:52:41.828434 (+ 18797us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:41.829031 (+ 597us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:41.838727 (+ 9696us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:41.838733 (+ 6us) write_op.cc:454] Released schema lock
1025 07:52:41.838807 (+ 74us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:41.838937 (+ 130us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:41.839037 (+ 100us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:41.829884 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:41.830019 (+ 135us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:41.830026 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:41.830028 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:41.830867 (+ 839us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:41.830884 (+ 17us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:41.830889 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:41.830984 (+ 95us) tablet.cc:699] Row locks acquired
1025 07:52:41.830987 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:41.831026 (+ 39us) write_op.cc:270] Start()
1025 07:52:41.831064 (+ 38us) write_op.cc:276] Timestamp: P: 1761378761831008 usec, L: 0
1025 07:52:41.831068 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:41.831268 (+ 200us) log.cc:844] Serialized 433 byte log entry
1025 07:52:41.834992 (+ 3724us) op_driver.cc:464] REPLICATION: finished
1025 07:52:41.835114 (+ 122us) write_op.cc:301] APPLY: starting
1025 07:52:41.835130 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:41.835178 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:41.835213 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:41.835342 (+ 129us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:41.835377 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:41.835384 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:41.835513 (+ 129us) log.cc:844] Serialized 29 byte log entry
1025 07:52:41.835578 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:41.835603 (+ 25us) write_op.cc:454] Released schema lock
1025 07:52:41.835613 (+ 10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":67,"apply.run_cpu_time_us":566,"apply.run_wall_time_us":564,"num_ops":2,"prepare.queue_time_us":268,"prepare.run_cpu_time_us":815,"prepare.run_wall_time_us":1484,"raft.queue_time_us":408,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":253,"replication_time_us":3896}]]}
I20251025 07:52:41.839721 1169 raft_consensus.cc:359] T 88eb51447e0f4080baa49dd902deb24f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:41.840245 1169 raft_consensus.cc:385] T 88eb51447e0f4080baa49dd902deb24f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:41.840431 1169 raft_consensus.cc:740] T 88eb51447e0f4080baa49dd902deb24f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:41.840874 1169 consensus_queue.cc:260] T 88eb51447e0f4080baa49dd902deb24f P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:41.841295 453 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_ba4b8cc7b930482690b7e192e0edff12" } new_table_name: "default.Table_a9026f5e095c4063b86d4c544c686e22" modify_external_catalogs: true
I20251025 07:52:41.841889 1169 ts_tablet_manager.cc:1434] T 88eb51447e0f4080baa49dd902deb24f P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251025 07:52:41.841885 2105 raft_consensus.cc:515] T 724d0b7e1b65445a9a1f056852983bdb P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:41.842211 2314 leader_election.cc:290] T 724d0b7e1b65445a9a1f056852983bdb P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:41.842923 2105 leader_election.cc:290] T 724d0b7e1b65445a9a1f056852983bdb P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:41.843333 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724d0b7e1b65445a9a1f056852983bdb" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:41.843498 768 raft_consensus.cc:3060] T 724d0b7e1b65445a9a1f056852983bdb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:41.843513 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724d0b7e1b65445a9a1f056852983bdb" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:41.843660 1049 raft_consensus.cc:3060] T 724d0b7e1b65445a9a1f056852983bdb P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:41.846393 768 raft_consensus.cc:2468] T 724d0b7e1b65445a9a1f056852983bdb P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:41.847016 847 leader_election.cc:304] T 724d0b7e1b65445a9a1f056852983bdb P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:41.847046 1049 raft_consensus.cc:2468] T 724d0b7e1b65445a9a1f056852983bdb P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:41.847637 2105 raft_consensus.cc:2804] T 724d0b7e1b65445a9a1f056852983bdb P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:41.847800 2105 raft_consensus.cc:697] T 724d0b7e1b65445a9a1f056852983bdb P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:41.848086 2105 consensus_queue.cc:237] T 724d0b7e1b65445a9a1f056852983bdb P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:41.850521 1659 raft_consensus.cc:493] T c9f7ba3709c745d4a625449d6e1c8ba1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:41.850646 1659 raft_consensus.cc:515] T c9f7ba3709c745d4a625449d6e1c8ba1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:41.851262 1659 leader_election.cc:290] T c9f7ba3709c745d4a625449d6e1c8ba1 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:41.852010 2166 ts_tablet_manager.cc:1929] T 2784bbfb97d5466f9ec856add7d0b29a P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:41.852118 2166 log.cc:1199] T 2784bbfb97d5466f9ec856add7d0b29a P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/2784bbfb97d5466f9ec856add7d0b29a
I20251025 07:52:41.851970 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9f7ba3709c745d4a625449d6e1c8ba1" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:41.853130 710 leader_election.cc:343] T c9f7ba3709c745d4a625449d6e1c8ba1 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:41.854764 2166 ts_tablet_manager.cc:1950] T 2784bbfb97d5466f9ec856add7d0b29a P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:41.855837 1169 tablet_bootstrap.cc:492] T b3a0af893a8947848e12093d0a7f0229 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:41.856696 1166 tablet_bootstrap.cc:492] T 4b28215f61574cf48c447dd01955ac77 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:41.856889 1166 ts_tablet_manager.cc:1403] T 4b28215f61574cf48c447dd01955ac77 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.069s user 0.000s sys 0.006s
I20251025 07:52:41.857707 1166 raft_consensus.cc:359] T 4b28215f61574cf48c447dd01955ac77 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.857944 1166 raft_consensus.cc:385] T 4b28215f61574cf48c447dd01955ac77 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:41.858031 1166 raft_consensus.cc:740] T 4b28215f61574cf48c447dd01955ac77 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:41.858271 1166 consensus_queue.cc:260] T 4b28215f61574cf48c447dd01955ac77 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.858543 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 74) took 3303 ms. Trace:
I20251025 07:52:41.858647 2166 rpcz_store.cc:276] 1025 07:52:38.555271 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.555314 (+ 43us) service_pool.cc:225] Handling call
1025 07:52:41.763604 (+3208290us) ts_tablet_manager.cc:1151] Deleting tablet 2784bbfb97d5466f9ec856add7d0b29a
1025 07:52:41.763614 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.792571 (+ 28957us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.851890 (+ 59319us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.858523 (+ 6633us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3208141}
I20251025 07:52:41.858954 1166 ts_tablet_manager.cc:1434] T 4b28215f61574cf48c447dd01955ac77 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:41.859433 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 2784bbfb97d5466f9ec856add7d0b29a (table default.Table_5aff13b5fcbc4899800760718fb3cc90 [id=ab3900039f5c491bbc1b96925deb1f1b]) successfully deleted
I20251025 07:52:41.859746 1166 tablet_bootstrap.cc:492] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:41.860572 1169 tablet_bootstrap.cc:654] T b3a0af893a8947848e12093d0a7f0229 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:41.861979 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 75) took 3299 ms. Trace:
I20251025 07:52:41.862113 2166 rpcz_store.cc:276] 1025 07:52:38.562656 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.562701 (+ 45us) service_pool.cc:225] Handling call
1025 07:52:41.861828 (+3299127us) ts_tablet_manager.cc:1151] Deleting tablet fd265cdbf6a8440393a2b5d8240d1872
1025 07:52:41.861840 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.861960 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3298951}
I20251025 07:52:41.862406 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9f7ba3709c745d4a625449d6e1c8ba1" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:41.862578 1659 raft_consensus.cc:493] T b772025852794a49b072f37cccbd3291 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3aea2869ff814cc7b142012faa274382)
W20251025 07:52:41.862622 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet fd265cdbf6a8440393a2b5d8240d1872 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd265cdbf6a8440393a2b5d8240d1872 already in progress: creating tablet
I20251025 07:52:41.862702 1659 raft_consensus.cc:515] T b772025852794a49b072f37cccbd3291 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.863190 1166 tablet_bootstrap.cc:654] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:41.866176 712 leader_election.cc:343] T c9f7ba3709c745d4a625449d6e1c8ba1 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:41.866293 712 leader_election.cc:304] T c9f7ba3709c745d4a625449d6e1c8ba1 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:41.866668 2420 raft_consensus.cc:2749] T c9f7ba3709c745d4a625449d6e1c8ba1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:41.867383 1659 leader_election.cc:290] T b772025852794a49b072f37cccbd3291 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
W20251025 07:52:41.868470 710 leader_election.cc:343] T b772025852794a49b072f37cccbd3291 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: b772025852794a49b072f37cccbd3291
W20251025 07:52:41.868567 712 leader_election.cc:343] T b772025852794a49b072f37cccbd3291 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: b772025852794a49b072f37cccbd3291
I20251025 07:52:41.868629 710 leader_election.cc:304] T b772025852794a49b072f37cccbd3291 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:41.868932 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 76) took 3302 ms. Trace:
I20251025 07:52:41.869033 1659 raft_consensus.cc:2749] T b772025852794a49b072f37cccbd3291 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 07:52:41.869009 2166 rpcz_store.cc:276] 1025 07:52:38.566582 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.566631 (+ 49us) service_pool.cc:225] Handling call
1025 07:52:41.868816 (+3302185us) ts_tablet_manager.cc:1151] Deleting tablet d77704b4fe6c4456a2ba40ffec0c60fc
1025 07:52:41.868826 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.868913 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3302021}
I20251025 07:52:41.869822 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 77) took 3302 ms. Trace:
W20251025 07:52:41.869818 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet d77704b4fe6c4456a2ba40ffec0c60fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d77704b4fe6c4456a2ba40ffec0c60fc already in progress: creating tablet
I20251025 07:52:41.870007 2166 rpcz_store.cc:276] 1025 07:52:38.567288 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.567334 (+ 46us) service_pool.cc:225] Handling call
1025 07:52:41.869666 (+3302332us) ts_tablet_manager.cc:1151] Deleting tablet e4da89f863cc43f3bf6df908bb9dc6bf
1025 07:52:41.869675 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.869808 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3302140}
I20251025 07:52:41.870651 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 78) took 3301 ms. Trace:
I20251025 07:52:41.870860 2166 rpcz_store.cc:276] 1025 07:52:38.568820 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.568864 (+ 44us) service_pool.cc:225] Handling call
1025 07:52:41.870494 (+3301630us) ts_tablet_manager.cc:1151] Deleting tablet e47c4f741f354a19b49777bd06e54764
1025 07:52:41.870505 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.870636 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3301457}
I20251025 07:52:41.871454 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 79) took 3301 ms. Trace:
I20251025 07:52:41.871625 2166 rpcz_store.cc:276] 1025 07:52:38.569573 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.569614 (+ 41us) service_pool.cc:225] Handling call
1025 07:52:41.871301 (+3301687us) ts_tablet_manager.cc:1151] Deleting tablet 72dc084f8efc45478771c247e07a16f4
1025 07:52:41.871313 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.871436 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3301507}
I20251025 07:52:41.872134 2166 tablet_replica.cc:333] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:41.872594 2166 raft_consensus.cc:2243] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:41.872709 1166 tablet_bootstrap.cc:492] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:41.873265 2166 raft_consensus.cc:2272] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:41.874244 2166 ts_tablet_manager.cc:1916] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:41.877614 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet e4da89f863cc43f3bf6df908bb9dc6bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4da89f863cc43f3bf6df908bb9dc6bf already in progress: creating tablet
I20251025 07:52:41.877588 1838 consensus_queue.cc:1048] T f97acc9a638b4d5a968dcc3412a384bc P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251025 07:52:41.877895 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet e47c4f741f354a19b49777bd06e54764 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e47c4f741f354a19b49777bd06e54764 already in progress: creating tablet
W20251025 07:52:41.878180 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 72dc084f8efc45478771c247e07a16f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72dc084f8efc45478771c247e07a16f4 already in progress: creating tablet
I20251025 07:52:41.872911 1166 ts_tablet_manager.cc:1403] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.004s
I20251025 07:52:41.881377 1166 raft_consensus.cc:359] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:41.881587 1166 raft_consensus.cc:385] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:41.881659 1166 raft_consensus.cc:740] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:41.881973 1166 consensus_queue.cc:260] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:41.883380 1166 ts_tablet_manager.cc:1434] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251025 07:52:41.883772 1166 tablet_bootstrap.cc:492] T 1912e59548874e0e8e989050ec0064c5 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:41.885704 1166 tablet_bootstrap.cc:654] T 1912e59548874e0e8e989050ec0064c5 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:41.890599 2146 ts_tablet_manager.cc:1916] T 5fca357d7e24494fb6af4bd73d98833c P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:41.893009 1096 heartbeater.cc:344] Connected to a master server at 127.0.104.61:42203
I20251025 07:52:41.894874 1167 tablet_bootstrap.cc:654] T 0f5f6ca6c3cd4d369690e425387a4344 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:41.895015 2420 raft_consensus.cc:493] T 26c849d376d143c6bef0c8e0e1d7d03a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:41.895152 2420 raft_consensus.cc:515] T 26c849d376d143c6bef0c8e0e1d7d03a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:41.895853 2420 leader_election.cc:290] T 26c849d376d143c6bef0c8e0e1d7d03a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:41.896324 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "26c849d376d143c6bef0c8e0e1d7d03a" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:41.896821 710 leader_election.cc:343] T 26c849d376d143c6bef0c8e0e1d7d03a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:41.897388 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "26c849d376d143c6bef0c8e0e1d7d03a" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:41.902331 1166 tablet_bootstrap.cc:492] T 1912e59548874e0e8e989050ec0064c5 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:41.902460 1166 ts_tablet_manager.cc:1403] T 1912e59548874e0e8e989050ec0064c5 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.001s
I20251025 07:52:41.904176 1166 raft_consensus.cc:359] T 1912e59548874e0e8e989050ec0064c5 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:41.904345 1166 raft_consensus.cc:385] T 1912e59548874e0e8e989050ec0064c5 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:41.904395 1166 raft_consensus.cc:740] T 1912e59548874e0e8e989050ec0064c5 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:41.904806 1166 consensus_queue.cc:260] T 1912e59548874e0e8e989050ec0064c5 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:41.906244 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 2784bbfb97d5466f9ec856add7d0b29a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.906464 745 tablet_service.cc:1552] Processing DeleteTablet for tablet fd265cdbf6a8440393a2b5d8240d1872 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.906615 756 tablet_service.cc:1552] Processing DeleteTablet for tablet d77704b4fe6c4456a2ba40ffec0c60fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.906764 755 tablet_service.cc:1552] Processing DeleteTablet for tablet e4da89f863cc43f3bf6df908bb9dc6bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.906939 741 tablet_service.cc:1552] Processing DeleteTablet for tablet e47c4f741f354a19b49777bd06e54764 with delete_type TABLET_DATA_DELETED (replaced by tablet 755f0d6c87044e20a8020f8410940123) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.907090 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 72dc084f8efc45478771c247e07a16f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.907236 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 716fb61312e94a4d9eb8ab8336ca140c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.907368 753 tablet_service.cc:1552] Processing DeleteTablet for tablet bef0ef23d90d480d9c99aef21d7271fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.907521 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f0c4f0e8f014e59be9e46d2afe0be9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.907954 1166 ts_tablet_manager.cc:1434] T 1912e59548874e0e8e989050ec0064c5 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251025 07:52:41.908257 1166 tablet_bootstrap.cc:492] T bc2d16b33ca8426abbd7bd38339c9f1d P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
W20251025 07:52:41.908480 712 leader_election.cc:343] T 26c849d376d143c6bef0c8e0e1d7d03a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:41.908593 712 leader_election.cc:304] T 26c849d376d143c6bef0c8e0e1d7d03a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:41.908995 751 tablet_service.cc:1552] Processing DeleteTablet for tablet f661d5909657455581dcb5e6eadd3c0b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:41.909369 1659 raft_consensus.cc:2749] T 26c849d376d143c6bef0c8e0e1d7d03a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:41.918938 1166 tablet_bootstrap.cc:654] T bc2d16b33ca8426abbd7bd38339c9f1d P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:41.928489 982 consensus_peers.cc:597] T 6ec421eb820b4f40afcff5a5bf80c508 P db42be7ad3c741a0aa427f27e245993b -> Peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Couldn't send request to peer 23eeffd3de254ce18db5febfde971c6b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6ec421eb820b4f40afcff5a5bf80c508. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:41.931241 1167 tablet_bootstrap.cc:492] T 0f5f6ca6c3cd4d369690e425387a4344 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:41.931494 1167 ts_tablet_manager.cc:1403] T 0f5f6ca6c3cd4d369690e425387a4344 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.108s user 0.004s sys 0.000s
I20251025 07:52:41.932221 1169 tablet_bootstrap.cc:492] T b3a0af893a8947848e12093d0a7f0229 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:41.932416 1169 ts_tablet_manager.cc:1403] T b3a0af893a8947848e12093d0a7f0229 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.077s user 0.005s sys 0.003s
I20251025 07:52:41.932590 1167 raft_consensus.cc:359] T 0f5f6ca6c3cd4d369690e425387a4344 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:41.932832 1167 raft_consensus.cc:385] T 0f5f6ca6c3cd4d369690e425387a4344 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:41.932907 1167 raft_consensus.cc:740] T 0f5f6ca6c3cd4d369690e425387a4344 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:41.933228 1167 consensus_queue.cc:260] T 0f5f6ca6c3cd4d369690e425387a4344 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:41.934090 1167 ts_tablet_manager.cc:1434] T 0f5f6ca6c3cd4d369690e425387a4344 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:41.934121 1169 raft_consensus.cc:359] T b3a0af893a8947848e12093d0a7f0229 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:41.934500 1169 raft_consensus.cc:385] T b3a0af893a8947848e12093d0a7f0229 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:41.934644 1169 raft_consensus.cc:740] T b3a0af893a8947848e12093d0a7f0229 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:41.935006 1169 consensus_queue.cc:260] T b3a0af893a8947848e12093d0a7f0229 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:41.935971 1169 ts_tablet_manager.cc:1434] T b3a0af893a8947848e12093d0a7f0229 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251025 07:52:41.936699 1169 tablet_bootstrap.cc:492] T 0325d462441349ef81841f1b0cab2a20 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:41.937304 2146 ts_tablet_manager.cc:1929] T 5fca357d7e24494fb6af4bd73d98833c P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:41.937448 2146 log.cc:1199] T 5fca357d7e24494fb6af4bd73d98833c P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/5fca357d7e24494fb6af4bd73d98833c
I20251025 07:52:41.937886 2146 ts_tablet_manager.cc:1950] T 5fca357d7e24494fb6af4bd73d98833c P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:41.938875 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 109) took 3467 ms. Trace:
I20251025 07:52:41.939004 2146 rpcz_store.cc:276] 1025 07:52:38.471802 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.498190 (+ 26388us) service_pool.cc:225] Handling call
1025 07:52:41.828637 (+3330447us) ts_tablet_manager.cc:1151] Deleting tablet 5fca357d7e24494fb6af4bd73d98833c
1025 07:52:41.828650 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.915499 (+ 86849us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.937155 (+ 21656us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.938847 (+ 1692us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3308828}
I20251025 07:52:41.939530 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 111) took 3440 ms. Trace:
I20251025 07:52:41.939625 2146 rpcz_store.cc:276] 1025 07:52:38.498638 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.498684 (+ 46us) service_pool.cc:225] Handling call
1025 07:52:41.939376 (+3440692us) ts_tablet_manager.cc:1151] Deleting tablet 2b594d4718344aafbdaec34040695589
1025 07:52:41.939389 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.939512 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3419507}
I20251025 07:52:41.942736 1167 tablet_bootstrap.cc:492] T 36c8fcafd5464b4e999276a8062a812a P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:41.944571 1167 tablet_bootstrap.cc:654] T 36c8fcafd5464b4e999276a8062a812a P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:41.941347 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 5fca357d7e24494fb6af4bd73d98833c (table default.Table_400f06d166054eeb97890ad56453afb2 [id=5f20de141a6f452a82ed736f41c29d9b]) successfully deleted
W20251025 07:52:41.946774 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 2b594d4718344aafbdaec34040695589 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b594d4718344aafbdaec34040695589 already in progress: creating tablet
W20251025 07:52:41.940436 454 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:35777 (request call id 192) took 2129 ms (client timeout 1000 ms). Trace:
W20251025 07:52:41.947150 454 rpcz_store.cc:269] 1025 07:52:39.810871 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:40.409097 (+598226us) service_pool.cc:225] Handling call
1025 07:52:41.940372 (+1531275us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:41.226070 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:41.226238 (+ 168us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:41.226247 (+ 9us) write_op.cc:435] Acquired schema lock
1025 07:52:41.226251 (+ 4us) tablet.cc:660] Decoding operations
1025 07:52:41.226379 (+ 128us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:41.226396 (+ 17us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:41.226402 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:41.226468 (+ 66us) tablet.cc:699] Row locks acquired
1025 07:52:41.226470 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:41.226508 (+ 38us) write_op.cc:270] Start()
1025 07:52:41.226547 (+ 39us) write_op.cc:276] Timestamp: P: 1761378761226492 usec, L: 0
1025 07:52:41.226553 (+ 6us) op_driver.cc:348] REPLICATION: starting
1025 07:52:41.226751 (+ 198us) log.cc:844] Serialized 356 byte log entry
1025 07:52:41.230450 (+ 3699us) op_driver.cc:464] REPLICATION: finished
1025 07:52:41.230603 (+ 153us) write_op.cc:301] APPLY: starting
1025 07:52:41.230624 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:41.230677 (+ 53us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:41.230733 (+ 56us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:41.230802 (+ 69us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:41.230852 (+ 50us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:41.230863 (+ 11us) write_op.cc:312] APPLY: finished
1025 07:52:41.230964 (+ 101us) log.cc:844] Serialized 23 byte log entry
1025 07:52:41.231029 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:41.231050 (+ 21us) write_op.cc:454] Released schema lock
1025 07:52:41.231056 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:41.231120 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:41.231125 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":72917,"reported_tablets":257,"child_traces":[["op",{"apply.queue_time_us":95,"apply.run_cpu_time_us":514,"apply.run_wall_time_us":514,"num_ops":1,"prepare.queue_time_us":1885,"prepare.run_cpu_time_us":815,"prepare.run_wall_time_us":815,"raft.queue_time_us":497,"raft.run_cpu_time_us":315,"raft.run_wall_time_us":316,"replication_time_us":3842}]]}
I20251025 07:52:41.948318 454 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_f29b6c4e31ef483baa4d57209ac4968b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:41.949090 454 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f29b6c4e31ef483baa4d57209ac4968b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:41.939987 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 112) took 3441 ms. Trace:
I20251025 07:52:41.956897 961 heartbeater.cc:344] Connected to a master server at 127.0.104.61:42203
I20251025 07:52:41.957202 454 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 63) took 3356 ms. Trace:
I20251025 07:52:41.953119 2146 rpcz_store.cc:276] 1025 07:52:38.498953 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.499353 (+ 400us) service_pool.cc:225] Handling call
1025 07:52:41.939883 (+3440530us) ts_tablet_manager.cc:1151] Deleting tablet 570201e7bf05406a97c81dde071d894b
1025 07:52:41.939891 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.939973 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3417311}
W20251025 07:52:41.958653 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 570201e7bf05406a97c81dde071d894b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 570201e7bf05406a97c81dde071d894b already in progress: creating tablet
I20251025 07:52:41.959100 2146 tablet_replica.cc:333] T a13bf4d7eb1440f7a28b85cb1bf99667 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:41.959422 2146 raft_consensus.cc:2243] T a13bf4d7eb1440f7a28b85cb1bf99667 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:41.957468 454 rpcz_store.cc:276] 1025 07:52:38.600736 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.948084 (+3347348us) service_pool.cc:225] Handling call
1025 07:52:41.949273 (+ 1189us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:41.949811 (+ 538us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:41.956833 (+ 7022us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:41.956966 (+ 133us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:41.957150 (+ 184us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:41.950462 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:41.950559 (+ 97us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:41.950565 (+ 6us) write_op.cc:435] Acquired schema lock
1025 07:52:41.950568 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:41.950641 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:41.950653 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:41.950657 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:41.950743 (+ 86us) tablet.cc:699] Row locks acquired
1025 07:52:41.950746 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:41.950781 (+ 35us) write_op.cc:270] Start()
1025 07:52:41.950808 (+ 27us) write_op.cc:276] Timestamp: P: 1761378761950760 usec, L: 0
1025 07:52:41.950812 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:41.951002 (+ 190us) log.cc:844] Serialized 433 byte log entry
1025 07:52:41.954561 (+ 3559us) op_driver.cc:464] REPLICATION: finished
1025 07:52:41.954696 (+ 135us) write_op.cc:301] APPLY: starting
1025 07:52:41.954712 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:41.954756 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:41.954791 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:41.954887 (+ 96us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:41.954941 (+ 54us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:41.954948 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:41.955024 (+ 76us) log.cc:844] Serialized 29 byte log entry
1025 07:52:41.955098 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:41.955118 (+ 20us) write_op.cc:454] Released schema lock
1025 07:52:41.955125 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:41.955206 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:41.955210 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":87,"apply.run_cpu_time_us":502,"apply.run_wall_time_us":502,"num_ops":2,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":632,"prepare.run_wall_time_us":634,"raft.queue_time_us":63,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":247,"replication_time_us":3716}]]}
I20251025 07:52:41.959954 2146 raft_consensus.cc:2272] T a13bf4d7eb1440f7a28b85cb1bf99667 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:41.960592 454 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_84b2dff61e654506ab500ed4e4354244"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:41.961419 2146 ts_tablet_manager.cc:1916] T a13bf4d7eb1440f7a28b85cb1bf99667 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:41.962004 454 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84b2dff61e654506ab500ed4e4354244 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:41.965407 1169 tablet_bootstrap.cc:654] T 0325d462441349ef81841f1b0cab2a20 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:41.966378 2420 raft_consensus.cc:493] T 050a5d3b5f934dd19bf2e221f2fe0780 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:41.966531 2420 raft_consensus.cc:515] T 050a5d3b5f934dd19bf2e221f2fe0780 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:41.967170 2420 leader_election.cc:290] T 050a5d3b5f934dd19bf2e221f2fe0780 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:41.970671 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "050a5d3b5f934dd19bf2e221f2fe0780" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:41.974452 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "050a5d3b5f934dd19bf2e221f2fe0780" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:41.977581 2348 raft_consensus.cc:493] T 7c7a9d7a4fb3449bb57eafa83230a839 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:41.977770 2348 raft_consensus.cc:515] T 7c7a9d7a4fb3449bb57eafa83230a839 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.978492 2348 leader_election.cc:290] T 7c7a9d7a4fb3449bb57eafa83230a839 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:41.982362 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c7a9d7a4fb3449bb57eafa83230a839" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:41.983201 712 leader_election.cc:343] T 050a5d3b5f934dd19bf2e221f2fe0780 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:41.984062 823 heartbeater.cc:344] Connected to a master server at 127.0.104.61:42203
W20251025 07:52:41.984337 710 leader_election.cc:343] T 050a5d3b5f934dd19bf2e221f2fe0780 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:41.984476 710 leader_election.cc:304] T 050a5d3b5f934dd19bf2e221f2fe0780 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:41.985383 2105 raft_consensus.cc:493] T a4e4b99be0434baaa75d988281797846 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:41.985545 2105 raft_consensus.cc:515] T a4e4b99be0434baaa75d988281797846 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:41.986317 2105 leader_election.cc:290] T a4e4b99be0434baaa75d988281797846 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:41.986855 2117 raft_consensus.cc:2749] T 050a5d3b5f934dd19bf2e221f2fe0780 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:41.987783 1169 tablet_bootstrap.cc:492] T 0325d462441349ef81841f1b0cab2a20 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:41.987984 1169 ts_tablet_manager.cc:1403] T 0325d462441349ef81841f1b0cab2a20 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.000s
I20251025 07:52:41.992424 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c7a9d7a4fb3449bb57eafa83230a839" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:41.992702 768 raft_consensus.cc:2393] T 7c7a9d7a4fb3449bb57eafa83230a839 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:41.992586 2166 ts_tablet_manager.cc:1929] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:41.993929 982 leader_election.cc:304] T 7c7a9d7a4fb3449bb57eafa83230a839 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:41.995096 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e4b99be0434baaa75d988281797846" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:41.995754 850 leader_election.cc:343] T a4e4b99be0434baaa75d988281797846 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:41.997139 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e4b99be0434baaa75d988281797846" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:41.998431 775 raft_consensus.cc:2468] T a4e4b99be0434baaa75d988281797846 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:41.998029 2105 consensus_queue.cc:1048] T ebf3a2a926754305b5bd95fcebb25d29 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:41.994349 2166 log.cc:1199] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/be5ab0db6ac34c2c9a370445e251eb46
I20251025 07:52:41.999820 2348 raft_consensus.cc:3060] T 7c7a9d7a4fb3449bb57eafa83230a839 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.002885 847 leader_election.cc:304] T a4e4b99be0434baaa75d988281797846 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.003499 2105 raft_consensus.cc:2804] T a4e4b99be0434baaa75d988281797846 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:42.003598 2105 raft_consensus.cc:493] T a4e4b99be0434baaa75d988281797846 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:42.003677 2105 raft_consensus.cc:3060] T a4e4b99be0434baaa75d988281797846 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.013515 2348 raft_consensus.cc:2749] T 7c7a9d7a4fb3449bb57eafa83230a839 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:42.016669 2105 raft_consensus.cc:515] T a4e4b99be0434baaa75d988281797846 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:42.017115 1095 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
I20251025 07:52:42.021449 2105 leader_election.cc:290] T a4e4b99be0434baaa75d988281797846 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:42.006908 2117 raft_consensus.cc:493] T b2044ca206634eb285197af4ba2073e7 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:42.035465 2166 ts_tablet_manager.cc:1950] T be5ab0db6ac34c2c9a370445e251eb46 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:42.035518 2117 raft_consensus.cc:515] T b2044ca206634eb285197af4ba2073e7 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.035812 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 80) took 3465 ms. Trace:
I20251025 07:52:42.035893 2166 rpcz_store.cc:276] 1025 07:52:38.570443 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.570487 (+ 44us) service_pool.cc:225] Handling call
1025 07:52:41.872061 (+3301574us) ts_tablet_manager.cc:1151] Deleting tablet be5ab0db6ac34c2c9a370445e251eb46
1025 07:52:41.872074 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:41.945170 (+ 73096us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:41.990964 (+ 45794us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.035794 (+ 44830us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3301415}
I20251025 07:52:42.036341 2117 leader_election.cc:290] T b2044ca206634eb285197af4ba2073e7 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:42.036386 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet be5ab0db6ac34c2c9a370445e251eb46 (table default.Table_8faecb0cb37949a39f4ef5d9d2740c3d [id=f03dd08b39184812884db77faf4bb39e]) successfully deleted
I20251025 07:52:42.036928 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2044ca206634eb285197af4ba2073e7" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:42.038051 712 leader_election.cc:343] T b2044ca206634eb285197af4ba2073e7 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.020090 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e4b99be0434baaa75d988281797846" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:42.038383 768 raft_consensus.cc:3060] T a4e4b99be0434baaa75d988281797846 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.030045 1167 tablet_bootstrap.cc:492] T 36c8fcafd5464b4e999276a8062a812a P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:42.038633 1167 ts_tablet_manager.cc:1403] T 36c8fcafd5464b4e999276a8062a812a P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.096s user 0.002s sys 0.003s
I20251025 07:52:42.039397 1167 raft_consensus.cc:359] T 36c8fcafd5464b4e999276a8062a812a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.039546 1167 raft_consensus.cc:385] T 36c8fcafd5464b4e999276a8062a812a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.039597 1167 raft_consensus.cc:740] T 36c8fcafd5464b4e999276a8062a812a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.039806 1167 consensus_queue.cc:260] T 36c8fcafd5464b4e999276a8062a812a P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.041496 1167 ts_tablet_manager.cc:1434] T 36c8fcafd5464b4e999276a8062a812a P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20251025 07:52:42.042085 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4e4b99be0434baaa75d988281797846" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:42.042649 1166 tablet_bootstrap.cc:492] T bc2d16b33ca8426abbd7bd38339c9f1d P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
W20251025 07:52:42.042707 850 leader_election.cc:343] T a4e4b99be0434baaa75d988281797846 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.042791 1166 ts_tablet_manager.cc:1403] T bc2d16b33ca8426abbd7bd38339c9f1d P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.135s user 0.026s sys 0.066s
I20251025 07:52:42.041757 768 raft_consensus.cc:2468] T a4e4b99be0434baaa75d988281797846 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:42.043608 847 leader_election.cc:304] T a4e4b99be0434baaa75d988281797846 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.043561 1166 raft_consensus.cc:359] T bc2d16b33ca8426abbd7bd38339c9f1d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.043717 1166 raft_consensus.cc:385] T bc2d16b33ca8426abbd7bd38339c9f1d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.043763 1166 raft_consensus.cc:740] T bc2d16b33ca8426abbd7bd38339c9f1d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.043989 1166 consensus_queue.cc:260] T bc2d16b33ca8426abbd7bd38339c9f1d P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.044127 2106 consensus_queue.cc:1048] T ebf3a2a926754305b5bd95fcebb25d29 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:42.044430 1166 ts_tablet_manager.cc:1434] T bc2d16b33ca8426abbd7bd38339c9f1d P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 07:52:42.044697 2106 raft_consensus.cc:2804] T a4e4b99be0434baaa75d988281797846 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:42.044786 2106 raft_consensus.cc:697] T a4e4b99be0434baaa75d988281797846 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:42.044967 2106 consensus_queue.cc:237] T a4e4b99be0434baaa75d988281797846 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:42.041778 1167 tablet_bootstrap.cc:492] T 1912e59548874e0e8e989050ec0064c5 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:42.047024 1166 tablet_bootstrap.cc:492] T 04d27d54fdb7483b869408af65ff85c9 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:42.047224 1169 raft_consensus.cc:359] T 0325d462441349ef81841f1b0cab2a20 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.047391 1169 raft_consensus.cc:385] T 0325d462441349ef81841f1b0cab2a20 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.047598 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2044ca206634eb285197af4ba2073e7" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:42.047436 1169 raft_consensus.cc:740] T 0325d462441349ef81841f1b0cab2a20 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.049019 1166 tablet_bootstrap.cc:654] T 04d27d54fdb7483b869408af65ff85c9 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:42.050717 710 leader_election.cc:343] T b2044ca206634eb285197af4ba2073e7 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.050855 710 leader_election.cc:304] T b2044ca206634eb285197af4ba2073e7 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.051373 2117 raft_consensus.cc:2749] T b2044ca206634eb285197af4ba2073e7 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:42.048159 1169 consensus_queue.cc:260] T 0325d462441349ef81841f1b0cab2a20 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.052753 2166 tablet_replica.cc:333] T 7f29b64704e147aa9b0b5f6cda860f09 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:42.055124 2166 raft_consensus.cc:2243] T 7f29b64704e147aa9b0b5f6cda860f09 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:42.057245 1169 ts_tablet_manager.cc:1434] T 0325d462441349ef81841f1b0cab2a20 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.069s user 0.003s sys 0.000s
I20251025 07:52:42.058396 2166 raft_consensus.cc:2272] T 7f29b64704e147aa9b0b5f6cda860f09 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:42.063081 2166 ts_tablet_manager.cc:1916] T 7f29b64704e147aa9b0b5f6cda860f09 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:42.069839 1167 tablet_bootstrap.cc:654] T 1912e59548874e0e8e989050ec0064c5 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:42.073004 983 consensus_peers.cc:597] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5fca357d7e24494fb6af4bd73d98833c. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:42.077415 1167 tablet_bootstrap.cc:492] T 1912e59548874e0e8e989050ec0064c5 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:42.077646 1167 ts_tablet_manager.cc:1403] T 1912e59548874e0e8e989050ec0064c5 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.036s user 0.001s sys 0.005s
I20251025 07:52:42.078406 1167 raft_consensus.cc:359] T 1912e59548874e0e8e989050ec0064c5 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.078634 1167 raft_consensus.cc:385] T 1912e59548874e0e8e989050ec0064c5 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.078717 1167 raft_consensus.cc:740] T 1912e59548874e0e8e989050ec0064c5 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.079012 1167 consensus_queue.cc:260] T 1912e59548874e0e8e989050ec0064c5 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.083420 1166 tablet_bootstrap.cc:492] T 04d27d54fdb7483b869408af65ff85c9 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:42.083575 1166 ts_tablet_manager.cc:1403] T 04d27d54fdb7483b869408af65ff85c9 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.037s user 0.006s sys 0.007s
I20251025 07:52:42.084357 1166 raft_consensus.cc:359] T 04d27d54fdb7483b869408af65ff85c9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:42.084533 1166 raft_consensus.cc:385] T 04d27d54fdb7483b869408af65ff85c9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.084589 1166 raft_consensus.cc:740] T 04d27d54fdb7483b869408af65ff85c9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.084779 1166 consensus_queue.cc:260] T 04d27d54fdb7483b869408af65ff85c9 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:42.085059 1167 ts_tablet_manager.cc:1434] T 1912e59548874e0e8e989050ec0064c5 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251025 07:52:42.085261 1166 ts_tablet_manager.cc:1434] T 04d27d54fdb7483b869408af65ff85c9 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 07:52:42.085520 1166 tablet_bootstrap.cc:492] T 292fe04055954afd8c08ac2636b68191 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:42.086115 1167 tablet_bootstrap.cc:492] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:42.087498 1166 tablet_bootstrap.cc:654] T 292fe04055954afd8c08ac2636b68191 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:42.090929 2106 consensus_queue.cc:1048] T 88eb51447e0f4080baa49dd902deb24f P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:42.101233 1167 tablet_bootstrap.cc:654] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:42.102938 822 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
I20251025 07:52:42.114104 1169 tablet_bootstrap.cc:492] T c2713942584e4734acbc0775ead772fa P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:42.118044 960 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
W20251025 07:52:42.124163 710 consensus_peers.cc:597] T 943ec703e65f4fd3b0c690e08c0bd226 P 23eeffd3de254ce18db5febfde971c6b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 943ec703e65f4fd3b0c690e08c0bd226. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:42.126644 2166 ts_tablet_manager.cc:1929] T 7f29b64704e147aa9b0b5f6cda860f09 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:42.127122 2117 raft_consensus.cc:493] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:42.127194 2166 log.cc:1199] T 7f29b64704e147aa9b0b5f6cda860f09 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/7f29b64704e147aa9b0b5f6cda860f09
I20251025 07:52:42.127274 2117 raft_consensus.cc:515] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.128234 2117 leader_election.cc:290] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:42.128796 2166 ts_tablet_manager.cc:1950] T 7f29b64704e147aa9b0b5f6cda860f09 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
W20251025 07:52:42.129418 712 leader_election.cc:343] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 2aa522477e934c89834026dc480fe7c0
I20251025 07:52:42.129984 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 81) took 3557 ms. Trace:
I20251025 07:52:42.130384 2166 rpcz_store.cc:276] 1025 07:52:38.572239 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.572284 (+ 45us) service_pool.cc:225] Handling call
1025 07:52:42.052228 (+3479944us) ts_tablet_manager.cc:1151] Deleting tablet 7f29b64704e147aa9b0b5f6cda860f09
1025 07:52:42.052469 (+ 241us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:42.092905 (+ 40436us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.126240 (+ 33335us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.129965 (+ 3725us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3479437}
I20251025 07:52:42.135016 1169 tablet_bootstrap.cc:654] T c2713942584e4734acbc0775ead772fa P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:42.136315 710 leader_election.cc:343] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 2aa522477e934c89834026dc480fe7c0
I20251025 07:52:42.136443 710 leader_election.cc:304] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.137331 2314 consensus_queue.cc:1048] T 7c7a9d7a4fb3449bb57eafa83230a839 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:42.138367 2117 raft_consensus.cc:2749] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:42.147682 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 82) took 3550 ms. Trace:
I20251025 07:52:42.148593 2314 raft_consensus.cc:493] T 1912e59548874e0e8e989050ec0064c5 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:42.148758 2314 raft_consensus.cc:515] T 1912e59548874e0e8e989050ec0064c5 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.149436 2314 leader_election.cc:290] T 1912e59548874e0e8e989050ec0064c5 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:42.149535 2166 rpcz_store.cc:276] 1025 07:52:38.596754 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.596810 (+ 56us) service_pool.cc:225] Handling call
1025 07:52:42.131705 (+3534895us) ts_tablet_manager.cc:1151] Deleting tablet 95efd553493e4966bc48a01cac6e1e10
1025 07:52:42.131716 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:42.147659 (+ 15943us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3527071}
I20251025 07:52:42.150192 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1912e59548874e0e8e989050ec0064c5" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:42.150414 2166 tablet_replica.cc:333] T 4f25c9caa05b4da4bd56a8a21fde423c P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:42.150671 2166 raft_consensus.cc:2243] T 4f25c9caa05b4da4bd56a8a21fde423c P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Raft consensus shutting down.
W20251025 07:52:42.150839 850 leader_election.cc:343] T 1912e59548874e0e8e989050ec0064c5 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.151059 2166 raft_consensus.cc:2272] T 4f25c9caa05b4da4bd56a8a21fde423c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:42.151710 2166 ts_tablet_manager.cc:1916] T 4f25c9caa05b4da4bd56a8a21fde423c P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:42.151804 2314 consensus_queue.cc:1048] T 0325d462441349ef81841f1b0cab2a20 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:42.152796 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1912e59548874e0e8e989050ec0064c5" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:42.158913 773 raft_consensus.cc:2468] T 1912e59548874e0e8e989050ec0064c5 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
W20251025 07:52:42.160292 712 consensus_peers.cc:597] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac0ea298bc114c56a8093e12def21ce1. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:42.161433 847 leader_election.cc:304] T 1912e59548874e0e8e989050ec0064c5 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.161856 2105 raft_consensus.cc:2804] T 1912e59548874e0e8e989050ec0064c5 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:42.162041 2105 raft_consensus.cc:493] T 1912e59548874e0e8e989050ec0064c5 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:42.162374 2105 raft_consensus.cc:3060] T 1912e59548874e0e8e989050ec0064c5 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.164520 2314 raft_consensus.cc:493] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:42.164638 2314 raft_consensus.cc:515] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.165197 2314 leader_election.cc:290] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:42.166103 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9bfd7c0c3ad4e5b98b6708793d61d41" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:42.166577 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9bfd7c0c3ad4e5b98b6708793d61d41" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:42.166740 773 raft_consensus.cc:2468] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
W20251025 07:52:42.167133 850 leader_election.cc:343] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.167550 847 leader_election.cc:304] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.167847 2105 raft_consensus.cc:515] T 1912e59548874e0e8e989050ec0064c5 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.167946 2314 raft_consensus.cc:2804] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:42.168051 2314 raft_consensus.cc:493] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:42.168138 2314 raft_consensus.cc:3060] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.168430 2105 leader_election.cc:290] T 1912e59548874e0e8e989050ec0064c5 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:42.169862 447 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5]). Expected version 1 got 0
I20251025 07:52:42.171422 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1912e59548874e0e8e989050ec0064c5" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:42.171904 850 leader_election.cc:343] T 1912e59548874e0e8e989050ec0064c5 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.174566 1167 tablet_bootstrap.cc:492] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:42.174738 1167 ts_tablet_manager.cc:1403] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.089s user 0.001s sys 0.008s
I20251025 07:52:42.174896 2314 raft_consensus.cc:515] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.175836 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9bfd7c0c3ad4e5b98b6708793d61d41" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:42.175978 1167 raft_consensus.cc:359] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.176141 1167 raft_consensus.cc:385] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.176184 1167 raft_consensus.cc:740] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
W20251025 07:52:42.176312 850 leader_election.cc:343] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.176560 1167 consensus_queue.cc:260] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.177368 2314 leader_election.cc:290] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:42.178223 1167 ts_tablet_manager.cc:1434] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251025 07:52:42.178232 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9bfd7c0c3ad4e5b98b6708793d61d41" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:42.178555 773 raft_consensus.cc:3060] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.181242 773 raft_consensus.cc:2468] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:42.182845 1167 tablet_bootstrap.cc:492] T f661d5909657455581dcb5e6eadd3c0b P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:42.185479 1167 tablet_bootstrap.cc:654] T f661d5909657455581dcb5e6eadd3c0b P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:42.187857 1169 tablet_bootstrap.cc:492] T c2713942584e4734acbc0775ead772fa P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:42.187907 454 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 64) took 3576 ms. Trace:
I20251025 07:52:42.188714 1169 ts_tablet_manager.cc:1403] T c2713942584e4734acbc0775ead772fa P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.075s user 0.004s sys 0.000s
I20251025 07:52:42.190677 1169 raft_consensus.cc:359] T c2713942584e4734acbc0775ead772fa P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.191022 847 leader_election.cc:304] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.191406 1169 raft_consensus.cc:385] T c2713942584e4734acbc0775ead772fa P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.191762 2146 ts_tablet_manager.cc:1929] T a13bf4d7eb1440f7a28b85cb1bf99667 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:42.191884 2146 log.cc:1199] T a13bf4d7eb1440f7a28b85cb1bf99667 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/a13bf4d7eb1440f7a28b85cb1bf99667
I20251025 07:52:42.191910 2314 raft_consensus.cc:2804] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:42.192160 2314 raft_consensus.cc:697] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:42.192365 2117 raft_consensus.cc:493] T 04d27d54fdb7483b869408af65ff85c9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:42.192498 2117 raft_consensus.cc:515] T 04d27d54fdb7483b869408af65ff85c9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:42.193465 1169 raft_consensus.cc:740] T c2713942584e4734acbc0775ead772fa P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.195428 2117 leader_election.cc:290] T 04d27d54fdb7483b869408af65ff85c9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:42.197925 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 7f29b64704e147aa9b0b5f6cda860f09 (table default.Table_2253f6700c2f4a3a9847023ba98b80ea [id=17c206c8e24647ab9f483d9944dd3191]) successfully deleted
W20251025 07:52:42.198343 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 95efd553493e4966bc48a01cac6e1e10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95efd553493e4966bc48a01cac6e1e10 already in progress: creating tablet
I20251025 07:52:42.198554 454 rpcz_store.cc:276] 1025 07:52:38.611872 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.960345 (+3348473us) service_pool.cc:225] Handling call
1025 07:52:41.965298 (+ 4953us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:42.181754 (+216456us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:42.187582 (+ 5828us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:42.187719 (+ 137us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:42.187841 (+ 122us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:42.182533 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:42.182667 (+ 134us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:42.182674 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:42.182676 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:42.182808 (+ 132us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:42.182823 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:42.182828 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:42.182943 (+ 115us) tablet.cc:699] Row locks acquired
1025 07:52:42.182948 (+ 5us) write_op.cc:260] PREPARE: finished
1025 07:52:42.182986 (+ 38us) write_op.cc:270] Start()
1025 07:52:42.183023 (+ 37us) write_op.cc:276] Timestamp: P: 1761378762182965 usec, L: 0
1025 07:52:42.183028 (+ 5us) op_driver.cc:348] REPLICATION: starting
1025 07:52:42.183206 (+ 178us) log.cc:844] Serialized 433 byte log entry
1025 07:52:42.186545 (+ 3339us) op_driver.cc:464] REPLICATION: finished
1025 07:52:42.186710 (+ 165us) write_op.cc:301] APPLY: starting
1025 07:52:42.186732 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:42.186791 (+ 59us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:42.186833 (+ 42us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:42.187002 (+ 169us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:42.187039 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:42.187046 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:42.187126 (+ 80us) log.cc:844] Serialized 29 byte log entry
1025 07:52:42.187190 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:42.187212 (+ 22us) write_op.cc:454] Released schema lock
1025 07:52:42.187218 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:42.187287 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:42.187292 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":89,"apply.run_cpu_time_us":573,"apply.run_wall_time_us":573,"num_ops":2,"prepare.queue_time_us":55,"prepare.run_cpu_time_us":779,"prepare.run_wall_time_us":778,"raft.queue_time_us":76,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":235,"replication_time_us":3086}]]}
I20251025 07:52:42.199846 454 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_af96c59e9f8646c897e9651a4f457d1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:42.200287 2314 consensus_queue.cc:237] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:42.200582 454 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af96c59e9f8646c897e9651a4f457d1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:42.194897 2467 raft_consensus.cc:493] T 88eb51447e0f4080baa49dd902deb24f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:42.201555 2467 raft_consensus.cc:515] T 88eb51447e0f4080baa49dd902deb24f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.202078 2467 leader_election.cc:290] T 88eb51447e0f4080baa49dd902deb24f P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:42.202215 2117 raft_consensus.cc:493] T 0b21b7da1e2f47829c3f6cdaee689e8a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3aea2869ff814cc7b142012faa274382)
I20251025 07:52:42.202338 2117 raft_consensus.cc:515] T 0b21b7da1e2f47829c3f6cdaee689e8a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.203018 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88eb51447e0f4080baa49dd902deb24f" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:42.196738 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04d27d54fdb7483b869408af65ff85c9" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:42.203933 712 leader_election.cc:343] T 04d27d54fdb7483b869408af65ff85c9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.204285 1169 consensus_queue.cc:260] T c2713942584e4734acbc0775ead772fa P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.204878 1169 ts_tablet_manager.cc:1434] T c2713942584e4734acbc0775ead772fa P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20251025 07:52:42.205164 1169 tablet_bootstrap.cc:492] T 36262ab8c6964dd684c01b4db6219a4f P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:42.205243 447 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3]). Expected version 1 got 0
I20251025 07:52:42.207074 1169 tablet_bootstrap.cc:654] T 36262ab8c6964dd684c01b4db6219a4f P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:42.207826 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04d27d54fdb7483b869408af65ff85c9" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:42.207986 909 raft_consensus.cc:2468] T 04d27d54fdb7483b869408af65ff85c9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:42.208577 454 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 65) took 3574 ms. Trace:
I20251025 07:52:42.208739 454 rpcz_store.cc:276] 1025 07:52:38.633846 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:42.199571 (+3565725us) service_pool.cc:225] Handling call
1025 07:52:42.200694 (+ 1123us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:42.201220 (+ 526us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:42.208341 (+ 7121us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:42.208448 (+ 107us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:42.208539 (+ 91us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:42.202062 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:42.202197 (+ 135us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:42.202203 (+ 6us) write_op.cc:435] Acquired schema lock
1025 07:52:42.202206 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:42.202301 (+ 95us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:42.202316 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:42.202320 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:42.202396 (+ 76us) tablet.cc:699] Row locks acquired
1025 07:52:42.202399 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:42.202431 (+ 32us) write_op.cc:270] Start()
1025 07:52:42.202465 (+ 34us) write_op.cc:276] Timestamp: P: 1761378762202411 usec, L: 0
1025 07:52:42.202487 (+ 22us) op_driver.cc:348] REPLICATION: starting
1025 07:52:42.202689 (+ 202us) log.cc:844] Serialized 433 byte log entry
1025 07:52:42.206261 (+ 3572us) op_driver.cc:464] REPLICATION: finished
1025 07:52:42.206394 (+ 133us) write_op.cc:301] APPLY: starting
1025 07:52:42.206412 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:42.206465 (+ 53us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:42.206493 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:42.206619 (+ 126us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:42.206683 (+ 64us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:42.206691 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:42.206778 (+ 87us) log.cc:844] Serialized 29 byte log entry
1025 07:52:42.206845 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:42.206888 (+ 43us) write_op.cc:454] Released schema lock
1025 07:52:42.206896 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:42.206974 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:42.206981 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":78,"apply.run_cpu_time_us":569,"apply.run_wall_time_us":569,"num_ops":2,"prepare.queue_time_us":233,"prepare.run_cpu_time_us":739,"prepare.run_wall_time_us":747,"raft.queue_time_us":461,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":265,"replication_time_us":3740}]]}
I20251025 07:52:42.209216 710 leader_election.cc:304] T 04d27d54fdb7483b869408af65ff85c9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.209709 1659 raft_consensus.cc:2804] T 04d27d54fdb7483b869408af65ff85c9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:42.209812 1659 raft_consensus.cc:493] T 04d27d54fdb7483b869408af65ff85c9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:42.209887 1659 raft_consensus.cc:3060] T 04d27d54fdb7483b869408af65ff85c9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.209781 454 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_d5f74e259f754aa49c24a74e75cdd23c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:42.210392 454 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5f74e259f754aa49c24a74e75cdd23c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:42.214002 2117 leader_election.cc:290] T 0b21b7da1e2f47829c3f6cdaee689e8a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:42.214715 447 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_a6c869949b744bc98efb735b0e73e7aa [id=bf77b5ff6f1048dd996d948437f9ff5e]). Expected version 2 got 0
I20251025 07:52:42.215343 1166 tablet_bootstrap.cc:492] T 292fe04055954afd8c08ac2636b68191 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:42.215473 1166 ts_tablet_manager.cc:1403] T 292fe04055954afd8c08ac2636b68191 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.130s user 0.018s sys 0.014s
I20251025 07:52:42.215453 1659 raft_consensus.cc:515] T 04d27d54fdb7483b869408af65ff85c9 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:42.216192 1166 raft_consensus.cc:359] T 292fe04055954afd8c08ac2636b68191 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.216338 1166 raft_consensus.cc:385] T 292fe04055954afd8c08ac2636b68191 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.216387 1166 raft_consensus.cc:740] T 292fe04055954afd8c08ac2636b68191 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.216550 1166 consensus_queue.cc:260] T 292fe04055954afd8c08ac2636b68191 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:42.217118 712 leader_election.cc:343] T 0b21b7da1e2f47829c3f6cdaee689e8a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 0b21b7da1e2f47829c3f6cdaee689e8a
I20251025 07:52:42.217811 1166 ts_tablet_manager.cc:1434] T 292fe04055954afd8c08ac2636b68191 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:42.217909 1659 leader_election.cc:290] T 04d27d54fdb7483b869408af65ff85c9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:42.217834 447 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f]). Expected version 1 got 0
I20251025 07:52:42.218094 1166 tablet_bootstrap.cc:492] T 36262ab8c6964dd684c01b4db6219a4f P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:42.218428 454 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 66) took 3573 ms. Trace:
W20251025 07:52:42.218604 710 leader_election.cc:343] T 0b21b7da1e2f47829c3f6cdaee689e8a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 0b21b7da1e2f47829c3f6cdaee689e8a
I20251025 07:52:42.218710 710 leader_election.cc:304] T 0b21b7da1e2f47829c3f6cdaee689e8a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.219007 1659 raft_consensus.cc:2749] T 0b21b7da1e2f47829c3f6cdaee689e8a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 07:52:42.218580 454 rpcz_store.cc:276] 1025 07:52:38.645452 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:42.209586 (+3564134us) service_pool.cc:225] Handling call
1025 07:52:42.210484 (+ 898us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:42.210984 (+ 500us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:42.218198 (+ 7214us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:42.218301 (+ 103us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:42.218392 (+ 91us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:42.211894 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:42.212042 (+ 148us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:42.212050 (+ 8us) write_op.cc:435] Acquired schema lock
1025 07:52:42.212054 (+ 4us) tablet.cc:660] Decoding operations
1025 07:52:42.212164 (+ 110us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:42.212183 (+ 19us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:42.212190 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:42.212279 (+ 89us) tablet.cc:699] Row locks acquired
1025 07:52:42.212283 (+ 4us) write_op.cc:260] PREPARE: finished
1025 07:52:42.212316 (+ 33us) write_op.cc:270] Start()
1025 07:52:42.212351 (+ 35us) write_op.cc:276] Timestamp: P: 1761378762212292 usec, L: 0
1025 07:52:42.212358 (+ 7us) op_driver.cc:348] REPLICATION: starting
1025 07:52:42.212548 (+ 190us) log.cc:844] Serialized 433 byte log entry
1025 07:52:42.215504 (+ 2956us) op_driver.cc:464] REPLICATION: finished
1025 07:52:42.215600 (+ 96us) write_op.cc:301] APPLY: starting
1025 07:52:42.215615 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:42.215654 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:42.215686 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:42.215776 (+ 90us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:42.215805 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:42.215812 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:42.215892 (+ 80us) log.cc:844] Serialized 29 byte log entry
1025 07:52:42.215946 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:42.215964 (+ 18us) write_op.cc:454] Released schema lock
1025 07:52:42.215970 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:42.216026 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:42.216029 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":53,"apply.run_cpu_time_us":419,"apply.run_wall_time_us":419,"num_ops":2,"prepare.queue_time_us":96,"prepare.run_cpu_time_us":760,"prepare.run_wall_time_us":759,"raft.queue_time_us":68,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":253,"replication_time_us":3120}]]}
I20251025 07:52:42.219702 454 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_c39c116c2e3648fd88670672a25a5481"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:42.220417 454 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c39c116c2e3648fd88670672a25a5481 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:42.222435 447 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7]). Expected version 1 got 0
I20251025 07:52:42.222734 447 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 872cb7512d104827b1f1f186d33a4a44 (table default.Table_d5c65dce8b1b4214ba519d964012bfc0 [id=d4be274b36a845a99a4b96b8846248fd]). Expected version 1 got 0
I20251025 07:52:42.222946 447 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet cb829649f73d4d3f8862d268c953bd86 (table default.Table_ba4b8cc7b930482690b7e192e0edff12 [id=7944225d3ce7439b881bfb9289d9f040]). Expected version 1 got 0
I20251025 07:52:42.223256 447 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 04d27d54fdb7483b869408af65ff85c9 (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]). Expected version 2 got 0
I20251025 07:52:42.224097 447 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 4c155aa7fa5b4d68bbab1918b8432372 (table default.Table_6c5a24747ba14352b0285a41ec2985a4 [id=28bbaad474164a8c8c97a76bb8081791]). Expected version 2 got 0
I20251025 07:52:42.224516 447 catalog_manager.cc:5649] T fe5033e5ff404a5ebc6c8443d7548327 P db42be7ad3c741a0aa427f27e245993b reported cstate change: term changed from 0 to 1, leader changed from <none> to db42be7ad3c741a0aa427f27e245993b (127.0.104.3). New cstate: current_term: 1 leader_uuid: "db42be7ad3c741a0aa427f27e245993b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: HEALTHY } } }
I20251025 07:52:42.229285 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88eb51447e0f4080baa49dd902deb24f" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:42.218364 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04d27d54fdb7483b869408af65ff85c9" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:42.232098 909 raft_consensus.cc:3060] T 04d27d54fdb7483b869408af65ff85c9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.225121 2106 consensus_queue.cc:1048] T 7c7a9d7a4fb3449bb57eafa83230a839 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:42.235913 909 raft_consensus.cc:2468] T 04d27d54fdb7483b869408af65ff85c9 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:42.236147 2166 ts_tablet_manager.cc:1929] T 4f25c9caa05b4da4bd56a8a21fde423c P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:42.236249 2166 log.cc:1199] T 4f25c9caa05b4da4bd56a8a21fde423c P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/4f25c9caa05b4da4bd56a8a21fde423c
I20251025 07:52:42.236584 2166 ts_tablet_manager.cc:1950] T 4f25c9caa05b4da4bd56a8a21fde423c P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:42.238857 1167 tablet_bootstrap.cc:492] T f661d5909657455581dcb5e6eadd3c0b P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:42.239032 1167 ts_tablet_manager.cc:1403] T f661d5909657455581dcb5e6eadd3c0b P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.056s user 0.008s sys 0.000s
I20251025 07:52:42.239894 1167 raft_consensus.cc:359] T f661d5909657455581dcb5e6eadd3c0b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.240090 1167 raft_consensus.cc:385] T f661d5909657455581dcb5e6eadd3c0b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.240177 1167 raft_consensus.cc:740] T f661d5909657455581dcb5e6eadd3c0b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.240383 1167 consensus_queue.cc:260] T f661d5909657455581dcb5e6eadd3c0b P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.240907 1167 ts_tablet_manager.cc:1434] T f661d5909657455581dcb5e6eadd3c0b P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:42.241189 1167 tablet_bootstrap.cc:492] T bdaeedbcf35a4aad97f9ef192991978a P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:42.242833 2117 raft_consensus.cc:493] T 30418858bf95438686bdfe64ac733144 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:42.242921 1167 tablet_bootstrap.cc:654] T bdaeedbcf35a4aad97f9ef192991978a P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:42.243086 2117 raft_consensus.cc:515] T 30418858bf95438686bdfe64ac733144 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.231740 454 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 67) took 3538 ms. Trace:
I20251025 07:52:42.244160 2117 leader_election.cc:290] T 30418858bf95438686bdfe64ac733144 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:42.244706 454 rpcz_store.cc:276] 1025 07:52:38.693576 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:42.219512 (+3525936us) service_pool.cc:225] Handling call
1025 07:52:42.220550 (+ 1038us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:42.225278 (+ 4728us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:42.231506 (+ 6228us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:42.231612 (+ 106us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:42.231700 (+ 88us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:42.225835 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:42.225965 (+ 130us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:42.225972 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:42.225975 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:42.226074 (+ 99us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:42.226090 (+ 16us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:42.226096 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:42.226206 (+ 110us) tablet.cc:699] Row locks acquired
1025 07:52:42.226209 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:42.226243 (+ 34us) write_op.cc:270] Start()
1025 07:52:42.226278 (+ 35us) write_op.cc:276] Timestamp: P: 1761378762226223 usec, L: 0
1025 07:52:42.226283 (+ 5us) op_driver.cc:348] REPLICATION: starting
1025 07:52:42.226446 (+ 163us) log.cc:844] Serialized 433 byte log entry
1025 07:52:42.229548 (+ 3102us) op_driver.cc:464] REPLICATION: finished
1025 07:52:42.229669 (+ 121us) write_op.cc:301] APPLY: starting
1025 07:52:42.229687 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:42.229731 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:42.229765 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:42.229906 (+ 141us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:42.229942 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:42.229954 (+ 12us) write_op.cc:312] APPLY: finished
1025 07:52:42.230029 (+ 75us) log.cc:844] Serialized 29 byte log entry
1025 07:52:42.230090 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:42.230112 (+ 22us) write_op.cc:454] Released schema lock
1025 07:52:42.230118 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:42.230375 (+ 257us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:42.230380 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":68,"apply.run_cpu_time_us":531,"apply.run_wall_time_us":693,"num_ops":2,"prepare.queue_time_us":48,"prepare.run_cpu_time_us":709,"prepare.run_wall_time_us":709,"raft.queue_time_us":375,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":222,"replication_time_us":3235}]]}
W20251025 07:52:42.250264 2159 log.cc:927] Time spent T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Append to log took a long time: real 0.104s user 0.001s sys 0.003s
I20251025 07:52:42.252460 1659 raft_consensus.cc:493] T 36262ab8c6964dd684c01b4db6219a4f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:42.252609 1659 raft_consensus.cc:515] T 36262ab8c6964dd684c01b4db6219a4f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.253350 1659 leader_election.cc:290] T 36262ab8c6964dd684c01b4db6219a4f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:42.255249 447 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet aa9d9ab8073945d28d52fb02213f201a (table default.Table_4791672658f043f6b97b9320a531ebb5 [id=dfd20cf9c3e541c4abe23d59e0aa7f2a]). Expected version 1 got 0
I20251025 07:52:42.255661 447 catalog_manager.cc:5649] T 88bfc23acd274041a9457079c1709d82 P db42be7ad3c741a0aa427f27e245993b reported cstate change: term changed from 0 to 1, leader changed from <none> to db42be7ad3c741a0aa427f27e245993b (127.0.104.3). New cstate: current_term: 1 leader_uuid: "db42be7ad3c741a0aa427f27e245993b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: HEALTHY } } }
I20251025 07:52:42.256234 447 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 292fe04055954afd8c08ac2636b68191 (table default.Table_9fa69b10b5e842639fc0b0a58cd9b6a5 [id=a253ce0253bd4da4814741c482193d8f]). Expected version 1 got 0
I20251025 07:52:42.256520 447 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]). Expected version 1 got 0
I20251025 07:52:42.256708 447 catalog_manager.cc:5649] T ef9a2c9f2ccf432ab835396020872208 P db42be7ad3c741a0aa427f27e245993b reported cstate change: term changed from 0 to 1, leader changed from <none> to db42be7ad3c741a0aa427f27e245993b (127.0.104.3). New cstate: current_term: 1 leader_uuid: "db42be7ad3c741a0aa427f27e245993b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: HEALTHY } } }
I20251025 07:52:42.257056 447 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet ef9a2c9f2ccf432ab835396020872208 (table default.Table_2b9b772d4c234ac286bef3658482e490 [id=409fab60f9d744798e6f92b868f0f7c2]). Expected version 1 got 0
I20251025 07:52:42.257283 447 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]). Expected version 1 got 0
I20251025 07:52:42.257669 447 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e]). Expected version 1 got 0
I20251025 07:52:42.257834 447 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet bc2d16b33ca8426abbd7bd38339c9f1d (table default.Table_4ea985df4c5b415c9570c64506a98cc4 [id=98d2b7797e33485e8872054aa2eb8ec9]). Expected version 1 got 0
I20251025 07:52:42.261997 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 84) took 3662 ms. Trace:
I20251025 07:52:42.262269 2166 rpcz_store.cc:276] 1025 07:52:38.599116 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.599170 (+ 54us) service_pool.cc:225] Handling call
1025 07:52:42.150350 (+3551180us) ts_tablet_manager.cc:1151] Deleting tablet 4f25c9caa05b4da4bd56a8a21fde423c
1025 07:52:42.150365 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:42.220725 (+ 70360us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.236054 (+ 15329us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.261953 (+ 25899us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3545840}
I20251025 07:52:42.262492 1166 tablet_bootstrap.cc:654] T 36262ab8c6964dd684c01b4db6219a4f P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:42.262562 710 leader_election.cc:304] T 04d27d54fdb7483b869408af65ff85c9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:42.263026 1659 raft_consensus.cc:2804] T 04d27d54fdb7483b869408af65ff85c9 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:42.263274 1659 raft_consensus.cc:697] T 04d27d54fdb7483b869408af65ff85c9 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:42.263622 1659 consensus_queue.cc:237] T 04d27d54fdb7483b869408af65ff85c9 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:42.267923 2166 tablet_replica.cc:333] T 6dcab53a020847a1aa27930265ba0205 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:42.268196 2166 raft_consensus.cc:2243] T 6dcab53a020847a1aa27930265ba0205 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:42.268375 2166 raft_consensus.cc:2272] T 6dcab53a020847a1aa27930265ba0205 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:42.268975 2166 ts_tablet_manager.cc:1916] T 6dcab53a020847a1aa27930265ba0205 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:42.268987 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1912e59548874e0e8e989050ec0064c5" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:42.269683 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30418858bf95438686bdfe64ac733144" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:42.272979 1167 tablet_bootstrap.cc:492] T bdaeedbcf35a4aad97f9ef192991978a P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:42.273146 1167 ts_tablet_manager.cc:1403] T bdaeedbcf35a4aad97f9ef192991978a P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20251025 07:52:42.273895 1167 raft_consensus.cc:359] T bdaeedbcf35a4aad97f9ef192991978a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.274099 1167 raft_consensus.cc:385] T bdaeedbcf35a4aad97f9ef192991978a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.274223 1167 raft_consensus.cc:740] T bdaeedbcf35a4aad97f9ef192991978a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.274482 1167 consensus_queue.cc:260] T bdaeedbcf35a4aad97f9ef192991978a P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.274541 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36262ab8c6964dd684c01b4db6219a4f" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:42.275018 1167 ts_tablet_manager.cc:1434] T bdaeedbcf35a4aad97f9ef192991978a P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:42.275350 1167 tablet_bootstrap.cc:492] T 4678f7eab5e8497db308e8a38e387618 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:42.275660 768 raft_consensus.cc:3060] T 1912e59548874e0e8e989050ec0064c5 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:42.275957 710 leader_election.cc:343] T 30418858bf95438686bdfe64ac733144 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.277539 1167 tablet_bootstrap.cc:654] T 4678f7eab5e8497db308e8a38e387618 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:42.216740 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04d27d54fdb7483b869408af65ff85c9" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:42.219797 1046 raft_consensus.cc:3060] T 88eb51447e0f4080baa49dd902deb24f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.224525 1048 raft_consensus.cc:3060] T 0325d462441349ef81841f1b0cab2a20 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:42.278326 710 leader_election.cc:343] T 36262ab8c6964dd684c01b4db6219a4f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:42.280407 712 leader_election.cc:343] T 04d27d54fdb7483b869408af65ff85c9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.234902 2169 ts_tablet_manager.cc:1929] T 896bd7870d87408e8adfdb7a78998111 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:42.236186 982 leader_election.cc:304] T 88eb51447e0f4080baa49dd902deb24f P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
W20251025 07:52:42.281643 982 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:46231, user_credentials={real_user=slave}} blocked reactor thread for 45512.4us
I20251025 07:52:42.236747 2316 raft_consensus.cc:493] T 34315f64b89148efbf1f20a43d5c5efb P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:42.282243 2316 raft_consensus.cc:515] T 34315f64b89148efbf1f20a43d5c5efb P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:42.282846 2316 leader_election.cc:290] T 34315f64b89148efbf1f20a43d5c5efb P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:42.253907 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30418858bf95438686bdfe64ac733144" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:42.254175 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36262ab8c6964dd684c01b4db6219a4f" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:42.283145 2050 raft_consensus.cc:2749] T 88eb51447e0f4080baa49dd902deb24f P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:42.283347 712 leader_election.cc:343] T 30418858bf95438686bdfe64ac733144 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.283466 712 leader_election.cc:304] T 30418858bf95438686bdfe64ac733144 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.283574 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34315f64b89148efbf1f20a43d5c5efb" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:42.283893 712 leader_election.cc:343] T 36262ab8c6964dd684c01b4db6219a4f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.284019 712 leader_election.cc:304] T 36262ab8c6964dd684c01b4db6219a4f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.284327 2420 raft_consensus.cc:2749] T 30418858bf95438686bdfe64ac733144 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:42.284385 983 leader_election.cc:343] T 34315f64b89148efbf1f20a43d5c5efb P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.284489 2420 raft_consensus.cc:2749] T 36262ab8c6964dd684c01b4db6219a4f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:42.286172 982 leader_election.cc:343] T 34315f64b89148efbf1f20a43d5c5efb P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: 34315f64b89148efbf1f20a43d5c5efb
I20251025 07:52:42.286268 982 leader_election.cc:304] T 34315f64b89148efbf1f20a43d5c5efb P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:42.286480 2348 raft_consensus.cc:2749] T 34315f64b89148efbf1f20a43d5c5efb P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:42.287564 2314 consensus_queue.cc:1048] T 0325d462441349ef81841f1b0cab2a20 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:42.288816 2314 consensus_queue.cc:1048] T 88eb51447e0f4080baa49dd902deb24f P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:42.262975 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 4f25c9caa05b4da4bd56a8a21fde423c (table default.Table_60bfcc345b8649e9bc73fe3d077f46c7 [id=ed06dbbb96a24ab39619f4186ba93191]) successfully deleted
I20251025 07:52:42.290685 447 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 5dab439bc50d4e869548154e1d222ad6 (table default.Table_bd82d871f84943f4baa967f5c0ec74a3 [id=45325c188eac4d34bb016d01de091f87]). Expected version 2 got 0
I20251025 07:52:42.291018 447 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_87d38832f85e4965bfab1a077990114f [id=ad812862698d421184ebf501ce79b7be]). Expected version 2 got 0
I20251025 07:52:42.291332 447 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet f70091024c034c118fb86578db19a31c (table default.Table_332b308220f64c6dbb5f29711d8463b3 [id=e041946c093b4c789ce175c736cc230b]). Expected version 1 got 0
I20251025 07:52:42.292192 447 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]). Expected version 3 got 0
I20251025 07:52:42.287340 2169 log.cc:1199] T 896bd7870d87408e8adfdb7a78998111 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/896bd7870d87408e8adfdb7a78998111
I20251025 07:52:42.292472 447 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 22f7db660616457b83b24e947a0b0bae (table default.Table_4a86191163f2408f9bcc795cd8d1bbf6 [id=bb316a7e000748ca902a9bfb494802d7]). Expected version 1 got 0
I20251025 07:52:42.292665 447 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 755f0d6c87044e20a8020f8410940123 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 4 got 0
I20251025 07:52:42.292949 2169 ts_tablet_manager.cc:1950] T 896bd7870d87408e8adfdb7a78998111 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:42.292950 447 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236]). Expected version 1 got 0
I20251025 07:52:42.293454 447 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84]). Expected version 1 got 0
I20251025 07:52:42.294026 447 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 0325d462441349ef81841f1b0cab2a20 (table default.Table_750ecb905839434fa9a1aa72f87394be [id=9ce3ba77c56a4ef09cb3b40ea0e488f6]). Expected version 1 got 0
I20251025 07:52:42.294274 447 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]). Expected version 1 got 0
I20251025 07:52:42.294554 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 100) took 3539 ms. Trace:
I20251025 07:52:42.295403 2169 rpcz_store.cc:276] 1025 07:52:38.755424 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.755901 (+ 477us) service_pool.cc:225] Handling call
1025 07:52:41.722368 (+2966467us) ts_tablet_manager.cc:1151] Deleting tablet 896bd7870d87408e8adfdb7a78998111
1025 07:52:41.723974 (+ 1606us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:42.176536 (+452562us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.234787 (+ 58251us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.294533 (+ 59746us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2966227}
I20251025 07:52:42.296612 2169 tablet_replica.cc:333] T a13bf4d7eb1440f7a28b85cb1bf99667 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:42.296630 2105 consensus_queue.cc:1048] T 80fe3255ed614270bd8352a2853fbc43 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:42.296976 2169 raft_consensus.cc:2243] T a13bf4d7eb1440f7a28b85cb1bf99667 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:42.297313 2169 raft_consensus.cc:2272] T a13bf4d7eb1440f7a28b85cb1bf99667 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:42.297946 2169 ts_tablet_manager.cc:1916] T a13bf4d7eb1440f7a28b85cb1bf99667 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:42.301563 768 raft_consensus.cc:2468] T 1912e59548874e0e8e989050ec0064c5 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:42.302348 847 leader_election.cc:304] T 1912e59548874e0e8e989050ec0064c5 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.302785 2105 raft_consensus.cc:2804] T 1912e59548874e0e8e989050ec0064c5 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:42.302934 2105 raft_consensus.cc:697] T 1912e59548874e0e8e989050ec0064c5 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:42.303198 2105 consensus_queue.cc:237] T 1912e59548874e0e8e989050ec0064c5 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.308476 2469 raft_consensus.cc:493] T 80fe3255ed614270bd8352a2853fbc43 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:42.308718 2469 raft_consensus.cc:515] T 80fe3255ed614270bd8352a2853fbc43 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:42.310161 2469 leader_election.cc:290] T 80fe3255ed614270bd8352a2853fbc43 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:42.310638 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "80fe3255ed614270bd8352a2853fbc43" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:42.218369 2146 ts_tablet_manager.cc:1950] T a13bf4d7eb1440f7a28b85cb1bf99667 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:42.311786 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 113) took 3812 ms. Trace:
I20251025 07:52:42.311923 2146 rpcz_store.cc:276] 1025 07:52:38.499595 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.499911 (+ 316us) service_pool.cc:225] Handling call
1025 07:52:41.959026 (+3459115us) ts_tablet_manager.cc:1151] Deleting tablet a13bf4d7eb1440f7a28b85cb1bf99667
1025 07:52:41.959042 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:42.029781 (+ 70739us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.191659 (+161878us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.311767 (+120108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3436376}
I20251025 07:52:42.312372 2146 tablet_replica.cc:333] T f3001314b56345f5a70314777cae0125 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:42.312618 2146 raft_consensus.cc:2243] T f3001314b56345f5a70314777cae0125 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:42.312848 2146 raft_consensus.cc:2272] T f3001314b56345f5a70314777cae0125 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:42.313575 2146 ts_tablet_manager.cc:1916] T f3001314b56345f5a70314777cae0125 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:42.314605 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "80fe3255ed614270bd8352a2853fbc43" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:42.296262 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 896bd7870d87408e8adfdb7a78998111 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]) successfully deleted
I20251025 07:52:42.316278 982 leader_election.cc:304] T 80fe3255ed614270bd8352a2853fbc43 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:42.312557 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet a13bf4d7eb1440f7a28b85cb1bf99667 (table default.Table_434ef1ade0a14af987864563993afae8 [id=3a3af35e11944a6f97fc0144cb9d08d4]) successfully deleted
I20251025 07:52:42.316766 2469 raft_consensus.cc:2749] T 80fe3255ed614270bd8352a2853fbc43 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:42.318991 1167 tablet_bootstrap.cc:492] T 4678f7eab5e8497db308e8a38e387618 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:42.319180 1167 ts_tablet_manager.cc:1403] T 4678f7eab5e8497db308e8a38e387618 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.044s user 0.001s sys 0.002s
I20251025 07:52:42.320366 1169 tablet_bootstrap.cc:492] T 36262ab8c6964dd684c01b4db6219a4f P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:42.320479 1169 ts_tablet_manager.cc:1403] T 36262ab8c6964dd684c01b4db6219a4f P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.115s user 0.000s sys 0.004s
I20251025 07:52:42.317062 447 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525]). Expected version 1 got 0
I20251025 07:52:42.322412 447 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_e90d7f5ff13b40878e5b625c97792c7d [id=fc8180c09d7b4e74b2e161f7acb8fe0e]). Expected version 1 got 0
I20251025 07:52:42.322767 447 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet b56209e4934e4407a8b4518ca4af0d46 (table default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de]). Expected version 1 got 0
I20251025 07:52:42.322976 447 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 830fdbc6812e416690d0db42a73af503 (table default.Table_92824b3f2a774564b3aea04aaa25d204 [id=fb21a45b86ab446e8c3dae4512a9598f]). Expected version 1 got 0
I20251025 07:52:42.323652 447 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c]). Expected version 1 got 0
I20251025 07:52:42.324007 1169 raft_consensus.cc:359] T 36262ab8c6964dd684c01b4db6219a4f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.324548 447 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 36262ab8c6964dd684c01b4db6219a4f (table default.Table_76158f8cf82e4cdb9a90580e46aaea48 [id=f249f30497bc450c887d659721469de8]). Expected version 1 got 0
I20251025 07:52:42.325011 1169 raft_consensus.cc:385] T 36262ab8c6964dd684c01b4db6219a4f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.325094 1169 raft_consensus.cc:740] T 36262ab8c6964dd684c01b4db6219a4f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.325340 1169 consensus_queue.cc:260] T 36262ab8c6964dd684c01b4db6219a4f P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.325351 447 catalog_manager.cc:5649] T 22c97b3ee92140c68a7e1b93033cb299 P db42be7ad3c741a0aa427f27e245993b reported cstate change: term changed from 0 to 1, leader changed from <none> to db42be7ad3c741a0aa427f27e245993b (127.0.104.3). New cstate: current_term: 1 leader_uuid: "db42be7ad3c741a0aa427f27e245993b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: HEALTHY } } }
I20251025 07:52:42.325917 1169 ts_tablet_manager.cc:1434] T 36262ab8c6964dd684c01b4db6219a4f P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251025 07:52:42.326347 1169 tablet_bootstrap.cc:492] T f43ef51ee45d47a99ec732b49d50d3b4 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:42.327868 1169 tablet_bootstrap.cc:654] T f43ef51ee45d47a99ec732b49d50d3b4 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:42.328215 1167 raft_consensus.cc:359] T 4678f7eab5e8497db308e8a38e387618 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.328557 1167 raft_consensus.cc:385] T 4678f7eab5e8497db308e8a38e387618 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.328755 1167 raft_consensus.cc:740] T 4678f7eab5e8497db308e8a38e387618 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.329195 2420 raft_consensus.cc:493] T 4037dee385d04e85bb78c8e77cb00379 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3aea2869ff814cc7b142012faa274382)
I20251025 07:52:42.329353 2420 raft_consensus.cc:515] T 4037dee385d04e85bb78c8e77cb00379 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.329267 1167 consensus_queue.cc:260] T 4678f7eab5e8497db308e8a38e387618 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.329875 2420 leader_election.cc:290] T 4037dee385d04e85bb78c8e77cb00379 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:42.330528 2166 ts_tablet_manager.cc:1929] T 6dcab53a020847a1aa27930265ba0205 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:42.334128 1166 tablet_bootstrap.cc:492] T 36262ab8c6964dd684c01b4db6219a4f P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:42.334293 1166 ts_tablet_manager.cc:1403] T 36262ab8c6964dd684c01b4db6219a4f P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.116s user 0.006s sys 0.001s
I20251025 07:52:42.334985 1838 consensus_queue.cc:1048] T c2713942584e4734acbc0775ead772fa P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251025 07:52:42.331358 710 leader_election.cc:343] T 4037dee385d04e85bb78c8e77cb00379 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 4037dee385d04e85bb78c8e77cb00379
I20251025 07:52:42.338323 2166 log.cc:1199] T 6dcab53a020847a1aa27930265ba0205 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/6dcab53a020847a1aa27930265ba0205
I20251025 07:52:42.338937 2166 ts_tablet_manager.cc:1950] T 6dcab53a020847a1aa27930265ba0205 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:42.339884 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 85) took 3736 ms. Trace:
I20251025 07:52:42.331710 1167 ts_tablet_manager.cc:1434] T 4678f7eab5e8497db308e8a38e387618 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20251025 07:52:42.339963 2471 raft_consensus.cc:493] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:42.340106 2471 raft_consensus.cc:515] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.340412 1167 tablet_bootstrap.cc:492] T f656945454fe4921ab819c772a089eb3 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:42.340689 2471 leader_election.cc:290] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:42.341228 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f5f6ca6c3cd4d369690e425387a4344" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:42.341581 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f5f6ca6c3cd4d369690e425387a4344" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:42.341625 768 raft_consensus.cc:2468] T 0f5f6ca6c3cd4d369690e425387a4344 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
W20251025 07:52:42.342006 850 leader_election.cc:343] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.342382 1167 tablet_bootstrap.cc:654] T f656945454fe4921ab819c772a089eb3 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:42.342521 1166 raft_consensus.cc:359] T 36262ab8c6964dd684c01b4db6219a4f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.342718 1166 raft_consensus.cc:385] T 36262ab8c6964dd684c01b4db6219a4f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.342491 847 leader_election.cc:304] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.342809 1166 raft_consensus.cc:740] T 36262ab8c6964dd684c01b4db6219a4f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.343464 1045 raft_consensus.cc:3060] T 80fe3255ed614270bd8352a2853fbc43 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.343456 1166 consensus_queue.cc:260] T 36262ab8c6964dd684c01b4db6219a4f P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.344128 2471 raft_consensus.cc:2804] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:42.344204 2471 raft_consensus.cc:493] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:42.344262 2471 raft_consensus.cc:3060] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.345166 1166 ts_tablet_manager.cc:1434] T 36262ab8c6964dd684c01b4db6219a4f P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20251025 07:52:42.347441 2471 raft_consensus.cc:515] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.348320 2471 leader_election.cc:290] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:42.348567 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f5f6ca6c3cd4d369690e425387a4344" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:42.348985 850 leader_election.cc:343] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.349154 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f5f6ca6c3cd4d369690e425387a4344" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:42.349473 768 raft_consensus.cc:3060] T 0f5f6ca6c3cd4d369690e425387a4344 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.350201 2166 rpcz_store.cc:276] 1025 07:52:38.603550 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.603599 (+ 49us) service_pool.cc:225] Handling call
1025 07:52:42.262543 (+3658944us) ts_tablet_manager.cc:1151] Deleting tablet 6dcab53a020847a1aa27930265ba0205
1025 07:52:42.267841 (+ 5298us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:42.302876 (+ 35035us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.330402 (+ 27526us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.339780 (+ 9378us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3657991}
I20251025 07:52:42.350651 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 6dcab53a020847a1aa27930265ba0205 (table default.Table_9b7fc5a1f4944e749c90c1119d0b0054 [id=cb79dc5b6680450cadf2350095389c9e]) successfully deleted
I20251025 07:52:42.351018 2166 tablet_replica.cc:333] T 896bd7870d87408e8adfdb7a78998111 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:42.351249 2166 raft_consensus.cc:2243] T 896bd7870d87408e8adfdb7a78998111 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
W20251025 07:52:42.351310 712 leader_election.cc:343] T 4037dee385d04e85bb78c8e77cb00379 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 4037dee385d04e85bb78c8e77cb00379
I20251025 07:52:42.351486 2166 raft_consensus.cc:2272] T 896bd7870d87408e8adfdb7a78998111 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:42.351461 712 leader_election.cc:304] T 4037dee385d04e85bb78c8e77cb00379 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.351967 1659 raft_consensus.cc:2749] T 4037dee385d04e85bb78c8e77cb00379 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 07:52:42.352053 2166 ts_tablet_manager.cc:1916] T 896bd7870d87408e8adfdb7a78998111 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:42.353459 2169 ts_tablet_manager.cc:1929] T a13bf4d7eb1440f7a28b85cb1bf99667 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:42.354067 1166 tablet_bootstrap.cc:492] T f43ef51ee45d47a99ec732b49d50d3b4 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:42.358407 447 catalog_manager.cc:5649] T 6a58931e7cd7454fb583f1c70cd2ad15 P db42be7ad3c741a0aa427f27e245993b reported cstate change: term changed from 0 to 1, leader changed from <none> to db42be7ad3c741a0aa427f27e245993b (127.0.104.3). New cstate: current_term: 1 leader_uuid: "db42be7ad3c741a0aa427f27e245993b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: HEALTHY } } }
I20251025 07:52:42.359159 2169 log.cc:1199] T a13bf4d7eb1440f7a28b85cb1bf99667 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/a13bf4d7eb1440f7a28b85cb1bf99667
I20251025 07:52:42.359179 447 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3]). Expected version 2 got 0
I20251025 07:52:42.359177 2105 consensus_queue.cc:1048] T 80fe3255ed614270bd8352a2853fbc43 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:42.359592 2169 ts_tablet_manager.cc:1950] T a13bf4d7eb1440f7a28b85cb1bf99667 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:42.359911 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 103) took 3602 ms. Trace:
I20251025 07:52:42.359992 2169 rpcz_store.cc:276] 1025 07:52:38.757344 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.757396 (+ 52us) service_pool.cc:225] Handling call
1025 07:52:42.296564 (+3539168us) ts_tablet_manager.cc:1151] Deleting tablet a13bf4d7eb1440f7a28b85cb1bf99667
1025 07:52:42.296574 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:42.302861 (+ 6287us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.353347 (+ 50486us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.359898 (+ 6551us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3538951}
I20251025 07:52:42.360159 447 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet ab42909afc524f2fa3c1041495d13a59 (table default.Table_7eb8424d5192485da1a2d67bfdb2bbbf [id=40c29df99590408cbd5c583d4fc741eb]). Expected version 2 got 0
I20251025 07:52:42.360333 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 102) took 3603 ms. Trace:
I20251025 07:52:42.360400 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet a13bf4d7eb1440f7a28b85cb1bf99667 (table default.Table_434ef1ade0a14af987864563993afae8 [id=3a3af35e11944a6f97fc0144cb9d08d4]) successfully deleted
I20251025 07:52:42.360399 2169 rpcz_store.cc:276] 1025 07:52:38.757160 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.757684 (+ 524us) service_pool.cc:225] Handling call
1025 07:52:42.360232 (+3602548us) ts_tablet_manager.cc:1151] Deleting tablet 570201e7bf05406a97c81dde071d894b
1025 07:52:42.360241 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:42.360320 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3602327}
I20251025 07:52:42.361940 2169 tablet_replica.cc:333] T f3001314b56345f5a70314777cae0125 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
W20251025 07:52:42.361929 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 570201e7bf05406a97c81dde071d894b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 570201e7bf05406a97c81dde071d894b already in progress: creating tablet
I20251025 07:52:42.362285 2169 raft_consensus.cc:2243] T f3001314b56345f5a70314777cae0125 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:42.365965 1166 tablet_bootstrap.cc:654] T f43ef51ee45d47a99ec732b49d50d3b4 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:42.367401 768 raft_consensus.cc:2468] T 0f5f6ca6c3cd4d369690e425387a4344 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:42.370679 1167 tablet_bootstrap.cc:492] T f656945454fe4921ab819c772a089eb3 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:42.370908 1167 ts_tablet_manager.cc:1403] T f656945454fe4921ab819c772a089eb3 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20251025 07:52:42.370846 847 leader_election.cc:304] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.371228 2169 raft_consensus.cc:2272] T f3001314b56345f5a70314777cae0125 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:42.371438 2105 raft_consensus.cc:2804] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:42.371596 2105 raft_consensus.cc:697] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:42.371734 1167 raft_consensus.cc:359] T f656945454fe4921ab819c772a089eb3 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.371939 1167 raft_consensus.cc:385] T f656945454fe4921ab819c772a089eb3 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.372021 1167 raft_consensus.cc:740] T f656945454fe4921ab819c772a089eb3 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.371866 2105 consensus_queue.cc:237] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.372100 2169 ts_tablet_manager.cc:1916] T f3001314b56345f5a70314777cae0125 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:42.372242 1167 consensus_queue.cc:260] T f656945454fe4921ab819c772a089eb3 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.372814 1167 ts_tablet_manager.cc:1434] T f656945454fe4921ab819c772a089eb3 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:42.373180 1167 tablet_bootstrap.cc:492] T ce1ebb1ec14b4801930a72765bca1a7d P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:42.375195 1167 tablet_bootstrap.cc:654] T ce1ebb1ec14b4801930a72765bca1a7d P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:42.377403 1169 tablet_bootstrap.cc:492] T f43ef51ee45d47a99ec732b49d50d3b4 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:42.379407 1169 ts_tablet_manager.cc:1403] T f43ef51ee45d47a99ec732b49d50d3b4 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.000s
I20251025 07:52:42.380319 1169 raft_consensus.cc:359] T f43ef51ee45d47a99ec732b49d50d3b4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:42.380515 1169 raft_consensus.cc:385] T f43ef51ee45d47a99ec732b49d50d3b4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.380604 1169 raft_consensus.cc:740] T f43ef51ee45d47a99ec732b49d50d3b4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.380851 1169 consensus_queue.cc:260] T f43ef51ee45d47a99ec732b49d50d3b4 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:42.383199 2169 ts_tablet_manager.cc:1929] T f3001314b56345f5a70314777cae0125 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:42.383392 2169 log.cc:1199] T f3001314b56345f5a70314777cae0125 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/f3001314b56345f5a70314777cae0125
I20251025 07:52:42.383652 2146 ts_tablet_manager.cc:1929] T f3001314b56345f5a70314777cae0125 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:42.383299 1169 ts_tablet_manager.cc:1434] T f43ef51ee45d47a99ec732b49d50d3b4 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:42.383770 2146 log.cc:1199] T f3001314b56345f5a70314777cae0125 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/f3001314b56345f5a70314777cae0125
I20251025 07:52:42.383935 2169 ts_tablet_manager.cc:1950] T f3001314b56345f5a70314777cae0125 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:42.384173 2146 ts_tablet_manager.cc:1950] T f3001314b56345f5a70314777cae0125 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:42.384912 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 114) took 3884 ms. Trace:
I20251025 07:52:42.388043 1169 tablet_bootstrap.cc:492] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:42.389626 1169 tablet_bootstrap.cc:654] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:42.390486 712 consensus_peers.cc:597] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:42.391072 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 104) took 3632 ms. Trace:
I20251025 07:52:42.391193 2169 rpcz_store.cc:276] 1025 07:52:38.758770 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.759095 (+ 325us) service_pool.cc:225] Handling call
1025 07:52:42.361895 (+3602800us) ts_tablet_manager.cc:1151] Deleting tablet f3001314b56345f5a70314777cae0125
1025 07:52:42.361906 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:42.376487 (+ 14581us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.383113 (+ 6626us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.391055 (+ 7942us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3602569}
I20251025 07:52:42.391634 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet f3001314b56345f5a70314777cae0125 (table default.Table_d771d85f5f6a4016bfa426dd9dbf586a [id=0a62837e6dda43879cc075d289326c5c]) successfully deleted
I20251025 07:52:42.391703 2169 tablet_replica.cc:333] T f170424be2f643e7af499834955472b0 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:42.391969 2169 raft_consensus.cc:2243] T f170424be2f643e7af499834955472b0 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:42.392163 2169 raft_consensus.cc:2272] T f170424be2f643e7af499834955472b0 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:42.392963 2169 ts_tablet_manager.cc:1916] T f170424be2f643e7af499834955472b0 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:42.396113 1045 raft_consensus.cc:3060] T c2713942584e4734acbc0775ead772fa P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.399576 2443 consensus_queue.cc:1048] T c2713942584e4734acbc0775ead772fa P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:42.402197 2146 rpcz_store.cc:276] 1025 07:52:38.500165 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.500456 (+ 291us) service_pool.cc:225] Handling call
1025 07:52:42.312329 (+3811873us) ts_tablet_manager.cc:1151] Deleting tablet f3001314b56345f5a70314777cae0125
1025 07:52:42.312339 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:42.373844 (+ 61505us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.383541 (+ 9697us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.384891 (+ 1350us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3789645}
I20251025 07:52:42.402529 2146 tablet_replica.cc:333] T f170424be2f643e7af499834955472b0 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:42.402704 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet f3001314b56345f5a70314777cae0125 (table default.Table_d771d85f5f6a4016bfa426dd9dbf586a [id=0a62837e6dda43879cc075d289326c5c]) successfully deleted
I20251025 07:52:42.404326 2146 raft_consensus.cc:2243] T f170424be2f643e7af499834955472b0 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:42.405905 2146 raft_consensus.cc:2272] T f170424be2f643e7af499834955472b0 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:42.409518 2117 raft_consensus.cc:493] T f661d5909657455581dcb5e6eadd3c0b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:42.409687 2117 raft_consensus.cc:515] T f661d5909657455581dcb5e6eadd3c0b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.411468 2117 leader_election.cc:290] T f661d5909657455581dcb5e6eadd3c0b P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:42.411716 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f661d5909657455581dcb5e6eadd3c0b" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:42.411387 1166 tablet_bootstrap.cc:492] T f43ef51ee45d47a99ec732b49d50d3b4 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:42.413405 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f661d5909657455581dcb5e6eadd3c0b" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:42.414427 1166 ts_tablet_manager.cc:1403] T f43ef51ee45d47a99ec732b49d50d3b4 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.060s user 0.005s sys 0.000s
I20251025 07:52:42.415357 1166 raft_consensus.cc:359] T f43ef51ee45d47a99ec732b49d50d3b4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:42.415652 1166 raft_consensus.cc:385] T f43ef51ee45d47a99ec732b49d50d3b4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.415787 1166 raft_consensus.cc:740] T f43ef51ee45d47a99ec732b49d50d3b4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.416148 1166 consensus_queue.cc:260] T f43ef51ee45d47a99ec732b49d50d3b4 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:42.416932 1166 ts_tablet_manager.cc:1434] T f43ef51ee45d47a99ec732b49d50d3b4 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:42.415860 2146 ts_tablet_manager.cc:1916] T f170424be2f643e7af499834955472b0 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:42.418061 1166 tablet_bootstrap.cc:492] T 26c849d376d143c6bef0c8e0e1d7d03a P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
W20251025 07:52:42.425632 712 leader_election.cc:343] T f661d5909657455581dcb5e6eadd3c0b P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:42.428002 710 leader_election.cc:343] T f661d5909657455581dcb5e6eadd3c0b P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.437314 751 tablet_service.cc:1505] Processing CreateTablet for tablet 7c458f56612b4c6da110b10a5049b6bd (DEFAULT_TABLE table=default.Table_7613dd8cbc914d8d922e21f8b4183d5b [id=0cd0eeb4c23341c8b1dff0a614541193]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:42.437856 1659 raft_consensus.cc:493] T 09898735796541b0a8f3be22cdb6cc30 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:42.437110 2117 raft_consensus.cc:493] T 4678f7eab5e8497db308e8a38e387618 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:42.438124 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c458f56612b4c6da110b10a5049b6bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:42.438717 1018 tablet_service.cc:1505] Processing CreateTablet for tablet 7c458f56612b4c6da110b10a5049b6bd (DEFAULT_TABLE table=default.Table_7613dd8cbc914d8d922e21f8b4183d5b [id=0cd0eeb4c23341c8b1dff0a614541193]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:42.439085 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c458f56612b4c6da110b10a5049b6bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:42.438210 1659 raft_consensus.cc:515] T 09898735796541b0a8f3be22cdb6cc30 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.441447 2166 ts_tablet_manager.cc:1929] T 896bd7870d87408e8adfdb7a78998111 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:42.441355 710 leader_election.cc:304] T f661d5909657455581dcb5e6eadd3c0b P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.441707 2117 raft_consensus.cc:515] T 4678f7eab5e8497db308e8a38e387618 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.442584 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4678f7eab5e8497db308e8a38e387618" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:42.442795 894 tablet_service.cc:1505] Processing CreateTablet for tablet 7c458f56612b4c6da110b10a5049b6bd (DEFAULT_TABLE table=default.Table_7613dd8cbc914d8d922e21f8b4183d5b [id=0cd0eeb4c23341c8b1dff0a614541193]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:42.443068 2420 raft_consensus.cc:2749] T f661d5909657455581dcb5e6eadd3c0b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:42.443166 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c458f56612b4c6da110b10a5049b6bd. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:42.443209 712 leader_election.cc:343] T 4678f7eab5e8497db308e8a38e387618 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.443267 2420 raft_consensus.cc:493] T 36c8fcafd5464b4e999276a8062a812a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:42.444046 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "09898735796541b0a8f3be22cdb6cc30" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:42.444063 1166 tablet_bootstrap.cc:654] T 26c849d376d143c6bef0c8e0e1d7d03a P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:42.445883 2443 raft_consensus.cc:493] T bc2d16b33ca8426abbd7bd38339c9f1d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:42.445982 2443 raft_consensus.cc:515] T bc2d16b33ca8426abbd7bd38339c9f1d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.446558 2443 leader_election.cc:290] T bc2d16b33ca8426abbd7bd38339c9f1d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:42.447189 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc2d16b33ca8426abbd7bd38339c9f1d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:42.447646 850 leader_election.cc:343] T bc2d16b33ca8426abbd7bd38339c9f1d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.452173 2420 raft_consensus.cc:515] T 36c8fcafd5464b4e999276a8062a812a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.452792 2420 leader_election.cc:290] T 36c8fcafd5464b4e999276a8062a812a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:42.453533 712 leader_election.cc:343] T 09898735796541b0a8f3be22cdb6cc30 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.454268 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36c8fcafd5464b4e999276a8062a812a" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:42.454836 1659 leader_election.cc:290] T 09898735796541b0a8f3be22cdb6cc30 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:42.455482 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc2d16b33ca8426abbd7bd38339c9f1d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:42.455672 768 raft_consensus.cc:2468] T bc2d16b33ca8426abbd7bd38339c9f1d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:42.456054 847 leader_election.cc:304] T bc2d16b33ca8426abbd7bd38339c9f1d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.456319 2443 raft_consensus.cc:2804] T bc2d16b33ca8426abbd7bd38339c9f1d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:42.456396 2443 raft_consensus.cc:493] T bc2d16b33ca8426abbd7bd38339c9f1d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:42.456465 2443 raft_consensus.cc:3060] T bc2d16b33ca8426abbd7bd38339c9f1d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.456449 2117 leader_election.cc:290] T 4678f7eab5e8497db308e8a38e387618 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:42.456578 712 leader_election.cc:343] T 36c8fcafd5464b4e999276a8062a812a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.459062 2443 raft_consensus.cc:515] T bc2d16b33ca8426abbd7bd38339c9f1d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.459937 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc2d16b33ca8426abbd7bd38339c9f1d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:42.460379 850 leader_election.cc:343] T bc2d16b33ca8426abbd7bd38339c9f1d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.460659 2443 leader_election.cc:290] T bc2d16b33ca8426abbd7bd38339c9f1d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:42.463996 2443 raft_consensus.cc:493] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:42.464056 2166 log.cc:1199] T 896bd7870d87408e8adfdb7a78998111 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/896bd7870d87408e8adfdb7a78998111
I20251025 07:52:42.464136 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc2d16b33ca8426abbd7bd38339c9f1d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:42.464148 2443 raft_consensus.cc:515] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.464601 775 raft_consensus.cc:3060] T bc2d16b33ca8426abbd7bd38339c9f1d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.464984 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb829649f73d4d3f8862d268c953bd86" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:42.465528 850 leader_election.cc:343] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.465768 2166 ts_tablet_manager.cc:1950] T 896bd7870d87408e8adfdb7a78998111 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:42.465860 2443 leader_election.cc:290] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:42.466737 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb829649f73d4d3f8862d268c953bd86" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:42.466933 768 raft_consensus.cc:2468] T cb829649f73d4d3f8862d268c953bd86 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:42.466986 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 83) took 3868 ms. Trace:
I20251025 07:52:42.467306 847 leader_election.cc:304] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.470333 2105 raft_consensus.cc:2804] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:42.470417 2105 raft_consensus.cc:493] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:42.470472 2105 raft_consensus.cc:3060] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.470667 2314 raft_consensus.cc:493] T 36262ab8c6964dd684c01b4db6219a4f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:42.470781 2314 raft_consensus.cc:515] T 36262ab8c6964dd684c01b4db6219a4f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.471326 2314 leader_election.cc:290] T 36262ab8c6964dd684c01b4db6219a4f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:42.471899 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36262ab8c6964dd684c01b4db6219a4f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:42.472056 1046 raft_consensus.cc:2468] T 36262ab8c6964dd684c01b4db6219a4f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:42.472460 850 leader_election.cc:304] T 36262ab8c6964dd684c01b4db6219a4f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:42.472591 2314 raft_consensus.cc:493] T 4b28215f61574cf48c447dd01955ac77 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:42.472684 2314 raft_consensus.cc:515] T 4b28215f61574cf48c447dd01955ac77 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:42.473167 2314 leader_election.cc:290] T 4b28215f61574cf48c447dd01955ac77 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:42.473119 2105 raft_consensus.cc:515] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.472671 2471 raft_consensus.cc:2804] T 36262ab8c6964dd684c01b4db6219a4f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:42.473604 2471 raft_consensus.cc:493] T 36262ab8c6964dd684c01b4db6219a4f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:42.473611 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b28215f61574cf48c447dd01955ac77" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:42.473733 2471 raft_consensus.cc:3060] T 36262ab8c6964dd684c01b4db6219a4f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.473877 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb829649f73d4d3f8862d268c953bd86" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:42.473978 2105 leader_election.cc:290] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
W20251025 07:52:42.474290 850 leader_election.cc:343] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:42.474686 850 leader_election.cc:343] T 4b28215f61574cf48c447dd01955ac77 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.476315 1166 tablet_bootstrap.cc:492] T 26c849d376d143c6bef0c8e0e1d7d03a P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:42.476539 1166 ts_tablet_manager.cc:1403] T 26c849d376d143c6bef0c8e0e1d7d03a P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.059s user 0.001s sys 0.005s
I20251025 07:52:42.477244 2471 raft_consensus.cc:515] T 36262ab8c6964dd684c01b4db6219a4f P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.478295 1169 tablet_bootstrap.cc:492] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:42.479131 1169 ts_tablet_manager.cc:1403] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.091s user 0.004s sys 0.003s
I20251025 07:52:42.477355 1166 raft_consensus.cc:359] T 26c849d376d143c6bef0c8e0e1d7d03a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.481534 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "09898735796541b0a8f3be22cdb6cc30" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:42.484078 1166 raft_consensus.cc:385] T 26c849d376d143c6bef0c8e0e1d7d03a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.481839 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4678f7eab5e8497db308e8a38e387618" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:42.483430 847 leader_election.cc:343] T 4b28215f61574cf48c447dd01955ac77 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: 4b28215f61574cf48c447dd01955ac77
I20251025 07:52:42.484308 1166 raft_consensus.cc:740] T 26c849d376d143c6bef0c8e0e1d7d03a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.484397 847 leader_election.cc:304] T 4b28215f61574cf48c447dd01955ac77 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.484629 2105 raft_consensus.cc:2749] T 4b28215f61574cf48c447dd01955ac77 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:42.481691 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36c8fcafd5464b4e999276a8062a812a" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:42.484907 1166 consensus_queue.cc:260] T 26c849d376d143c6bef0c8e0e1d7d03a P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.485949 1166 ts_tablet_manager.cc:1434] T 26c849d376d143c6bef0c8e0e1d7d03a P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251025 07:52:42.486601 2471 leader_election.cc:290] T 36262ab8c6964dd684c01b4db6219a4f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:42.486702 2169 ts_tablet_manager.cc:1929] T f170424be2f643e7af499834955472b0 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:42.486799 2169 log.cc:1199] T f170424be2f643e7af499834955472b0 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/f170424be2f643e7af499834955472b0
I20251025 07:52:42.486891 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36262ab8c6964dd684c01b4db6219a4f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:42.487231 2169 ts_tablet_manager.cc:1950] T f170424be2f643e7af499834955472b0 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:42.487305 1045 raft_consensus.cc:3060] T 36262ab8c6964dd684c01b4db6219a4f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.490072 1045 raft_consensus.cc:2468] T 36262ab8c6964dd684c01b4db6219a4f P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:42.490527 850 leader_election.cc:304] T 36262ab8c6964dd684c01b4db6219a4f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:42.490880 2471 raft_consensus.cc:2804] T 36262ab8c6964dd684c01b4db6219a4f P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:42.491062 2471 raft_consensus.cc:697] T 36262ab8c6964dd684c01b4db6219a4f P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:42.491343 2471 consensus_queue.cc:237] T 36262ab8c6964dd684c01b4db6219a4f P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.493425 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 105) took 3733 ms. Trace:
I20251025 07:52:42.493563 2169 rpcz_store.cc:276] 1025 07:52:38.760421 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.760747 (+ 326us) service_pool.cc:225] Handling call
1025 07:52:42.391656 (+3630909us) ts_tablet_manager.cc:1151] Deleting tablet f170424be2f643e7af499834955472b0
1025 07:52:42.391666 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:42.420742 (+ 29076us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.486592 (+ 65850us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.493406 (+ 6814us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3630685}
I20251025 07:52:42.494081 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet f170424be2f643e7af499834955472b0 (table default.Table_f3cbba9b75614002b5464930ccab8c7f [id=4f5379fa0372460ea325339f39289a10]) successfully deleted
I20251025 07:52:42.494179 2169 tablet_replica.cc:333] T 943ec703e65f4fd3b0c690e08c0bd226 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:42.494468 2169 raft_consensus.cc:2243] T 943ec703e65f4fd3b0c690e08c0bd226 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:42.494685 2169 raft_consensus.cc:2272] T 943ec703e65f4fd3b0c690e08c0bd226 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:42.495318 2169 ts_tablet_manager.cc:1916] T 943ec703e65f4fd3b0c690e08c0bd226 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:42.496191 1166 tablet_bootstrap.cc:492] T 0551e5d694894e8fbeb66b5e0c1cd75b P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:42.497766 1166 tablet_bootstrap.cc:654] T 0551e5d694894e8fbeb66b5e0c1cd75b P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:42.502450 2117 raft_consensus.cc:493] T 887e7c7a5d0e4aaf88f26246306df19c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3aea2869ff814cc7b142012faa274382)
I20251025 07:52:42.502621 2117 raft_consensus.cc:515] T 887e7c7a5d0e4aaf88f26246306df19c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.502786 775 raft_consensus.cc:2468] T bc2d16b33ca8426abbd7bd38339c9f1d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:42.501288 1169 raft_consensus.cc:359] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.503345 2117 leader_election.cc:290] T 887e7c7a5d0e4aaf88f26246306df19c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:42.503679 1167 tablet_bootstrap.cc:492] T ce1ebb1ec14b4801930a72765bca1a7d P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:42.503715 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb829649f73d4d3f8862d268c953bd86" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:42.503830 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36262ab8c6964dd684c01b4db6219a4f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:42.503633 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36262ab8c6964dd684c01b4db6219a4f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:42.503937 770 raft_consensus.cc:3060] T cb829649f73d4d3f8862d268c953bd86 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.503979 774 raft_consensus.cc:3060] T 36262ab8c6964dd684c01b4db6219a4f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.503909 1167 ts_tablet_manager.cc:1403] T ce1ebb1ec14b4801930a72765bca1a7d P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.131s user 0.002s sys 0.004s
I20251025 07:52:42.504997 1167 raft_consensus.cc:359] T ce1ebb1ec14b4801930a72765bca1a7d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.505163 1167 raft_consensus.cc:385] T ce1ebb1ec14b4801930a72765bca1a7d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.505249 1167 raft_consensus.cc:740] T ce1ebb1ec14b4801930a72765bca1a7d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.505470 1167 consensus_queue.cc:260] T ce1ebb1ec14b4801930a72765bca1a7d P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.505996 1167 ts_tablet_manager.cc:1434] T ce1ebb1ec14b4801930a72765bca1a7d P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:42.506301 1167 tablet_bootstrap.cc:492] T a77eb82f26ae43b48de1e85693372d34 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:42.506989 770 raft_consensus.cc:2468] T cb829649f73d4d3f8862d268c953bd86 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
W20251025 07:52:42.507896 712 leader_election.cc:343] T 887e7c7a5d0e4aaf88f26246306df19c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 887e7c7a5d0e4aaf88f26246306df19c
I20251025 07:52:42.508668 1167 tablet_bootstrap.cc:654] T a77eb82f26ae43b48de1e85693372d34 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:42.510362 847 leader_election.cc:304] T bc2d16b33ca8426abbd7bd38339c9f1d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.510939 847 leader_election.cc:304] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.511204 2443 raft_consensus.cc:2804] T bc2d16b33ca8426abbd7bd38339c9f1d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:42.511248 2471 raft_consensus.cc:2804] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:42.511353 2443 raft_consensus.cc:697] T bc2d16b33ca8426abbd7bd38339c9f1d P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:42.511396 2471 raft_consensus.cc:697] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:42.511653 2443 consensus_queue.cc:237] T bc2d16b33ca8426abbd7bd38339c9f1d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.512965 774 raft_consensus.cc:2468] T 36262ab8c6964dd684c01b4db6219a4f P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
W20251025 07:52:42.513871 710 leader_election.cc:343] T 09898735796541b0a8f3be22cdb6cc30 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.514025 710 leader_election.cc:304] T 09898735796541b0a8f3be22cdb6cc30 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:42.514533 710 leader_election.cc:343] T 4678f7eab5e8497db308e8a38e387618 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.514694 710 leader_election.cc:304] T 4678f7eab5e8497db308e8a38e387618 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.515075 2508 raft_consensus.cc:2749] T 09898735796541b0a8f3be22cdb6cc30 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:42.515193 710 leader_election.cc:343] T 36c8fcafd5464b4e999276a8062a812a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.515314 2508 raft_consensus.cc:2749] T 4678f7eab5e8497db308e8a38e387618 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:42.515700 442 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_a6c869949b744bc98efb735b0e73e7aa [id=bf77b5ff6f1048dd996d948437f9ff5e], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet d46673a283fe42b78ddeb99fbd2678eb consensus state has no leader
I20251025 07:52:42.517133 2471 consensus_queue.cc:237] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.518891 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_a6c869949b744bc98efb735b0e73e7aa [id=bf77b5ff6f1048dd996d948437f9ff5e]). Expected version 2 got 0
I20251025 07:52:42.503253 1169 raft_consensus.cc:385] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.519436 2146 ts_tablet_manager.cc:1929] T f170424be2f643e7af499834955472b0 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:42.519312 1169 raft_consensus.cc:740] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.520285 2146 log.cc:1199] T f170424be2f643e7af499834955472b0 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/f170424be2f643e7af499834955472b0
I20251025 07:52:42.520697 2146 ts_tablet_manager.cc:1950] T f170424be2f643e7af499834955472b0 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:42.516131 2169 ts_tablet_manager.cc:1929] T 943ec703e65f4fd3b0c690e08c0bd226 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:42.521450 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3]). Expected version 1 got 0
I20251025 07:52:42.525677 1166 tablet_bootstrap.cc:492] T 0551e5d694894e8fbeb66b5e0c1cd75b P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:42.526105 1166 ts_tablet_manager.cc:1403] T 0551e5d694894e8fbeb66b5e0c1cd75b P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20251025 07:52:42.526999 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 115) took 4026 ms. Trace:
I20251025 07:52:42.527899 2169 log.cc:1199] T 943ec703e65f4fd3b0c690e08c0bd226 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/943ec703e65f4fd3b0c690e08c0bd226
I20251025 07:52:42.527417 2146 rpcz_store.cc:276] 1025 07:52:38.500700 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.501549 (+ 849us) service_pool.cc:225] Handling call
1025 07:52:42.402476 (+3900927us) ts_tablet_manager.cc:1151] Deleting tablet f170424be2f643e7af499834955472b0
1025 07:52:42.402488 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:42.457845 (+ 55357us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.518654 (+ 60809us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.526828 (+ 8174us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3878546}
I20251025 07:52:42.529886 2169 ts_tablet_manager.cc:1950] T 943ec703e65f4fd3b0c690e08c0bd226 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:42.529839 2146 tablet_replica.cc:333] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:42.530856 1167 tablet_bootstrap.cc:492] T a77eb82f26ae43b48de1e85693372d34 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:42.531045 1167 ts_tablet_manager.cc:1403] T a77eb82f26ae43b48de1e85693372d34 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.001s
I20251025 07:52:42.530441 1166 raft_consensus.cc:359] T 0551e5d694894e8fbeb66b5e0c1cd75b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.531045 2146 raft_consensus.cc:2243] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:42.531473 1166 raft_consensus.cc:385] T 0551e5d694894e8fbeb66b5e0c1cd75b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.531898 1167 raft_consensus.cc:359] T a77eb82f26ae43b48de1e85693372d34 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.532115 1167 raft_consensus.cc:385] T a77eb82f26ae43b48de1e85693372d34 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.531840 1166 raft_consensus.cc:740] T 0551e5d694894e8fbeb66b5e0c1cd75b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.532203 1167 raft_consensus.cc:740] T a77eb82f26ae43b48de1e85693372d34 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.532141 2146 raft_consensus.cc:2272] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:42.532397 1167 consensus_queue.cc:260] T a77eb82f26ae43b48de1e85693372d34 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.533084 1167 ts_tablet_manager.cc:1434] T a77eb82f26ae43b48de1e85693372d34 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:42.533414 1167 tablet_bootstrap.cc:492] T 91728987303b41fd840eb56f3a311c5f P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:42.533250 1166 consensus_queue.cc:260] T 0551e5d694894e8fbeb66b5e0c1cd75b P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.535553 1167 tablet_bootstrap.cc:654] T 91728987303b41fd840eb56f3a311c5f P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:42.536458 2146 ts_tablet_manager.cc:1916] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:42.537451 1166 ts_tablet_manager.cc:1434] T 0551e5d694894e8fbeb66b5e0c1cd75b P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.011s user 0.002s sys 0.001s
I20251025 07:52:42.528621 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet f170424be2f643e7af499834955472b0 (table default.Table_f3cbba9b75614002b5464930ccab8c7f [id=4f5379fa0372460ea325339f39289a10]) successfully deleted
I20251025 07:52:42.538543 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5]). Expected version 1 got 0
I20251025 07:52:42.537868 2166 rpcz_store.cc:276] 1025 07:52:38.598557 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.598620 (+ 63us) service_pool.cc:225] Handling call
1025 07:52:42.350935 (+3752315us) ts_tablet_manager.cc:1151] Deleting tablet 896bd7870d87408e8adfdb7a78998111
1025 07:52:42.350945 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:42.355165 (+ 4220us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.428070 (+ 72905us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.466945 (+ 38875us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":90,"tablet-delete.queue_time_us":3746460}
I20251025 07:52:42.539409 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 896bd7870d87408e8adfdb7a78998111 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]) successfully deleted
I20251025 07:52:42.539803 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 86) took 3934 ms. Trace:
I20251025 07:52:42.539885 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f]). Expected version 1 got 0
I20251025 07:52:42.539934 2166 rpcz_store.cc:276] 1025 07:52:38.605566 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.605616 (+ 50us) service_pool.cc:225] Handling call
1025 07:52:42.539246 (+3933630us) ts_tablet_manager.cc:1151] Deleting tablet 1b5ac7e845a24b2eb5b2429d47361232
1025 07:52:42.539635 (+ 389us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:42.539769 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3933442}
W20251025 07:52:42.540449 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 1b5ac7e845a24b2eb5b2429d47361232 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b5ac7e845a24b2eb5b2429d47361232 already in progress: creating tablet
I20251025 07:52:42.540555 1166 tablet_bootstrap.cc:492] T c9f7ba3709c745d4a625449d6e1c8ba1 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:42.540624 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 106) took 3779 ms. Trace:
I20251025 07:52:42.540700 2166 tablet_replica.cc:333] T 21e52928580c4762b0ae7152490ef955 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:42.540717 2169 rpcz_store.cc:276] 1025 07:52:38.762944 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.763104 (+ 160us) service_pool.cc:225] Handling call
1025 07:52:42.494095 (+3730991us) ts_tablet_manager.cc:1151] Deleting tablet 943ec703e65f4fd3b0c690e08c0bd226
1025 07:52:42.494109 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:42.512918 (+ 18809us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.516034 (+ 3116us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.540603 (+ 24569us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3730779}
I20251025 07:52:42.540899 2166 raft_consensus.cc:2243] T 21e52928580c4762b0ae7152490ef955 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:42.540967 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7]). Expected version 1 got 0
I20251025 07:52:42.541049 2166 raft_consensus.cc:2272] T 21e52928580c4762b0ae7152490ef955 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:42.541350 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 872cb7512d104827b1f1f186d33a4a44 (table default.Table_d5c65dce8b1b4214ba519d964012bfc0 [id=d4be274b36a845a99a4b96b8846248fd]). Expected version 1 got 0
I20251025 07:52:42.521847 1169 consensus_queue.cc:260] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.545120 2166 ts_tablet_manager.cc:1916] T 21e52928580c4762b0ae7152490ef955 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:42.545513 712 consensus_peers.cc:597] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 27651312cc2e42a8bfcd245f5d301679. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:42.546883 2169 tablet_replica.cc:333] T 3775a2e08d4e40bd9f57e4cda759d535 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:42.547466 2169 raft_consensus.cc:2243] T 3775a2e08d4e40bd9f57e4cda759d535 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:42.541704 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 943ec703e65f4fd3b0c690e08c0bd226 (table default.Table_db98182d07a54725823623925f51a0cb [id=62cf364c073b47a1939f41dd69222582]) successfully deleted
I20251025 07:52:42.549401 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 04d27d54fdb7483b869408af65ff85c9 (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]). Expected version 2 got 0
I20251025 07:52:42.549868 2169 raft_consensus.cc:2272] T 3775a2e08d4e40bd9f57e4cda759d535 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:42.550805 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 4c155aa7fa5b4d68bbab1918b8432372 (table default.Table_6c5a24747ba14352b0285a41ec2985a4 [id=28bbaad474164a8c8c97a76bb8081791]). Expected version 2 got 0
I20251025 07:52:42.551419 2169 ts_tablet_manager.cc:1916] T 3775a2e08d4e40bd9f57e4cda759d535 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:42.547264 1169 ts_tablet_manager.cc:1434] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.068s user 0.001s sys 0.001s
I20251025 07:52:42.555173 1167 tablet_bootstrap.cc:492] T 91728987303b41fd840eb56f3a311c5f P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:42.555495 1167 ts_tablet_manager.cc:1403] T 91728987303b41fd840eb56f3a311c5f P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20251025 07:52:42.556216 1166 tablet_bootstrap.cc:654] T c9f7ba3709c745d4a625449d6e1c8ba1 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:42.556342 1167 raft_consensus.cc:359] T 91728987303b41fd840eb56f3a311c5f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.556515 1167 raft_consensus.cc:385] T 91728987303b41fd840eb56f3a311c5f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.556577 1167 raft_consensus.cc:740] T 91728987303b41fd840eb56f3a311c5f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.556782 1167 consensus_queue.cc:260] T 91728987303b41fd840eb56f3a311c5f P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.557406 1167 ts_tablet_manager.cc:1434] T 91728987303b41fd840eb56f3a311c5f P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:42.557709 1167 tablet_bootstrap.cc:492] T 11bfe0c87ca941a3846462c8a4512d83 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:42.559049 2471 consensus_queue.cc:1048] T 4c155aa7fa5b4d68bbab1918b8432372 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:42.559401 1167 tablet_bootstrap.cc:654] T 11bfe0c87ca941a3846462c8a4512d83 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:42.560112 1169 tablet_bootstrap.cc:492] T 292fe04055954afd8c08ac2636b68191 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:42.564203 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet cb829649f73d4d3f8862d268c953bd86 (table default.Table_ba4b8cc7b930482690b7e192e0edff12 [id=7944225d3ce7439b881bfb9289d9f040]). Expected version 1 got 0
I20251025 07:52:42.565339 446 catalog_manager.cc:5649] T ebf3a2a926754305b5bd95fcebb25d29 P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } }
I20251025 07:52:42.566372 1169 tablet_bootstrap.cc:654] T 292fe04055954afd8c08ac2636b68191 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:42.567528 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 292fe04055954afd8c08ac2636b68191 (table default.Table_9fa69b10b5e842639fc0b0a58cd9b6a5 [id=a253ce0253bd4da4814741c482193d8f]). Expected version 1 got 0
I20251025 07:52:42.575217 2166 ts_tablet_manager.cc:1929] T 21e52928580c4762b0ae7152490ef955 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:42.575683 2469 raft_consensus.cc:493] T f43ef51ee45d47a99ec732b49d50d3b4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:42.575812 2469 raft_consensus.cc:515] T f43ef51ee45d47a99ec732b49d50d3b4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:42.575910 2166 log.cc:1199] T 21e52928580c4762b0ae7152490ef955 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/21e52928580c4762b0ae7152490ef955
I20251025 07:52:42.576300 2166 ts_tablet_manager.cc:1950] T 21e52928580c4762b0ae7152490ef955 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:42.576377 2469 leader_election.cc:290] T f43ef51ee45d47a99ec732b49d50d3b4 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:42.576598 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 87) took 3967 ms. Trace:
I20251025 07:52:42.576707 2146 ts_tablet_manager.cc:1929] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:42.577476 2146 log.cc:1199] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/02e4c4d6ffb34cecaf17beb5ba197e81
I20251025 07:52:42.577740 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f43ef51ee45d47a99ec732b49d50d3b4" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:42.577948 909 raft_consensus.cc:2468] T f43ef51ee45d47a99ec732b49d50d3b4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:42.578634 983 leader_election.cc:304] T f43ef51ee45d47a99ec732b49d50d3b4 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:42.578941 2469 raft_consensus.cc:2804] T f43ef51ee45d47a99ec732b49d50d3b4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:42.579038 2469 raft_consensus.cc:493] T f43ef51ee45d47a99ec732b49d50d3b4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:42.579097 2469 raft_consensus.cc:3060] T f43ef51ee45d47a99ec732b49d50d3b4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.579272 2146 ts_tablet_manager.cc:1950] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:42.580689 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 116) took 4078 ms. Trace:
I20251025 07:52:42.581624 2469 raft_consensus.cc:515] T f43ef51ee45d47a99ec732b49d50d3b4 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:42.577160 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f43ef51ee45d47a99ec732b49d50d3b4" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:42.582404 774 raft_consensus.cc:2468] T f43ef51ee45d47a99ec732b49d50d3b4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:42.581288 2146 rpcz_store.cc:276] 1025 07:52:38.501801 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.514229 (+ 12428us) service_pool.cc:225] Handling call
1025 07:52:42.529152 (+4014923us) ts_tablet_manager.cc:1151] Deleting tablet 02e4c4d6ffb34cecaf17beb5ba197e81
1025 07:52:42.529275 (+ 123us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:42.560289 (+ 31014us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.576207 (+ 15918us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.580560 (+ 4353us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":434,"tablet-delete.queue_time_us":4004829}
I20251025 07:52:42.582451 2471 raft_consensus.cc:493] T 26c849d376d143c6bef0c8e0e1d7d03a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:42.582594 2471 raft_consensus.cc:515] T 26c849d376d143c6bef0c8e0e1d7d03a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.582643 2166 rpcz_store.cc:276] 1025 07:52:38.609565 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.609619 (+ 54us) service_pool.cc:225] Handling call
1025 07:52:42.540263 (+3930644us) ts_tablet_manager.cc:1151] Deleting tablet 21e52928580c4762b0ae7152490ef955
1025 07:52:42.540654 (+ 391us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:42.548813 (+ 8159us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.575099 (+ 26286us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.576581 (+ 1482us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3930458}
I20251025 07:52:42.583315 2471 leader_election.cc:290] T 26c849d376d143c6bef0c8e0e1d7d03a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:42.583876 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "26c849d376d143c6bef0c8e0e1d7d03a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:42.583889 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f43ef51ee45d47a99ec732b49d50d3b4" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:42.583988 1166 tablet_bootstrap.cc:492] T c9f7ba3709c745d4a625449d6e1c8ba1 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:42.584044 774 raft_consensus.cc:3060] T f43ef51ee45d47a99ec732b49d50d3b4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.584239 1166 ts_tablet_manager.cc:1403] T c9f7ba3709c745d4a625449d6e1c8ba1 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.046s user 0.006s sys 0.002s
W20251025 07:52:42.584399 850 leader_election.cc:343] T 26c849d376d143c6bef0c8e0e1d7d03a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.585132 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "26c849d376d143c6bef0c8e0e1d7d03a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:42.585282 770 raft_consensus.cc:2468] T 26c849d376d143c6bef0c8e0e1d7d03a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:42.585649 847 leader_election.cc:304] T 26c849d376d143c6bef0c8e0e1d7d03a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.585923 2471 raft_consensus.cc:2804] T 26c849d376d143c6bef0c8e0e1d7d03a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:42.586164 2471 raft_consensus.cc:493] T 26c849d376d143c6bef0c8e0e1d7d03a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:42.586050 1166 raft_consensus.cc:359] T c9f7ba3709c745d4a625449d6e1c8ba1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.586283 2471 raft_consensus.cc:3060] T 26c849d376d143c6bef0c8e0e1d7d03a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.586347 1166 raft_consensus.cc:385] T c9f7ba3709c745d4a625449d6e1c8ba1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.586485 1166 raft_consensus.cc:740] T c9f7ba3709c745d4a625449d6e1c8ba1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.586709 774 raft_consensus.cc:2468] T f43ef51ee45d47a99ec732b49d50d3b4 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:42.587039 1167 tablet_bootstrap.cc:492] T 11bfe0c87ca941a3846462c8a4512d83 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:42.586948 1166 consensus_queue.cc:260] T c9f7ba3709c745d4a625449d6e1c8ba1 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.587181 1167 ts_tablet_manager.cc:1403] T 11bfe0c87ca941a3846462c8a4512d83 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20251025 07:52:42.587180 982 leader_election.cc:304] T f43ef51ee45d47a99ec732b49d50d3b4 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:42.587447 2050 raft_consensus.cc:2804] T f43ef51ee45d47a99ec732b49d50d3b4 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:42.587539 2050 raft_consensus.cc:697] T f43ef51ee45d47a99ec732b49d50d3b4 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:42.587744 2050 consensus_queue.cc:237] T f43ef51ee45d47a99ec732b49d50d3b4 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:42.587879 1167 raft_consensus.cc:359] T 11bfe0c87ca941a3846462c8a4512d83 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.588016 1167 raft_consensus.cc:385] T 11bfe0c87ca941a3846462c8a4512d83 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.588063 1167 raft_consensus.cc:740] T 11bfe0c87ca941a3846462c8a4512d83 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.588227 1167 consensus_queue.cc:260] T 11bfe0c87ca941a3846462c8a4512d83 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.588762 1167 ts_tablet_manager.cc:1434] T 11bfe0c87ca941a3846462c8a4512d83 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 07:52:42.589017 1167 tablet_bootstrap.cc:492] T ebc824ec4c60411dab1a2ea4a385148e P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:42.589058 2471 raft_consensus.cc:515] T 26c849d376d143c6bef0c8e0e1d7d03a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.589728 2471 leader_election.cc:290] T 26c849d376d143c6bef0c8e0e1d7d03a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:42.590024 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "26c849d376d143c6bef0c8e0e1d7d03a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:42.590548 1167 tablet_bootstrap.cc:654] T ebc824ec4c60411dab1a2ea4a385148e P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:42.590545 850 leader_election.cc:343] T 26c849d376d143c6bef0c8e0e1d7d03a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.592375 2469 leader_election.cc:290] T f43ef51ee45d47a99ec732b49d50d3b4 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:42.590862 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 88) took 3979 ms. Trace:
I20251025 07:52:42.593119 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f43ef51ee45d47a99ec732b49d50d3b4" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:42.591111 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "26c849d376d143c6bef0c8e0e1d7d03a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:42.593325 774 raft_consensus.cc:3060] T 26c849d376d143c6bef0c8e0e1d7d03a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.593340 909 raft_consensus.cc:3060] T f43ef51ee45d47a99ec732b49d50d3b4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.595932 1166 ts_tablet_manager.cc:1434] T c9f7ba3709c745d4a625449d6e1c8ba1 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20251025 07:52:42.595861 2146 tablet_replica.cc:333] T 7c7a9d7a4fb3449bb57eafa83230a839 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:42.596019 774 raft_consensus.cc:2468] T 26c849d376d143c6bef0c8e0e1d7d03a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:42.596451 847 leader_election.cc:304] T 26c849d376d143c6bef0c8e0e1d7d03a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.596868 2166 rpcz_store.cc:276] 1025 07:52:38.611432 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.611480 (+ 48us) service_pool.cc:225] Handling call
1025 07:52:42.590722 (+3979242us) ts_tablet_manager.cc:1151] Deleting tablet a7ab3150a35b4caca9119a1374d3f2f7
1025 07:52:42.590731 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:42.590845 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3979018}
I20251025 07:52:42.597325 2166 tablet_replica.cc:333] T cb824bd780524832b2b60f5618a39da5 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:42.597548 2166 raft_consensus.cc:2243] T cb824bd780524832b2b60f5618a39da5 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:42.597904 909 raft_consensus.cc:2468] T f43ef51ee45d47a99ec732b49d50d3b4 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:42.599071 2471 raft_consensus.cc:2804] T 26c849d376d143c6bef0c8e0e1d7d03a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:42.599166 2166 raft_consensus.cc:2272] T cb824bd780524832b2b60f5618a39da5 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:42.599368 2471 raft_consensus.cc:697] T 26c849d376d143c6bef0c8e0e1d7d03a P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:42.599778 2166 ts_tablet_manager.cc:1916] T cb824bd780524832b2b60f5618a39da5 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:42.599742 2471 consensus_queue.cc:237] T 26c849d376d143c6bef0c8e0e1d7d03a P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.583141 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 21e52928580c4762b0ae7152490ef955 (table default.Table_e7d32e91b9c64ce1b3fcff3d806e959f [id=bab869e5001c4d999daa22083fca128a]) successfully deleted
W20251025 07:52:42.602560 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet a7ab3150a35b4caca9119a1374d3f2f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7ab3150a35b4caca9119a1374d3f2f7 already in progress: creating tablet
I20251025 07:52:42.595937 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 02e4c4d6ffb34cecaf17beb5ba197e81 (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]) successfully deleted
I20251025 07:52:42.603039 446 catalog_manager.cc:5649] T e9f7cd765d614d59a7c190b744b0844d P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } }
I20251025 07:52:42.605392 2146 raft_consensus.cc:2243] T 7c7a9d7a4fb3449bb57eafa83230a839 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:42.605578 1166 tablet_bootstrap.cc:492] T 66f0d398e59a4175aa9fe90715124d1a P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:42.606011 2508 raft_consensus.cc:493] T 91728987303b41fd840eb56f3a311c5f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:42.606199 2508 raft_consensus.cc:515] T 91728987303b41fd840eb56f3a311c5f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.606873 2508 leader_election.cc:290] T 91728987303b41fd840eb56f3a311c5f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:42.606745 2146 raft_consensus.cc:2272] T 7c7a9d7a4fb3449bb57eafa83230a839 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:42.607414 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "91728987303b41fd840eb56f3a311c5f" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:42.607928 712 leader_election.cc:343] T 91728987303b41fd840eb56f3a311c5f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.609544 2146 ts_tablet_manager.cc:1916] T 7c7a9d7a4fb3449bb57eafa83230a839 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:42.611122 1169 tablet_bootstrap.cc:492] T 292fe04055954afd8c08ac2636b68191 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:42.611268 1169 ts_tablet_manager.cc:1403] T 292fe04055954afd8c08ac2636b68191 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.001s
I20251025 07:52:42.612003 1169 raft_consensus.cc:359] T 292fe04055954afd8c08ac2636b68191 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.612182 1169 raft_consensus.cc:385] T 292fe04055954afd8c08ac2636b68191 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.612241 1169 raft_consensus.cc:740] T 292fe04055954afd8c08ac2636b68191 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.612421 1169 consensus_queue.cc:260] T 292fe04055954afd8c08ac2636b68191 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.612882 1169 ts_tablet_manager.cc:1434] T 292fe04055954afd8c08ac2636b68191 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 07:52:42.613154 1169 tablet_bootstrap.cc:492] T 04449cc97f404a19b6a1b20dea074f95 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:42.613464 2467 raft_consensus.cc:493] T b3a0af893a8947848e12093d0a7f0229 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:42.613569 2467 raft_consensus.cc:515] T b3a0af893a8947848e12093d0a7f0229 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.613667 1166 tablet_bootstrap.cc:654] T 66f0d398e59a4175aa9fe90715124d1a P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:42.614516 2508 raft_consensus.cc:493] T f656945454fe4921ab819c772a089eb3 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:42.614634 2508 raft_consensus.cc:515] T f656945454fe4921ab819c772a089eb3 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.615180 2508 leader_election.cc:290] T f656945454fe4921ab819c772a089eb3 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:42.615387 1167 tablet_bootstrap.cc:492] T ebc824ec4c60411dab1a2ea4a385148e P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:42.615585 1167 ts_tablet_manager.cc:1403] T ebc824ec4c60411dab1a2ea4a385148e P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20251025 07:52:42.615670 2467 leader_election.cc:290] T b3a0af893a8947848e12093d0a7f0229 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:42.616430 1167 raft_consensus.cc:359] T ebc824ec4c60411dab1a2ea4a385148e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.616608 1167 raft_consensus.cc:385] T ebc824ec4c60411dab1a2ea4a385148e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.616580 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f656945454fe4921ab819c772a089eb3" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:42.616626 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3a0af893a8947848e12093d0a7f0229" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:42.616688 1167 raft_consensus.cc:740] T ebc824ec4c60411dab1a2ea4a385148e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.616842 909 raft_consensus.cc:2468] T b3a0af893a8947848e12093d0a7f0229 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:42.616916 1167 consensus_queue.cc:260] T ebc824ec4c60411dab1a2ea4a385148e P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
W20251025 07:52:42.617194 982 leader_election.cc:343] T b3a0af893a8947848e12093d0a7f0229 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: b3a0af893a8947848e12093d0a7f0229
I20251025 07:52:42.617537 1167 ts_tablet_manager.cc:1434] T ebc824ec4c60411dab1a2ea4a385148e P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:42.617842 1167 tablet_bootstrap.cc:492] T 2e2e6893a48449969b71ee1c08d1960e P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:42.618247 983 leader_election.cc:304] T b3a0af893a8947848e12093d0a7f0229 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b
W20251025 07:52:42.618691 712 leader_election.cc:343] T f656945454fe4921ab819c772a089eb3 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.618932 2469 raft_consensus.cc:2804] T b3a0af893a8947848e12093d0a7f0229 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:42.618999 2469 raft_consensus.cc:493] T b3a0af893a8947848e12093d0a7f0229 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:42.619065 2469 raft_consensus.cc:3060] T b3a0af893a8947848e12093d0a7f0229 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.619488 1167 tablet_bootstrap.cc:654] T 2e2e6893a48449969b71ee1c08d1960e P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:42.621628 2469 raft_consensus.cc:515] T b3a0af893a8947848e12093d0a7f0229 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.622030 446 catalog_manager.cc:5649] T aa9d9ab8073945d28d52fb02213f201a P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } }
I20251025 07:52:42.622501 2469 leader_election.cc:290] T b3a0af893a8947848e12093d0a7f0229 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:42.622541 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet aa9d9ab8073945d28d52fb02213f201a (table default.Table_4791672658f043f6b97b9320a531ebb5 [id=dfd20cf9c3e541c4abe23d59e0aa7f2a]). Expected version 1 got 0
I20251025 07:52:42.623086 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]). Expected version 1 got 0
W20251025 07:52:42.623095 982 leader_election.cc:343] T b3a0af893a8947848e12093d0a7f0229 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: b3a0af893a8947848e12093d0a7f0229
I20251025 07:52:42.623312 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 5dab439bc50d4e869548154e1d222ad6 (table default.Table_bd82d871f84943f4baa967f5c0ec74a3 [id=45325c188eac4d34bb016d01de091f87]). Expected version 2 got 0
I20251025 07:52:42.623531 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_87d38832f85e4965bfab1a077990114f [id=ad812862698d421184ebf501ce79b7be]). Expected version 2 got 0
I20251025 07:52:42.623512 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3a0af893a8947848e12093d0a7f0229" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:42.623852 909 raft_consensus.cc:3060] T b3a0af893a8947848e12093d0a7f0229 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.624172 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet f70091024c034c118fb86578db19a31c (table default.Table_332b308220f64c6dbb5f29711d8463b3 [id=e041946c093b4c789ce175c736cc230b]). Expected version 1 got 0
I20251025 07:52:42.624576 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]). Expected version 3 got 0
I20251025 07:52:42.624765 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84]). Expected version 1 got 0
I20251025 07:52:42.625087 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 22f7db660616457b83b24e947a0b0bae (table default.Table_4a86191163f2408f9bcc795cd8d1bbf6 [id=bb316a7e000748ca902a9bfb494802d7]). Expected version 1 got 0
I20251025 07:52:42.625277 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet b56209e4934e4407a8b4518ca4af0d46 (table default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de]). Expected version 1 got 0
I20251025 07:52:42.625435 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 830fdbc6812e416690d0db42a73af503 (table default.Table_92824b3f2a774564b3aea04aaa25d204 [id=fb21a45b86ab446e8c3dae4512a9598f]). Expected version 1 got 0
I20251025 07:52:42.625574 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 755f0d6c87044e20a8020f8410940123 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 4 got 0
I20251025 07:52:42.627213 909 raft_consensus.cc:2468] T b3a0af893a8947848e12093d0a7f0229 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:42.627627 983 leader_election.cc:304] T b3a0af893a8947848e12093d0a7f0229 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b
I20251025 07:52:42.628127 2050 raft_consensus.cc:2804] T b3a0af893a8947848e12093d0a7f0229 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:42.628275 2050 raft_consensus.cc:697] T b3a0af893a8947848e12093d0a7f0229 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:42.628552 2050 consensus_queue.cc:237] T b3a0af893a8947848e12093d0a7f0229 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.629114 2169 ts_tablet_manager.cc:1929] T 3775a2e08d4e40bd9f57e4cda759d535 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:42.629408 2166 ts_tablet_manager.cc:1929] T cb824bd780524832b2b60f5618a39da5 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:42.629577 2166 log.cc:1199] T cb824bd780524832b2b60f5618a39da5 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/cb824bd780524832b2b60f5618a39da5
I20251025 07:52:42.630003 2166 ts_tablet_manager.cc:1950] T cb824bd780524832b2b60f5618a39da5 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:42.638835 2169 log.cc:1199] T 3775a2e08d4e40bd9f57e4cda759d535 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/3775a2e08d4e40bd9f57e4cda759d535
I20251025 07:52:42.638818 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 89) took 4026 ms. Trace:
I20251025 07:52:42.638924 2166 rpcz_store.cc:276] 1025 07:52:38.612823 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.612871 (+ 48us) service_pool.cc:225] Handling call
1025 07:52:42.597280 (+3984409us) ts_tablet_manager.cc:1151] Deleting tablet cb824bd780524832b2b60f5618a39da5
1025 07:52:42.597290 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:42.626345 (+ 29055us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.629320 (+ 2975us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.638792 (+ 9472us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3984230}
I20251025 07:52:42.639322 2169 ts_tablet_manager.cc:1950] T 3775a2e08d4e40bd9f57e4cda759d535 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:42.640305 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 107) took 3876 ms. Trace:
I20251025 07:52:42.640576 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 0325d462441349ef81841f1b0cab2a20 (table default.Table_750ecb905839434fa9a1aa72f87394be [id=9ce3ba77c56a4ef09cb3b40ea0e488f6]). Expected version 1 got 0
I20251025 07:52:42.640926 1166 tablet_bootstrap.cc:492] T 66f0d398e59a4175aa9fe90715124d1a P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:42.641032 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]). Expected version 1 got 0
I20251025 07:52:42.641144 1166 ts_tablet_manager.cc:1403] T 66f0d398e59a4175aa9fe90715124d1a P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.001s
I20251025 07:52:42.641790 2146 ts_tablet_manager.cc:1929] T 7c7a9d7a4fb3449bb57eafa83230a839 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:42.642036 2146 log.cc:1199] T 7c7a9d7a4fb3449bb57eafa83230a839 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/7c7a9d7a4fb3449bb57eafa83230a839
I20251025 07:52:42.642524 1166 raft_consensus.cc:359] T 66f0d398e59a4175aa9fe90715124d1a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.642740 1166 raft_consensus.cc:385] T 66f0d398e59a4175aa9fe90715124d1a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.642844 1166 raft_consensus.cc:740] T 66f0d398e59a4175aa9fe90715124d1a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.642850 2146 ts_tablet_manager.cc:1950] T 7c7a9d7a4fb3449bb57eafa83230a839 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:42.643126 1166 consensus_queue.cc:260] T 66f0d398e59a4175aa9fe90715124d1a P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.643249 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525]). Expected version 1 got 0
I20251025 07:52:42.643445 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_e90d7f5ff13b40878e5b625c97792c7d [id=fc8180c09d7b4e74b2e161f7acb8fe0e]). Expected version 1 got 0
I20251025 07:52:42.643685 1166 ts_tablet_manager.cc:1434] T 66f0d398e59a4175aa9fe90715124d1a P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:42.644258 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236]). Expected version 1 got 0
I20251025 07:52:42.644490 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c]). Expected version 1 got 0
I20251025 07:52:42.644968 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3]). Expected version 2 got 0
W20251025 07:52:42.645182 850 consensus_peers.cc:597] T 4c155aa7fa5b4d68bbab1918b8432372 P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:42.645344 2169 rpcz_store.cc:276] 1025 07:52:38.764648 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.764860 (+ 212us) service_pool.cc:225] Handling call
1025 07:52:42.546838 (+3781978us) ts_tablet_manager.cc:1151] Deleting tablet 3775a2e08d4e40bd9f57e4cda759d535
1025 07:52:42.546847 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:42.604401 (+ 57554us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.629023 (+ 24622us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.640283 (+ 11260us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3781728}
I20251025 07:52:42.645848 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 3775a2e08d4e40bd9f57e4cda759d535 (table default.Table_37ad28b796a342f3a6e962ead289ec5b [id=a4d25738410b4a168aa0c76094daf69b]) successfully deleted
I20251025 07:52:42.637202 1169 tablet_bootstrap.cc:654] T 04449cc97f404a19b6a1b20dea074f95 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:42.646483 1166 tablet_bootstrap.cc:492] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:42.648190 1166 tablet_bootstrap.cc:654] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:42.654537 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 90) took 4041 ms. Trace:
I20251025 07:52:42.654644 2166 rpcz_store.cc:276] 1025 07:52:38.613256 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.613302 (+ 46us) service_pool.cc:225] Handling call
1025 07:52:42.654370 (+4041068us) ts_tablet_manager.cc:1151] Deleting tablet cc407c7eb12e4eadb41fb21385611c94
1025 07:52:42.654383 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:42.654513 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4040872}
I20251025 07:52:42.656456 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet cb824bd780524832b2b60f5618a39da5 (table default.Table_c1743cb60e4d45939f387235c1726ef3 [id=18e29b94f8a6443fb8e6a876119fd9a3]) successfully deleted
I20251025 07:52:42.640292 1167 tablet_bootstrap.cc:492] T 2e2e6893a48449969b71ee1c08d1960e P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:42.656782 1167 ts_tablet_manager.cc:1403] T 2e2e6893a48449969b71ee1c08d1960e P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
I20251025 07:52:42.657765 1167 raft_consensus.cc:359] T 2e2e6893a48449969b71ee1c08d1960e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.658012 1167 raft_consensus.cc:385] T 2e2e6893a48449969b71ee1c08d1960e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.658111 1167 raft_consensus.cc:740] T 2e2e6893a48449969b71ee1c08d1960e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.658397 1167 consensus_queue.cc:260] T 2e2e6893a48449969b71ee1c08d1960e P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.659148 1167 ts_tablet_manager.cc:1434] T 2e2e6893a48449969b71ee1c08d1960e P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:42.659497 1167 tablet_bootstrap.cc:492] T c0c6324cbae34dcebade3ef9f89a7150 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:42.660436 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 91) took 4045 ms. Trace:
W20251025 07:52:42.660449 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet cc407c7eb12e4eadb41fb21385611c94 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc407c7eb12e4eadb41fb21385611c94 already in progress: creating tablet
I20251025 07:52:42.660583 2166 rpcz_store.cc:276] 1025 07:52:38.615314 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.615370 (+ 56us) service_pool.cc:225] Handling call
1025 07:52:42.660289 (+4044919us) ts_tablet_manager.cc:1151] Deleting tablet 2e098c9c4613459295b6043048cba587
1025 07:52:42.660302 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:42.660417 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4044739}
I20251025 07:52:42.660962 2166 tablet_replica.cc:333] T abf94c32ff2945119b710905c075d63c P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
W20251025 07:52:42.663663 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 2e098c9c4613459295b6043048cba587 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e098c9c4613459295b6043048cba587 already in progress: creating tablet
I20251025 07:52:42.664017 2166 raft_consensus.cc:2243] T abf94c32ff2945119b710905c075d63c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:42.663717 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 117) took 4148 ms. Trace:
I20251025 07:52:42.664335 2146 rpcz_store.cc:276] 1025 07:52:38.518673 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.518847 (+ 174us) service_pool.cc:225] Handling call
1025 07:52:42.595511 (+4076664us) ts_tablet_manager.cc:1151] Deleting tablet 7c7a9d7a4fb3449bb57eafa83230a839
1025 07:52:42.595520 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:42.628651 (+ 33131us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.641107 (+ 12456us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.663506 (+ 22399us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":435,"tablet-delete.queue_time_us":4071132}
I20251025 07:52:42.666011 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 7c7a9d7a4fb3449bb57eafa83230a839 (table default.Table_192549a3e67840f3aefc31029cf2ff52 [id=a37be02f8e994f82baf2f4f5ec800926]) successfully deleted
I20251025 07:52:42.667204 2146 tablet_replica.cc:333] T 70fe9aa2f58d4abea3b3b71066a29653 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:42.668730 2146 raft_consensus.cc:2243] T 70fe9aa2f58d4abea3b3b71066a29653 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:42.669363 2166 raft_consensus.cc:2272] T abf94c32ff2945119b710905c075d63c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:42.669756 2146 raft_consensus.cc:2272] T 70fe9aa2f58d4abea3b3b71066a29653 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:42.670584 2166 ts_tablet_manager.cc:1916] T abf94c32ff2945119b710905c075d63c P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:42.672658 1167 tablet_bootstrap.cc:654] T c0c6324cbae34dcebade3ef9f89a7150 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:42.672590 710 leader_election.cc:304] T 36c8fcafd5464b4e999276a8062a812a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.673928 2146 ts_tablet_manager.cc:1916] T 70fe9aa2f58d4abea3b3b71066a29653 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:42.674451 1166 tablet_bootstrap.cc:492] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:42.674439 2508 raft_consensus.cc:2749] T 36c8fcafd5464b4e999276a8062a812a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:42.674625 1166 ts_tablet_manager.cc:1403] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.002s
W20251025 07:52:42.674793 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:43563, user_credentials={real_user=slave}} blocked reactor thread for 159651us
I20251025 07:52:42.674926 2169 tablet_replica.cc:333] T 627a10a0714f4be8bdbced503fd6388d P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:42.675136 2169 raft_consensus.cc:2243] T 627a10a0714f4be8bdbced503fd6388d P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:42.675318 2169 raft_consensus.cc:2272] T 627a10a0714f4be8bdbced503fd6388d P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:42.675483 1166 raft_consensus.cc:359] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.675665 1166 raft_consensus.cc:385] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.675756 1166 raft_consensus.cc:740] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.675782 2169 ts_tablet_manager.cc:1916] T 627a10a0714f4be8bdbced503fd6388d P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:42.676005 1166 consensus_queue.cc:260] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.676632 1166 ts_tablet_manager.cc:1434] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:42.676892 1166 tablet_bootstrap.cc:492] T 04449cc97f404a19b6a1b20dea074f95 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:42.679018 1166 tablet_bootstrap.cc:654] T 04449cc97f404a19b6a1b20dea074f95 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:42.682590 2508 raft_consensus.cc:493] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:42.682714 2508 raft_consensus.cc:515] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.683264 2508 leader_election.cc:290] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:42.684000 2508 raft_consensus.cc:493] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:42.684128 2508 raft_consensus.cc:515] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
W20251025 07:52:42.684233 712 leader_election.cc:343] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 87a93778620f4c84b3b3f7a3d7d7e87f
I20251025 07:52:42.684604 2508 leader_election.cc:290] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:42.684882 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0551e5d694894e8fbeb66b5e0c1cd75b" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:42.685364 712 leader_election.cc:343] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.688334 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 36262ab8c6964dd684c01b4db6219a4f (table default.Table_76158f8cf82e4cdb9a90580e46aaea48 [id=f249f30497bc450c887d659721469de8]). Expected version 1 got 0
I20251025 07:52:42.689388 1169 tablet_bootstrap.cc:492] T 04449cc97f404a19b6a1b20dea074f95 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:42.689576 1169 ts_tablet_manager.cc:1403] T 04449cc97f404a19b6a1b20dea074f95 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.076s user 0.004s sys 0.000s
I20251025 07:52:42.689848 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet ef9a2c9f2ccf432ab835396020872208 (table default.Table_2b9b772d4c234ac286bef3658482e490 [id=409fab60f9d744798e6f92b868f0f7c2]). Expected version 1 got 0
I20251025 07:52:42.690476 1169 raft_consensus.cc:359] T 04449cc97f404a19b6a1b20dea074f95 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.690688 1169 raft_consensus.cc:385] T 04449cc97f404a19b6a1b20dea074f95 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.690770 1169 raft_consensus.cc:740] T 04449cc97f404a19b6a1b20dea074f95 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.691005 1169 consensus_queue.cc:260] T 04449cc97f404a19b6a1b20dea074f95 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.682349 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f656945454fe4921ab819c772a089eb3" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:42.694221 1169 ts_tablet_manager.cc:1434] T 04449cc97f404a19b6a1b20dea074f95 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251025 07:52:42.682803 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "91728987303b41fd840eb56f3a311c5f" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:42.685510 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0551e5d694894e8fbeb66b5e0c1cd75b" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:42.694487 1169 tablet_bootstrap.cc:492] T d9bfd7c0c3ad4e5b98b6708793d61d41 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:42.688380 2314 consensus_queue.cc:1048] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:42.694561 910 raft_consensus.cc:2468] T 0551e5d694894e8fbeb66b5e0c1cd75b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
W20251025 07:52:42.695120 710 leader_election.cc:343] T 887e7c7a5d0e4aaf88f26246306df19c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 887e7c7a5d0e4aaf88f26246306df19c
I20251025 07:52:42.695236 710 leader_election.cc:304] T 887e7c7a5d0e4aaf88f26246306df19c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.695487 1167 tablet_bootstrap.cc:492] T c0c6324cbae34dcebade3ef9f89a7150 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:42.695611 1167 ts_tablet_manager.cc:1403] T c0c6324cbae34dcebade3ef9f89a7150 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.000s
I20251025 07:52:42.700544 2166 ts_tablet_manager.cc:1929] T abf94c32ff2945119b710905c075d63c P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:42.712134 1045 raft_consensus.cc:3060] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.713340 1166 tablet_bootstrap.cc:492] T 04449cc97f404a19b6a1b20dea074f95 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:42.713513 1166 ts_tablet_manager.cc:1403] T 04449cc97f404a19b6a1b20dea074f95 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.003s
I20251025 07:52:42.714267 1166 raft_consensus.cc:359] T 04449cc97f404a19b6a1b20dea074f95 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.714463 1166 raft_consensus.cc:385] T 04449cc97f404a19b6a1b20dea074f95 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.714529 1166 raft_consensus.cc:740] T 04449cc97f404a19b6a1b20dea074f95 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.714773 1166 consensus_queue.cc:260] T 04449cc97f404a19b6a1b20dea074f95 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.690043 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]). Expected version 1 got 0
I20251025 07:52:42.715320 1166 ts_tablet_manager.cc:1434] T 04449cc97f404a19b6a1b20dea074f95 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:42.715616 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet ab42909afc524f2fa3c1041495d13a59 (table default.Table_7eb8424d5192485da1a2d67bfdb2bbbf [id=40c29df99590408cbd5c583d4fc741eb]). Expected version 2 got 0
I20251025 07:52:42.715850 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e]). Expected version 1 got 0
I20251025 07:52:42.716089 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet bc2d16b33ca8426abbd7bd38339c9f1d (table default.Table_4ea985df4c5b415c9570c64506a98cc4 [id=98d2b7797e33485e8872054aa2eb8ec9]). Expected version 1 got 0
I20251025 07:52:42.716296 2314 consensus_queue.cc:1048] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:42.718971 2314 raft_consensus.cc:493] T c9f7ba3709c745d4a625449d6e1c8ba1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:42.719127 2314 raft_consensus.cc:515] T c9f7ba3709c745d4a625449d6e1c8ba1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.719666 2314 leader_election.cc:290] T c9f7ba3709c745d4a625449d6e1c8ba1 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:42.720191 1169 tablet_bootstrap.cc:654] T d9bfd7c0c3ad4e5b98b6708793d61d41 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:42.720235 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9f7ba3709c745d4a625449d6e1c8ba1" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:42.720912 850 leader_election.cc:343] T c9f7ba3709c745d4a625449d6e1c8ba1 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.723433 1166 tablet_bootstrap.cc:492] T 36c8fcafd5464b4e999276a8062a812a P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:42.725206 1166 tablet_bootstrap.cc:654] T 36c8fcafd5464b4e999276a8062a812a P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:42.700726 2508 raft_consensus.cc:2749] T 887e7c7a5d0e4aaf88f26246306df19c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 07:52:42.703270 1167 raft_consensus.cc:359] T c0c6324cbae34dcebade3ef9f89a7150 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.735190 1167 raft_consensus.cc:385] T c0c6324cbae34dcebade3ef9f89a7150 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.735281 1167 raft_consensus.cc:740] T c0c6324cbae34dcebade3ef9f89a7150 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.735606 1167 consensus_queue.cc:260] T c0c6324cbae34dcebade3ef9f89a7150 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.736285 1167 ts_tablet_manager.cc:1434] T c0c6324cbae34dcebade3ef9f89a7150 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.041s user 0.002s sys 0.000s
I20251025 07:52:42.736596 1167 tablet_bootstrap.cc:492] T 37c7938e8c5e41f08506aff6747801c9 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:42.738559 1167 tablet_bootstrap.cc:654] T 37c7938e8c5e41f08506aff6747801c9 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:42.717134 710 leader_election.cc:343] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 87a93778620f4c84b3b3f7a3d7d7e87f
I20251025 07:52:42.720080 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9f7ba3709c745d4a625449d6e1c8ba1" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:42.739926 773 raft_consensus.cc:2468] T c9f7ba3709c745d4a625449d6e1c8ba1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:42.740478 847 leader_election.cc:304] T c9f7ba3709c745d4a625449d6e1c8ba1 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.741096 2314 raft_consensus.cc:2804] T c9f7ba3709c745d4a625449d6e1c8ba1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:42.741210 2314 raft_consensus.cc:493] T c9f7ba3709c745d4a625449d6e1c8ba1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:42.741362 2314 raft_consensus.cc:3060] T c9f7ba3709c745d4a625449d6e1c8ba1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.741850 1166 tablet_bootstrap.cc:492] T 36c8fcafd5464b4e999276a8062a812a P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:42.741995 1166 ts_tablet_manager.cc:1403] T 36c8fcafd5464b4e999276a8062a812a P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.001s
I20251025 07:52:42.742794 1166 raft_consensus.cc:359] T 36c8fcafd5464b4e999276a8062a812a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.742976 1166 raft_consensus.cc:385] T 36c8fcafd5464b4e999276a8062a812a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.743073 1166 raft_consensus.cc:740] T 36c8fcafd5464b4e999276a8062a812a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.743276 1166 consensus_queue.cc:260] T 36c8fcafd5464b4e999276a8062a812a P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.744911 1166 ts_tablet_manager.cc:1434] T 36c8fcafd5464b4e999276a8062a812a P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251025 07:52:42.745209 1166 tablet_bootstrap.cc:492] T f661d5909657455581dcb5e6eadd3c0b P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:42.745221 2314 raft_consensus.cc:515] T c9f7ba3709c745d4a625449d6e1c8ba1 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.747249 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9f7ba3709c745d4a625449d6e1c8ba1" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:42.747416 773 raft_consensus.cc:3060] T c9f7ba3709c745d4a625449d6e1c8ba1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.747573 1166 tablet_bootstrap.cc:654] T f661d5909657455581dcb5e6eadd3c0b P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:42.750257 2314 leader_election.cc:290] T c9f7ba3709c745d4a625449d6e1c8ba1 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:42.750613 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9f7ba3709c745d4a625449d6e1c8ba1" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:42.751334 850 leader_election.cc:343] T c9f7ba3709c745d4a625449d6e1c8ba1 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.751968 773 raft_consensus.cc:2468] T c9f7ba3709c745d4a625449d6e1c8ba1 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:42.752542 847 leader_election.cc:304] T c9f7ba3709c745d4a625449d6e1c8ba1 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.753063 2169 ts_tablet_manager.cc:1929] T 627a10a0714f4be8bdbced503fd6388d P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 07:52:42.753132 2314 raft_consensus.cc:2804] T c9f7ba3709c745d4a625449d6e1c8ba1 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:42.753373 2314 raft_consensus.cc:697] T c9f7ba3709c745d4a625449d6e1c8ba1 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:42.753703 2314 consensus_queue.cc:237] T c9f7ba3709c745d4a625449d6e1c8ba1 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.755088 710 leader_election.cc:304] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.755446 2117 raft_consensus.cc:2749] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:42.755846 710 consensus_peers.cc:597] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 23eeffd3de254ce18db5febfde971c6b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02e4c4d6ffb34cecaf17beb5ba197e81. This is attempt 1: this message will repeat every 5th retry.
W20251025 07:52:42.756201 710 leader_election.cc:343] T f656945454fe4921ab819c772a089eb3 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.756409 710 leader_election.cc:304] T f656945454fe4921ab819c772a089eb3 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.756817 2117 raft_consensus.cc:2749] T f656945454fe4921ab819c772a089eb3 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:42.756974 710 leader_election.cc:343] T 91728987303b41fd840eb56f3a311c5f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.757120 710 leader_election.cc:304] T 91728987303b41fd840eb56f3a311c5f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.757333 2117 raft_consensus.cc:2749] T 91728987303b41fd840eb56f3a311c5f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:42.757524 710 leader_election.cc:304] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.757865 2117 raft_consensus.cc:2804] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:42.757931 2117 raft_consensus.cc:493] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:42.758001 2117 raft_consensus.cc:3060] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.758193 2420 raft_consensus.cc:493] T 11bfe0c87ca941a3846462c8a4512d83 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:42.758273 2420 raft_consensus.cc:515] T 11bfe0c87ca941a3846462c8a4512d83 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.758901 2420 leader_election.cc:290] T 11bfe0c87ca941a3846462c8a4512d83 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:42.760836 1166 tablet_bootstrap.cc:492] T f661d5909657455581dcb5e6eadd3c0b P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:42.762646 2117 raft_consensus.cc:515] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.764015 2117 leader_election.cc:290] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:42.762399 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11bfe0c87ca941a3846462c8a4512d83" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:42.764276 1166 ts_tablet_manager.cc:1403] T f661d5909657455581dcb5e6eadd3c0b P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.004s
I20251025 07:52:42.765039 1166 raft_consensus.cc:359] T f661d5909657455581dcb5e6eadd3c0b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.765264 1166 raft_consensus.cc:385] T f661d5909657455581dcb5e6eadd3c0b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.763514 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0551e5d694894e8fbeb66b5e0c1cd75b" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:42.770215 1166 raft_consensus.cc:740] T f661d5909657455581dcb5e6eadd3c0b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.764245 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11bfe0c87ca941a3846462c8a4512d83" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:42.770587 712 leader_election.cc:343] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.770519 1166 consensus_queue.cc:260] T f661d5909657455581dcb5e6eadd3c0b P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
W20251025 07:52:42.770793 712 leader_election.cc:343] T 11bfe0c87ca941a3846462c8a4512d83 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.770989 2169 log.cc:1199] T 627a10a0714f4be8bdbced503fd6388d P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/627a10a0714f4be8bdbced503fd6388d
I20251025 07:52:42.771095 1166 ts_tablet_manager.cc:1434] T f661d5909657455581dcb5e6eadd3c0b P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20251025 07:52:42.777935 2169 ts_tablet_manager.cc:1950] T 627a10a0714f4be8bdbced503fd6388d P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:42.778772 1169 tablet_bootstrap.cc:492] T d9bfd7c0c3ad4e5b98b6708793d61d41 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:42.779009 1169 ts_tablet_manager.cc:1403] T d9bfd7c0c3ad4e5b98b6708793d61d41 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.085s user 0.004s sys 0.000s
I20251025 07:52:42.780179 1169 raft_consensus.cc:359] T d9bfd7c0c3ad4e5b98b6708793d61d41 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.780480 1169 raft_consensus.cc:385] T d9bfd7c0c3ad4e5b98b6708793d61d41 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.780694 1169 raft_consensus.cc:740] T d9bfd7c0c3ad4e5b98b6708793d61d41 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.781342 1169 consensus_queue.cc:260] T d9bfd7c0c3ad4e5b98b6708793d61d41 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.782552 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0551e5d694894e8fbeb66b5e0c1cd75b" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:42.782718 902 raft_consensus.cc:3060] T 0551e5d694894e8fbeb66b5e0c1cd75b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.785509 902 raft_consensus.cc:2468] T 0551e5d694894e8fbeb66b5e0c1cd75b P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:42.786916 2544 tablet.cc:1722] T ef9a2c9f2ccf432ab835396020872208 P 3aea2869ff814cc7b142012faa274382: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251025 07:52:42.788687 710 leader_election.cc:343] T 11bfe0c87ca941a3846462c8a4512d83 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.788961 710 leader_election.cc:304] T 11bfe0c87ca941a3846462c8a4512d83 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.789296 2117 raft_consensus.cc:2749] T 11bfe0c87ca941a3846462c8a4512d83 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:42.789702 710 leader_election.cc:304] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.789902 2166 log.cc:1199] T abf94c32ff2945119b710905c075d63c P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/abf94c32ff2945119b710905c075d63c
I20251025 07:52:42.790264 2545 tablet.cc:1722] T ef9a2c9f2ccf432ab835396020872208 P 23eeffd3de254ce18db5febfde971c6b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 07:52:42.790508 2117 raft_consensus.cc:2804] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:42.790755 2117 raft_consensus.cc:697] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:42.791060 2117 consensus_queue.cc:237] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.793097 2146 ts_tablet_manager.cc:1929] T 70fe9aa2f58d4abea3b3b71066a29653 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:42.802685 2146 log.cc:1199] T 70fe9aa2f58d4abea3b3b71066a29653 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/70fe9aa2f58d4abea3b3b71066a29653
I20251025 07:52:42.804011 2166 ts_tablet_manager.cc:1950] T abf94c32ff2945119b710905c075d63c P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:42.805526 2146 ts_tablet_manager.cc:1950] T 70fe9aa2f58d4abea3b3b71066a29653 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:42.808650 2117 raft_consensus.cc:493] T a6ddefa1738f4ff98586900c344359d2 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db42be7ad3c741a0aa427f27e245993b)
I20251025 07:52:42.808835 2117 raft_consensus.cc:515] T a6ddefa1738f4ff98586900c344359d2 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:42.810081 712 leader_election.cc:343] T a6ddefa1738f4ff98586900c344359d2 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: a6ddefa1738f4ff98586900c344359d2
I20251025 07:52:42.810429 2117 leader_election.cc:290] T a6ddefa1738f4ff98586900c344359d2 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:42.782729 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 108) took 4013 ms. Trace:
I20251025 07:52:42.786484 2543 tablet.cc:1722] T ef9a2c9f2ccf432ab835396020872208 P db42be7ad3c741a0aa427f27e245993b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 07:52:42.810669 2169 rpcz_store.cc:276] 1025 07:52:38.769957 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.770029 (+ 72us) service_pool.cc:225] Handling call
1025 07:52:42.645702 (+3875673us) ts_tablet_manager.cc:1151] Deleting tablet 627a10a0714f4be8bdbced503fd6388d
1025 07:52:42.645712 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:42.692420 (+ 46708us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.752955 (+ 60535us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.782709 (+ 29754us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3875456}
I20251025 07:52:42.798264 2467 raft_consensus.cc:493] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 23eeffd3de254ce18db5febfde971c6b)
I20251025 07:52:42.811179 2467 raft_consensus.cc:515] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.811262 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 627a10a0714f4be8bdbced503fd6388d (table default.Table_8773c967a71644baba838946c154aee2 [id=7fc07b0c33f74604be526c4a0df949e9]) successfully deleted
I20251025 07:52:42.811615 2169 tablet_replica.cc:333] T 35ba9d23262248cda6fdf03c51be11a4 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:42.811823 2467 leader_election.cc:290] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:42.811980 710 leader_election.cc:343] T a6ddefa1738f4ff98586900c344359d2 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: a6ddefa1738f4ff98586900c344359d2
I20251025 07:52:42.812108 710 leader_election.cc:304] T a6ddefa1738f4ff98586900c344359d2 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:42.812650 982 leader_election.cc:343] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: 7fbae8be73bf4e18b5a5c4ec4cdebc12
I20251025 07:52:42.812688 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7fbae8be73bf4e18b5a5c4ec4cdebc12" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:42.812810 2117 raft_consensus.cc:2749] T a6ddefa1738f4ff98586900c344359d2 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 07:52:42.813056 2169 raft_consensus.cc:2243] T 35ba9d23262248cda6fdf03c51be11a4 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:42.813177 983 leader_election.cc:304] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:42.813365 1169 ts_tablet_manager.cc:1434] T d9bfd7c0c3ad4e5b98b6708793d61d41 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.034s user 0.002s sys 0.000s
I20251025 07:52:42.813457 2169 raft_consensus.cc:2272] T 35ba9d23262248cda6fdf03c51be11a4 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:42.813804 1169 tablet_bootstrap.cc:492] T a4e4b99be0434baaa75d988281797846 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:42.814016 2169 ts_tablet_manager.cc:1916] T 35ba9d23262248cda6fdf03c51be11a4 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:42.814220 2467 raft_consensus.cc:2749] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 07:52:42.816857 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 118) took 4297 ms. Trace:
I20251025 07:52:42.819366 1169 tablet_bootstrap.cc:654] T a4e4b99be0434baaa75d988281797846 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:42.817548 2146 rpcz_store.cc:276] 1025 07:52:38.518837 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.519033 (+ 196us) service_pool.cc:225] Handling call
1025 07:52:42.666646 (+4147613us) ts_tablet_manager.cc:1151] Deleting tablet 70fe9aa2f58d4abea3b3b71066a29653
1025 07:52:42.666818 (+ 172us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:42.740945 (+ 74127us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.792988 (+ 52043us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.815905 (+ 22917us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":455,"tablet-delete.queue_time_us":4142224}
I20251025 07:52:42.820112 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 70fe9aa2f58d4abea3b3b71066a29653 (table default.Table_517752d265854c7cbd706e2742536620 [id=aa0642dd1fa34cf8bd1fe2070037a55a]) successfully deleted
I20251025 07:52:42.821709 2146 tablet_replica.cc:333] T c59f6e6eb9594a8e83f109a09b034045 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:42.823192 2146 raft_consensus.cc:2243] T c59f6e6eb9594a8e83f109a09b034045 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:42.825583 2146 raft_consensus.cc:2272] T c59f6e6eb9594a8e83f109a09b034045 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:42.827723 1167 tablet_bootstrap.cc:492] T 37c7938e8c5e41f08506aff6747801c9 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:42.827934 1167 ts_tablet_manager.cc:1403] T 37c7938e8c5e41f08506aff6747801c9 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.091s user 0.004s sys 0.004s
I20251025 07:52:42.829293 2146 ts_tablet_manager.cc:1916] T c59f6e6eb9594a8e83f109a09b034045 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:42.830401 1167 raft_consensus.cc:359] T 37c7938e8c5e41f08506aff6747801c9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.830675 1167 raft_consensus.cc:385] T 37c7938e8c5e41f08506aff6747801c9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.830797 1167 raft_consensus.cc:740] T 37c7938e8c5e41f08506aff6747801c9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.831069 1167 consensus_queue.cc:260] T 37c7938e8c5e41f08506aff6747801c9 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.831612 1167 ts_tablet_manager.cc:1434] T 37c7938e8c5e41f08506aff6747801c9 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251025 07:52:42.833822 1166 tablet_bootstrap.cc:492] T b2044ca206634eb285197af4ba2073e7 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:42.838308 2467 raft_consensus.cc:493] T 04449cc97f404a19b6a1b20dea074f95 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:42.838451 2467 raft_consensus.cc:515] T 04449cc97f404a19b6a1b20dea074f95 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.839037 2467 leader_election.cc:290] T 04449cc97f404a19b6a1b20dea074f95 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:42.839570 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04449cc97f404a19b6a1b20dea074f95" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:42.839743 773 raft_consensus.cc:2468] T 04449cc97f404a19b6a1b20dea074f95 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:42.840159 982 leader_election.cc:304] T 04449cc97f404a19b6a1b20dea074f95 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:42.840413 2467 raft_consensus.cc:2804] T 04449cc97f404a19b6a1b20dea074f95 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:42.840498 2467 raft_consensus.cc:493] T 04449cc97f404a19b6a1b20dea074f95 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:42.840559 2467 raft_consensus.cc:3060] T 04449cc97f404a19b6a1b20dea074f95 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.842108 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 92) took 4225 ms. Trace:
I20251025 07:52:42.842206 2166 rpcz_store.cc:276] 1025 07:52:38.616785 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.616825 (+ 40us) service_pool.cc:225] Handling call
1025 07:52:42.660922 (+4044097us) ts_tablet_manager.cc:1151] Deleting tablet abf94c32ff2945119b710905c075d63c
1025 07:52:42.660930 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:42.697499 (+ 36569us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.700465 (+ 2966us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.842091 (+141626us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5142,"tablet-delete.queue_time_us":4043909}
I20251025 07:52:42.842654 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet abf94c32ff2945119b710905c075d63c (table default.Table_4b63840ca3ac4296b23104bf46030bb4 [id=0010b5f16dd14d64b8c93d941803cc75]) successfully deleted
I20251025 07:52:42.843101 2166 tablet_replica.cc:333] T 1912e59548874e0e8e989050ec0064c5 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:42.843273 2166 raft_consensus.cc:2243] T 1912e59548874e0e8e989050ec0064c5 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:42.843276 2467 raft_consensus.cc:515] T 04449cc97f404a19b6a1b20dea074f95 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.843397 2166 raft_consensus.cc:2272] T 1912e59548874e0e8e989050ec0064c5 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:42.843919 2166 ts_tablet_manager.cc:1916] T 1912e59548874e0e8e989050ec0064c5 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:42.844880 2467 leader_election.cc:290] T 04449cc97f404a19b6a1b20dea074f95 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:42.845144 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04449cc97f404a19b6a1b20dea074f95" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:42.845320 774 raft_consensus.cc:3060] T 04449cc97f404a19b6a1b20dea074f95 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.846444 1167 tablet_bootstrap.cc:492] T 627a9cfd398346f2aa34c8932a11b5a8 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:42.848116 1167 tablet_bootstrap.cc:654] T 627a9cfd398346f2aa34c8932a11b5a8 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:42.848201 774 raft_consensus.cc:2468] T 04449cc97f404a19b6a1b20dea074f95 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:42.848747 982 leader_election.cc:304] T 04449cc97f404a19b6a1b20dea074f95 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:42.849046 2467 raft_consensus.cc:2804] T 04449cc97f404a19b6a1b20dea074f95 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:42.849291 2467 raft_consensus.cc:697] T 04449cc97f404a19b6a1b20dea074f95 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:42.849608 2467 consensus_queue.cc:237] T 04449cc97f404a19b6a1b20dea074f95 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.839759 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04449cc97f404a19b6a1b20dea074f95" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:42.841269 2105 consensus_queue.cc:1048] T 724d0b7e1b65445a9a1f056852983bdb P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:42.851945 910 raft_consensus.cc:2468] T 04449cc97f404a19b6a1b20dea074f95 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:42.844761 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04449cc97f404a19b6a1b20dea074f95" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:42.852241 902 raft_consensus.cc:3060] T 04449cc97f404a19b6a1b20dea074f95 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.853303 1166 tablet_bootstrap.cc:654] T b2044ca206634eb285197af4ba2073e7 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:42.855041 902 raft_consensus.cc:2468] T 04449cc97f404a19b6a1b20dea074f95 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:42.863161 2484 raft_consensus.cc:493] T 292fe04055954afd8c08ac2636b68191 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:42.863305 2484 raft_consensus.cc:515] T 292fe04055954afd8c08ac2636b68191 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.863969 2484 leader_election.cc:290] T 292fe04055954afd8c08ac2636b68191 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:42.864646 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "292fe04055954afd8c08ac2636b68191" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:42.864840 773 raft_consensus.cc:2468] T 292fe04055954afd8c08ac2636b68191 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
W20251025 07:52:42.865177 850 consensus_peers.cc:597] T aaef24ccdfba45fe8204327723df834e P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:42.865228 982 leader_election.cc:304] T 292fe04055954afd8c08ac2636b68191 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:42.865520 2484 raft_consensus.cc:2804] T 292fe04055954afd8c08ac2636b68191 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:42.865586 2484 raft_consensus.cc:493] T 292fe04055954afd8c08ac2636b68191 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:42.865641 2484 raft_consensus.cc:3060] T 292fe04055954afd8c08ac2636b68191 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.866701 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "292fe04055954afd8c08ac2636b68191" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:42.866876 902 raft_consensus.cc:2468] T 292fe04055954afd8c08ac2636b68191 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:42.868314 2484 raft_consensus.cc:515] T 292fe04055954afd8c08ac2636b68191 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.869146 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "292fe04055954afd8c08ac2636b68191" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:42.869354 770 raft_consensus.cc:3060] T 292fe04055954afd8c08ac2636b68191 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.869695 2105 consensus_queue.cc:1048] T aaef24ccdfba45fe8204327723df834e P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:42.869990 2484 leader_election.cc:290] T 292fe04055954afd8c08ac2636b68191 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:42.872331 770 raft_consensus.cc:2468] T 292fe04055954afd8c08ac2636b68191 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:42.872834 982 leader_election.cc:304] T 292fe04055954afd8c08ac2636b68191 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:42.873162 2484 raft_consensus.cc:2804] T 292fe04055954afd8c08ac2636b68191 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:42.873126 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "292fe04055954afd8c08ac2636b68191" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:42.873271 910 raft_consensus.cc:3060] T 292fe04055954afd8c08ac2636b68191 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.873456 2484 raft_consensus.cc:697] T 292fe04055954afd8c08ac2636b68191 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:42.873788 2484 consensus_queue.cc:237] T 292fe04055954afd8c08ac2636b68191 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.876039 910 raft_consensus.cc:2468] T 292fe04055954afd8c08ac2636b68191 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:42.899925 2166 ts_tablet_manager.cc:1929] T 1912e59548874e0e8e989050ec0064c5 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:42.904402 2169 ts_tablet_manager.cc:1929] T 35ba9d23262248cda6fdf03c51be11a4 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:42.904718 2169 log.cc:1199] T 35ba9d23262248cda6fdf03c51be11a4 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/35ba9d23262248cda6fdf03c51be11a4
I20251025 07:52:42.905284 2169 ts_tablet_manager.cc:1950] T 35ba9d23262248cda6fdf03c51be11a4 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:42.910585 2106 consensus_queue.cc:1048] T 724d0b7e1b65445a9a1f056852983bdb P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:42.908566 1166 tablet_bootstrap.cc:492] T b2044ca206634eb285197af4ba2073e7 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:42.923293 1166 ts_tablet_manager.cc:1403] T b2044ca206634eb285197af4ba2073e7 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.090s user 0.005s sys 0.001s
I20251025 07:52:42.927768 2106 raft_consensus.cc:493] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:42.927964 2106 raft_consensus.cc:515] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.929222 2106 leader_election.cc:290] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:42.936728 1166 raft_consensus.cc:359] T b2044ca206634eb285197af4ba2073e7 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.937817 1166 raft_consensus.cc:385] T b2044ca206634eb285197af4ba2073e7 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.910182 1169 tablet_bootstrap.cc:492] T a4e4b99be0434baaa75d988281797846 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:42.938500 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f]). Expected version 1 got 0
I20251025 07:52:42.938681 1169 ts_tablet_manager.cc:1403] T a4e4b99be0434baaa75d988281797846 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.125s user 0.009s sys 0.007s
I20251025 07:52:42.938920 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 4c155aa7fa5b4d68bbab1918b8432372 (table default.Table_6c5a24747ba14352b0285a41ec2985a4 [id=28bbaad474164a8c8c97a76bb8081791]). Expected version 2 got 0
I20251025 07:52:42.938514 1166 raft_consensus.cc:740] T b2044ca206634eb285197af4ba2073e7 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.939172 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7]). Expected version 1 got 0
I20251025 07:52:42.939378 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 04d27d54fdb7483b869408af65ff85c9 (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]). Expected version 2 got 0
I20251025 07:52:42.939481 1169 raft_consensus.cc:359] T a4e4b99be0434baaa75d988281797846 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:42.939644 1169 raft_consensus.cc:385] T a4e4b99be0434baaa75d988281797846 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.939661 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet cb829649f73d4d3f8862d268c953bd86 (table default.Table_ba4b8cc7b930482690b7e192e0edff12 [id=7944225d3ce7439b881bfb9289d9f040]). Expected version 1 got 0
I20251025 07:52:42.939698 1169 raft_consensus.cc:740] T a4e4b99be0434baaa75d988281797846 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.939890 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 872cb7512d104827b1f1f186d33a4a44 (table default.Table_d5c65dce8b1b4214ba519d964012bfc0 [id=d4be274b36a845a99a4b96b8846248fd]). Expected version 1 got 0
I20251025 07:52:42.939889 1169 consensus_queue.cc:260] T a4e4b99be0434baaa75d988281797846 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:42.940385 1169 ts_tablet_manager.cc:1434] T a4e4b99be0434baaa75d988281797846 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:42.940688 1169 tablet_bootstrap.cc:492] T 0f5f6ca6c3cd4d369690e425387a4344 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:42.940245 2146 ts_tablet_manager.cc:1929] T c59f6e6eb9594a8e83f109a09b034045 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:42.940799 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet aa9d9ab8073945d28d52fb02213f201a (table default.Table_4791672658f043f6b97b9320a531ebb5 [id=dfd20cf9c3e541c4abe23d59e0aa7f2a]). Expected version 1 got 0
I20251025 07:52:42.941037 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]). Expected version 1 got 0
I20251025 07:52:42.941172 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet ef9a2c9f2ccf432ab835396020872208 (table default.Table_2b9b772d4c234ac286bef3658482e490 [id=409fab60f9d744798e6f92b868f0f7c2]). Expected version 1 got 0
I20251025 07:52:42.941331 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]). Expected version 1 got 0
I20251025 07:52:42.941524 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_a6c869949b744bc98efb735b0e73e7aa [id=bf77b5ff6f1048dd996d948437f9ff5e]). Expected version 2 got 0
I20251025 07:52:42.941421 2146 log.cc:1199] T c59f6e6eb9594a8e83f109a09b034045 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/c59f6e6eb9594a8e83f109a09b034045
I20251025 07:52:42.940491 1166 consensus_queue.cc:260] T b2044ca206634eb285197af4ba2073e7 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.941911 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet ef1854d35a3047e3a9fbe0166ab1295a (table default.Table_088aad2a46b041bfa29ed8e6929760b1 [id=834669c54dae449790818684fd8659bc]). Expected version 2 got 0
I20251025 07:52:42.942673 1169 tablet_bootstrap.cc:654] T 0f5f6ca6c3cd4d369690e425387a4344 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:42.929705 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c87ab6a3d08f43e0b2fe58a6d7b11588" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:42.944049 850 leader_election.cc:343] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.937150 2484 raft_consensus.cc:493] T d9bfd7c0c3ad4e5b98b6708793d61d41 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:42.942158 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet bc2d16b33ca8426abbd7bd38339c9f1d (table default.Table_4ea985df4c5b415c9570c64506a98cc4 [id=98d2b7797e33485e8872054aa2eb8ec9]). Expected version 1 got 0
I20251025 07:52:42.944346 2484 raft_consensus.cc:515] T d9bfd7c0c3ad4e5b98b6708793d61d41 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:42.944584 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e]). Expected version 1 got 0
I20251025 07:52:42.944839 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet ab42909afc524f2fa3c1041495d13a59 (table default.Table_7eb8424d5192485da1a2d67bfdb2bbbf [id=40c29df99590408cbd5c583d4fc741eb]). Expected version 2 got 0
I20251025 07:52:42.945108 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 509c94f2a422493ea4390ecee576db72 (table default.Table_3f081b7bb4e1420c8121ea4115439f3d [id=696a1285fbfe4038a8b1b38708c8ea49]). Expected version 1 got 0
I20251025 07:52:42.945504 2146 ts_tablet_manager.cc:1950] T c59f6e6eb9594a8e83f109a09b034045 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:42.946179 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 5dab439bc50d4e869548154e1d222ad6 (table default.Table_bd82d871f84943f4baa967f5c0ec74a3 [id=45325c188eac4d34bb016d01de091f87]). Expected version 2 got 0
I20251025 07:52:42.946403 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9bfd7c0c3ad4e5b98b6708793d61d41" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:42.946316 1166 ts_tablet_manager.cc:1434] T b2044ca206634eb285197af4ba2073e7 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20251025 07:52:42.947182 2484 leader_election.cc:290] T d9bfd7c0c3ad4e5b98b6708793d61d41 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:42.947184 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_87d38832f85e4965bfab1a077990114f [id=ad812862698d421184ebf501ce79b7be]). Expected version 2 got 0
I20251025 07:52:42.947801 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet f70091024c034c118fb86578db19a31c (table default.Table_332b308220f64c6dbb5f29711d8463b3 [id=e041946c093b4c789ce175c736cc230b]). Expected version 1 got 0
I20251025 07:52:42.948938 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]). Expected version 3 got 0
I20251025 07:52:42.949244 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]). Expected version 1 got 0
I20251025 07:52:42.949478 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c]). Expected version 1 got 0
I20251025 07:52:42.949477 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 109) took 4178 ms. Trace:
I20251025 07:52:42.949576 2169 rpcz_store.cc:276] 1025 07:52:38.772082 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.772200 (+ 118us) service_pool.cc:225] Handling call
1025 07:52:42.811564 (+4039364us) ts_tablet_manager.cc:1151] Deleting tablet 35ba9d23262248cda6fdf03c51be11a4
1025 07:52:42.811574 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:42.819316 (+ 7742us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.904285 (+ 84969us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.949455 (+ 45170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1167,"tablet-delete.queue_time_us":4039175}
I20251025 07:52:42.949941 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 755f0d6c87044e20a8020f8410940123 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 4 got 0
I20251025 07:52:42.950070 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 35ba9d23262248cda6fdf03c51be11a4 (table default.Table_072e0092adb64ef0ac6834dc18688ccf [id=1d4cbe4a4f38445297ceeebb8ffdaa8e]) successfully deleted
I20251025 07:52:42.901086 1167 tablet_bootstrap.cc:492] T 627a9cfd398346f2aa34c8932a11b5a8 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:42.950284 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 22f7db660616457b83b24e947a0b0bae (table default.Table_4a86191163f2408f9bcc795cd8d1bbf6 [id=bb316a7e000748ca902a9bfb494802d7]). Expected version 1 got 0
I20251025 07:52:42.935148 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c87ab6a3d08f43e0b2fe58a6d7b11588" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:42.945572 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9bfd7c0c3ad4e5b98b6708793d61d41" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:42.950484 1167 ts_tablet_manager.cc:1403] T 627a9cfd398346f2aa34c8932a11b5a8 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.104s user 0.003s sys 0.001s
I20251025 07:52:42.950567 773 raft_consensus.cc:2393] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:42.950659 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236]). Expected version 1 got 0
I20251025 07:52:42.950978 982 leader_election.cc:304] T d9bfd7c0c3ad4e5b98b6708793d61d41 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:42.951068 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84]). Expected version 1 got 0
I20251025 07:52:42.951337 1167 raft_consensus.cc:359] T 627a9cfd398346f2aa34c8932a11b5a8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.951434 774 raft_consensus.cc:2468] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:42.951504 1167 raft_consensus.cc:385] T 627a9cfd398346f2aa34c8932a11b5a8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.951561 1167 raft_consensus.cc:740] T 627a9cfd398346f2aa34c8932a11b5a8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.951843 847 leader_election.cc:304] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.951938 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 0325d462441349ef81841f1b0cab2a20 (table default.Table_750ecb905839434fa9a1aa72f87394be [id=9ce3ba77c56a4ef09cb3b40ea0e488f6]). Expected version 1 got 0
I20251025 07:52:42.951884 1167 consensus_queue.cc:260] T 627a9cfd398346f2aa34c8932a11b5a8 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.952255 2314 raft_consensus.cc:2804] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:42.952322 2314 raft_consensus.cc:493] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:42.952374 2314 raft_consensus.cc:3060] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.952543 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5]). Expected version 1 got 0
I20251025 07:52:42.952867 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525]). Expected version 1 got 0
I20251025 07:52:42.953049 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_e90d7f5ff13b40878e5b625c97792c7d [id=fc8180c09d7b4e74b2e161f7acb8fe0e]). Expected version 1 got 0
I20251025 07:52:42.953687 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet b56209e4934e4407a8b4518ca4af0d46 (table default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de]). Expected version 1 got 0
I20251025 07:52:42.953933 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 830fdbc6812e416690d0db42a73af503 (table default.Table_92824b3f2a774564b3aea04aaa25d204 [id=fb21a45b86ab446e8c3dae4512a9598f]). Expected version 1 got 0
I20251025 07:52:42.954833 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3]). Expected version 2 got 0
I20251025 07:52:42.955081 2314 raft_consensus.cc:515] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.956305 1167 ts_tablet_manager.cc:1434] T 627a9cfd398346f2aa34c8932a11b5a8 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251025 07:52:42.956305 2314 leader_election.cc:290] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:42.956527 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c87ab6a3d08f43e0b2fe58a6d7b11588" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:42.956533 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c87ab6a3d08f43e0b2fe58a6d7b11588" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:42.956743 774 raft_consensus.cc:3060] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:42.957082 850 leader_election.cc:343] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.957386 2166 log.cc:1199] T 1912e59548874e0e8e989050ec0064c5 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/1912e59548874e0e8e989050ec0064c5
I20251025 07:52:42.960393 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 36262ab8c6964dd684c01b4db6219a4f (table default.Table_76158f8cf82e4cdb9a90580e46aaea48 [id=f249f30497bc450c887d659721469de8]). Expected version 1 got 0
I20251025 07:52:42.960855 2166 ts_tablet_manager.cc:1950] T 1912e59548874e0e8e989050ec0064c5 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:42.960824 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3]). Expected version 1 got 0
I20251025 07:52:42.961220 2484 raft_consensus.cc:3060] T d9bfd7c0c3ad4e5b98b6708793d61d41 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:42.959677 774 raft_consensus.cc:2468] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:42.961724 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 292fe04055954afd8c08ac2636b68191 (table default.Table_9fa69b10b5e842639fc0b0a58cd9b6a5 [id=a253ce0253bd4da4814741c482193d8f]). Expected version 1 got 0
I20251025 07:52:42.961964 847 leader_election.cc:304] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.963392 2314 raft_consensus.cc:2804] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:42.963567 2314 raft_consensus.cc:697] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:42.963929 2314 consensus_queue.cc:237] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.964879 2484 raft_consensus.cc:2749] T d9bfd7c0c3ad4e5b98b6708793d61d41 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:42.965687 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 120) took 4446 ms. Trace:
I20251025 07:52:42.965799 2146 rpcz_store.cc:276] 1025 07:52:38.519051 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.519109 (+ 58us) service_pool.cc:225] Handling call
1025 07:52:42.821174 (+4302065us) ts_tablet_manager.cc:1151] Deleting tablet c59f6e6eb9594a8e83f109a09b034045
1025 07:52:42.821440 (+ 266us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:42.867982 (+ 46542us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.939250 (+ 71268us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.965667 (+ 26417us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":452,"spinlock_wait_cycles":549120,"tablet-delete.queue_time_us":4296500}
I20251025 07:52:42.966462 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet c59f6e6eb9594a8e83f109a09b034045 (table default.Table_500d949c884c4644b70943543bd3e0c5 [id=71ac1bb83b444af2aede5644bf5619a3]) successfully deleted
I20251025 07:52:42.967075 1166 tablet_bootstrap.cc:492] T 34315f64b89148efbf1f20a43d5c5efb P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:42.967883 1167 tablet_bootstrap.cc:492] T 143aa3fce9ba4850b000a94054e5dc3a P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:42.969475 1167 tablet_bootstrap.cc:654] T 143aa3fce9ba4850b000a94054e5dc3a P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:42.975817 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 93) took 4358 ms. Trace:
I20251025 07:52:42.975921 2166 rpcz_store.cc:276] 1025 07:52:38.617310 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.617367 (+ 57us) service_pool.cc:225] Handling call
1025 07:52:42.843057 (+4225690us) ts_tablet_manager.cc:1151] Deleting tablet 1912e59548874e0e8e989050ec0064c5
1025 07:52:42.843066 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:42.867240 (+ 24174us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.899814 (+ 32574us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:42.975794 (+ 75980us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4225541}
I20251025 07:52:42.976281 2166 tablet_replica.cc:333] T 887e7c7a5d0e4aaf88f26246306df19c P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:42.976491 2166 raft_consensus.cc:2243] T 887e7c7a5d0e4aaf88f26246306df19c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:42.976648 2166 raft_consensus.cc:2272] T 887e7c7a5d0e4aaf88f26246306df19c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:42.977183 2166 ts_tablet_manager.cc:1916] T 887e7c7a5d0e4aaf88f26246306df19c P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:42.978319 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 1912e59548874e0e8e989050ec0064c5 (table default.Table_8d39f75f88854f528f5cac09c3b0423a [id=74ec1fa4786e40109252885ea225ed66]) successfully deleted
I20251025 07:52:42.978526 2508 raft_consensus.cc:493] T bdaeedbcf35a4aad97f9ef192991978a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:42.978677 2508 raft_consensus.cc:515] T bdaeedbcf35a4aad97f9ef192991978a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.979364 2508 leader_election.cc:290] T bdaeedbcf35a4aad97f9ef192991978a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:42.979694 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 110) took 4207 ms. Trace:
I20251025 07:52:42.979784 2169 rpcz_store.cc:276] 1025 07:52:38.773339 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.773445 (+ 106us) service_pool.cc:225] Handling call
1025 07:52:42.979572 (+4206127us) ts_tablet_manager.cc:1151] Deleting tablet 2209089667dd4937abd96615f6929449
1025 07:52:42.979585 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:42.979671 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4205902}
W20251025 07:52:42.980243 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 2209089667dd4937abd96615f6929449 because the tablet was not found. No further retry: Not found: Tablet not found: 2209089667dd4937abd96615f6929449
I20251025 07:52:42.980765 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bdaeedbcf35a4aad97f9ef192991978a" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:42.981145 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bdaeedbcf35a4aad97f9ef192991978a" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:42.981321 710 leader_election.cc:343] T bdaeedbcf35a4aad97f9ef192991978a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.981608 2169 tablet_replica.cc:333] T 74e2d78c024849b0af5c0bde1f48c36f P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
W20251025 07:52:42.981717 712 leader_election.cc:343] T bdaeedbcf35a4aad97f9ef192991978a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:42.981866 2169 raft_consensus.cc:2243] T 74e2d78c024849b0af5c0bde1f48c36f P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:42.981827 712 leader_election.cc:304] T bdaeedbcf35a4aad97f9ef192991978a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:42.982236 2169 raft_consensus.cc:2272] T 74e2d78c024849b0af5c0bde1f48c36f P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:42.982332 2508 raft_consensus.cc:2749] T bdaeedbcf35a4aad97f9ef192991978a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:42.982972 2169 ts_tablet_manager.cc:1916] T 74e2d78c024849b0af5c0bde1f48c36f P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:42.985574 1167 tablet_bootstrap.cc:492] T 143aa3fce9ba4850b000a94054e5dc3a P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:42.985765 1167 ts_tablet_manager.cc:1403] T 143aa3fce9ba4850b000a94054e5dc3a P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.006s
I20251025 07:52:42.986680 1167 raft_consensus.cc:359] T 143aa3fce9ba4850b000a94054e5dc3a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.986905 1167 raft_consensus.cc:385] T 143aa3fce9ba4850b000a94054e5dc3a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:42.987011 1167 raft_consensus.cc:740] T 143aa3fce9ba4850b000a94054e5dc3a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:42.987334 1167 consensus_queue.cc:260] T 143aa3fce9ba4850b000a94054e5dc3a P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:42.994354 2106 consensus_queue.cc:1048] T a4e4b99be0434baaa75d988281797846 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:42.995504 1166 tablet_bootstrap.cc:654] T 34315f64b89148efbf1f20a43d5c5efb P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:42.998260 2146 tablet_replica.cc:333] T f4578ce69a0d4322944e5ce7c8a59831 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:42.998485 2146 raft_consensus.cc:2243] T f4578ce69a0d4322944e5ce7c8a59831 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:42.998641 2146 raft_consensus.cc:2272] T f4578ce69a0d4322944e5ce7c8a59831 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:42.999172 2146 ts_tablet_manager.cc:1916] T f4578ce69a0d4322944e5ce7c8a59831 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:43.014528 1167 ts_tablet_manager.cc:1434] T 143aa3fce9ba4850b000a94054e5dc3a P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20251025 07:52:43.014755 1096 heartbeater.cc:344] Connected to a master server at 127.0.104.61:42203
I20251025 07:52:43.014907 1167 tablet_bootstrap.cc:492] T 5b6383c99e9e4c90a374da9ccd830a6f P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:43.016553 1167 tablet_bootstrap.cc:654] T 5b6383c99e9e4c90a374da9ccd830a6f P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:43.019601 1169 tablet_bootstrap.cc:492] T 0f5f6ca6c3cd4d369690e425387a4344 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:43.019834 1169 ts_tablet_manager.cc:1403] T 0f5f6ca6c3cd4d369690e425387a4344 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.079s user 0.008s sys 0.002s
I20251025 07:52:43.020597 1169 raft_consensus.cc:359] T 0f5f6ca6c3cd4d369690e425387a4344 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.020991 1169 raft_consensus.cc:385] T 0f5f6ca6c3cd4d369690e425387a4344 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:43.021196 1169 raft_consensus.cc:740] T 0f5f6ca6c3cd4d369690e425387a4344 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:43.021456 1169 consensus_queue.cc:260] T 0f5f6ca6c3cd4d369690e425387a4344 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.021970 1169 ts_tablet_manager.cc:1434] T 0f5f6ca6c3cd4d369690e425387a4344 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:43.022321 1169 tablet_bootstrap.cc:492] T 04d27d54fdb7483b869408af65ff85c9 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:43.023957 1169 tablet_bootstrap.cc:654] T 04d27d54fdb7483b869408af65ff85c9 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:43.026969 1166 tablet_bootstrap.cc:492] T 34315f64b89148efbf1f20a43d5c5efb P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:43.027171 1166 ts_tablet_manager.cc:1403] T 34315f64b89148efbf1f20a43d5c5efb P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.079s user 0.002s sys 0.001s
I20251025 07:52:43.027976 1166 raft_consensus.cc:359] T 34315f64b89148efbf1f20a43d5c5efb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:43.028196 1166 raft_consensus.cc:385] T 34315f64b89148efbf1f20a43d5c5efb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:43.028275 1166 raft_consensus.cc:740] T 34315f64b89148efbf1f20a43d5c5efb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:43.028631 1166 consensus_queue.cc:260] T 34315f64b89148efbf1f20a43d5c5efb P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:43.029196 1166 ts_tablet_manager.cc:1434] T 34315f64b89148efbf1f20a43d5c5efb P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:43.029338 2166 ts_tablet_manager.cc:1929] T 887e7c7a5d0e4aaf88f26246306df19c P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:43.029454 2166 log.cc:1199] T 887e7c7a5d0e4aaf88f26246306df19c P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/887e7c7a5d0e4aaf88f26246306df19c
I20251025 07:52:43.029805 2166 ts_tablet_manager.cc:1950] T 887e7c7a5d0e4aaf88f26246306df19c P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:43.034907 1166 tablet_bootstrap.cc:492] T 050a5d3b5f934dd19bf2e221f2fe0780 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:43.036310 1166 tablet_bootstrap.cc:654] T 050a5d3b5f934dd19bf2e221f2fe0780 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:43.037056 1046 raft_consensus.cc:3060] T a4e4b99be0434baaa75d988281797846 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:43.039669 2484 raft_consensus.cc:493] T 0f5f6ca6c3cd4d369690e425387a4344 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:43.039852 2484 raft_consensus.cc:515] T 0f5f6ca6c3cd4d369690e425387a4344 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.040905 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f5f6ca6c3cd4d369690e425387a4344" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:43.041056 2471 consensus_queue.cc:1048] T a4e4b99be0434baaa75d988281797846 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:43.041460 2484 leader_election.cc:290] T 0f5f6ca6c3cd4d369690e425387a4344 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:43.042227 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f5f6ca6c3cd4d369690e425387a4344" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:43.042425 773 raft_consensus.cc:2393] T 0f5f6ca6c3cd4d369690e425387a4344 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:43.042863 982 leader_election.cc:304] T 0f5f6ca6c3cd4d369690e425387a4344 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:43.043303 2050 raft_consensus.cc:3060] T 0f5f6ca6c3cd4d369690e425387a4344 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:43.045961 2050 raft_consensus.cc:2749] T 0f5f6ca6c3cd4d369690e425387a4344 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:43.047850 1166 tablet_bootstrap.cc:492] T 050a5d3b5f934dd19bf2e221f2fe0780 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:43.048050 1166 ts_tablet_manager.cc:1403] T 050a5d3b5f934dd19bf2e221f2fe0780 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.002s
I20251025 07:52:43.048763 1166 raft_consensus.cc:359] T 050a5d3b5f934dd19bf2e221f2fe0780 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.048947 1166 raft_consensus.cc:385] T 050a5d3b5f934dd19bf2e221f2fe0780 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:43.049021 1166 raft_consensus.cc:740] T 050a5d3b5f934dd19bf2e221f2fe0780 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:43.049264 1166 consensus_queue.cc:260] T 050a5d3b5f934dd19bf2e221f2fe0780 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.049789 1166 ts_tablet_manager.cc:1434] T 050a5d3b5f934dd19bf2e221f2fe0780 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:43.050206 1166 tablet_bootstrap.cc:492] T 09898735796541b0a8f3be22cdb6cc30 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:43.051695 1166 tablet_bootstrap.cc:654] T 09898735796541b0a8f3be22cdb6cc30 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:43.054927 1167 tablet_bootstrap.cc:492] T 5b6383c99e9e4c90a374da9ccd830a6f P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:43.055074 1167 ts_tablet_manager.cc:1403] T 5b6383c99e9e4c90a374da9ccd830a6f P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.002s
I20251025 07:52:43.055747 1167 raft_consensus.cc:359] T 5b6383c99e9e4c90a374da9ccd830a6f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.055946 1167 raft_consensus.cc:385] T 5b6383c99e9e4c90a374da9ccd830a6f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:43.056018 1167 raft_consensus.cc:740] T 5b6383c99e9e4c90a374da9ccd830a6f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:43.056212 1167 consensus_queue.cc:260] T 5b6383c99e9e4c90a374da9ccd830a6f P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.056886 1167 ts_tablet_manager.cc:1434] T 5b6383c99e9e4c90a374da9ccd830a6f P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:43.056965 2169 ts_tablet_manager.cc:1929] T 74e2d78c024849b0af5c0bde1f48c36f P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:43.057117 2169 log.cc:1199] T 74e2d78c024849b0af5c0bde1f48c36f P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/74e2d78c024849b0af5c0bde1f48c36f
I20251025 07:52:43.057154 1167 tablet_bootstrap.cc:492] T 4a1ec8c2ed7f41abb588f592f62097d1 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:43.057469 2169 ts_tablet_manager.cc:1950] T 74e2d78c024849b0af5c0bde1f48c36f P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:43.030102 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 94) took 4409 ms. Trace:
I20251025 07:52:43.057603 2166 rpcz_store.cc:276] 1025 07:52:38.620365 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.620427 (+ 62us) service_pool.cc:225] Handling call
1025 07:52:42.976239 (+4355812us) ts_tablet_manager.cc:1151] Deleting tablet 887e7c7a5d0e4aaf88f26246306df19c
1025 07:52:42.976248 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:43.026450 (+ 50202us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:43.029255 (+ 2805us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:43.030087 (+ 832us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4355619}
I20251025 07:52:43.058528 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 111) took 4284 ms. Trace:
I20251025 07:52:43.058912 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 887e7c7a5d0e4aaf88f26246306df19c (table default.Table_2c1cdea0079240b889637e5d8d13da95 [id=8e07d0ad2db84d1b8a3a3a3efa2d7f69]) successfully deleted
I20251025 07:52:43.059049 1167 tablet_bootstrap.cc:654] T 4a1ec8c2ed7f41abb588f592f62097d1 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:43.076011 2146 ts_tablet_manager.cc:1929] T f4578ce69a0d4322944e5ce7c8a59831 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:43.076264 2146 log.cc:1199] T f4578ce69a0d4322944e5ce7c8a59831 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/f4578ce69a0d4322944e5ce7c8a59831
I20251025 07:52:43.076807 2146 ts_tablet_manager.cc:1950] T f4578ce69a0d4322944e5ce7c8a59831 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:43.079288 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 95) took 4458 ms. Trace:
I20251025 07:52:43.079427 2166 rpcz_store.cc:276] 1025 07:52:38.620802 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.620844 (+ 42us) service_pool.cc:225] Handling call
1025 07:52:43.059246 (+4438402us) ts_tablet_manager.cc:1151] Deleting tablet ee1c8ea694284d80a30f9c05be4cbf49
1025 07:52:43.059255 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:43.079266 (+ 20011us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4438234}
W20251025 07:52:43.079998 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet ee1c8ea694284d80a30f9c05be4cbf49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee1c8ea694284d80a30f9c05be4cbf49 already in progress: creating tablet
I20251025 07:52:43.081218 2508 raft_consensus.cc:493] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:43.081197 2169 rpcz_store.cc:276] 1025 07:52:38.774319 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.774623 (+ 304us) service_pool.cc:225] Handling call
1025 07:52:42.981557 (+4206934us) ts_tablet_manager.cc:1151] Deleting tablet 74e2d78c024849b0af5c0bde1f48c36f
1025 07:52:42.981569 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:43.035332 (+ 53763us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:43.056848 (+ 21516us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:43.058508 (+ 1660us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4206726}
I20251025 07:52:43.081400 2508 raft_consensus.cc:515] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:43.081575 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 74e2d78c024849b0af5c0bde1f48c36f (table default.Table_434ef1ade0a14af987864563993afae8 [id=3a3af35e11944a6f97fc0144cb9d08d4]) successfully deleted
I20251025 07:52:43.082114 2508 leader_election.cc:290] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:43.083705 712 leader_election.cc:343] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 5cba9c05dd424cc788febc614a34819a
I20251025 07:52:43.084384 2169 tablet_replica.cc:333] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:43.084606 2169 raft_consensus.cc:2243] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:43.087985 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 96) took 4466 ms. Trace:
I20251025 07:52:43.088106 2166 rpcz_store.cc:276] 1025 07:52:38.621535 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.621584 (+ 49us) service_pool.cc:225] Handling call
1025 07:52:43.080328 (+4458744us) ts_tablet_manager.cc:1151] Deleting tablet 570201e7bf05406a97c81dde071d894b
1025 07:52:43.087864 (+ 7536us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:43.087966 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4458543}
W20251025 07:52:43.088563 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 570201e7bf05406a97c81dde071d894b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 570201e7bf05406a97c81dde071d894b already in progress: creating tablet
I20251025 07:52:43.088791 2166 tablet_replica.cc:333] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:43.089099 2166 raft_consensus.cc:2243] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:43.090178 1166 tablet_bootstrap.cc:492] T 09898735796541b0a8f3be22cdb6cc30 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
W20251025 07:52:43.089128 710 leader_election.cc:343] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 5cba9c05dd424cc788febc614a34819a
I20251025 07:52:43.090408 2169 raft_consensus.cc:2272] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:43.090407 710 leader_election.cc:304] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:43.089515 1169 tablet_bootstrap.cc:492] T 04d27d54fdb7483b869408af65ff85c9 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:43.090248 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 119) took 4571 ms. Trace:
I20251025 07:52:43.090967 1169 ts_tablet_manager.cc:1403] T 04d27d54fdb7483b869408af65ff85c9 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.069s user 0.004s sys 0.000s
I20251025 07:52:43.091091 2508 raft_consensus.cc:2706] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Received pre-election callback for term 1 while not running: Illegal state: RaftConsensus is not running: State = Stopping
I20251025 07:52:43.091188 2166 raft_consensus.cc:2272] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:43.091758 1169 raft_consensus.cc:359] T 04d27d54fdb7483b869408af65ff85c9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:43.091969 1169 raft_consensus.cc:385] T 04d27d54fdb7483b869408af65ff85c9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:43.092067 1169 raft_consensus.cc:740] T 04d27d54fdb7483b869408af65ff85c9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:43.091325 2146 rpcz_store.cc:276] 1025 07:52:38.518954 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.519183 (+ 229us) service_pool.cc:225] Handling call
1025 07:52:42.966995 (+4447812us) ts_tablet_manager.cc:1151] Deleting tablet f4578ce69a0d4322944e5ce7c8a59831
1025 07:52:42.998201 (+ 31206us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:43.021525 (+ 23324us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:43.075832 (+ 54307us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:43.090232 (+ 14400us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":451,"tablet-delete.queue_time_us":4442504}
I20251025 07:52:43.092314 1169 consensus_queue.cc:260] T 04d27d54fdb7483b869408af65ff85c9 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:43.092895 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet f4578ce69a0d4322944e5ce7c8a59831 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]) successfully deleted
I20251025 07:52:43.092929 1169 ts_tablet_manager.cc:1434] T 04d27d54fdb7483b869408af65ff85c9 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:43.093192 1169 tablet_bootstrap.cc:492] T 1912e59548874e0e8e989050ec0064c5 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:43.094004 2146 tablet_replica.cc:333] T 71432a6848944a0da06ef9b81f160dc4 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:43.090349 1166 ts_tablet_manager.cc:1403] T 09898735796541b0a8f3be22cdb6cc30 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.040s user 0.006s sys 0.005s
I20251025 07:52:43.094763 2169 ts_tablet_manager.cc:1916] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:43.094868 2166 ts_tablet_manager.cc:1916] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:43.094966 1166 raft_consensus.cc:359] T 09898735796541b0a8f3be22cdb6cc30 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.095122 1166 raft_consensus.cc:385] T 09898735796541b0a8f3be22cdb6cc30 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:43.095382 1166 raft_consensus.cc:740] T 09898735796541b0a8f3be22cdb6cc30 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:43.095604 1166 consensus_queue.cc:260] T 09898735796541b0a8f3be22cdb6cc30 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.096063 1166 ts_tablet_manager.cc:1434] T 09898735796541b0a8f3be22cdb6cc30 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:43.097730 1167 tablet_bootstrap.cc:492] T 4a1ec8c2ed7f41abb588f592f62097d1 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:43.097853 1167 ts_tablet_manager.cc:1403] T 4a1ec8c2ed7f41abb588f592f62097d1 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.000s
I20251025 07:52:43.097554 2146 raft_consensus.cc:2243] T 71432a6848944a0da06ef9b81f160dc4 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:43.098657 1167 raft_consensus.cc:359] T 4a1ec8c2ed7f41abb588f592f62097d1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.098812 1167 raft_consensus.cc:385] T 4a1ec8c2ed7f41abb588f592f62097d1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:43.098863 1167 raft_consensus.cc:740] T 4a1ec8c2ed7f41abb588f592f62097d1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:43.099074 1167 consensus_queue.cc:260] T 4a1ec8c2ed7f41abb588f592f62097d1 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.098917 2146 raft_consensus.cc:2272] T 71432a6848944a0da06ef9b81f160dc4 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:43.100147 1167 ts_tablet_manager.cc:1434] T 4a1ec8c2ed7f41abb588f592f62097d1 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:43.101810 2314 consensus_queue.cc:1048] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:43.102222 2471 raft_consensus.cc:493] T 34315f64b89148efbf1f20a43d5c5efb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:43.102340 2471 raft_consensus.cc:515] T 34315f64b89148efbf1f20a43d5c5efb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:43.102906 2471 leader_election.cc:290] T 34315f64b89148efbf1f20a43d5c5efb P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
W20251025 07:52:43.103711 847 leader_election.cc:343] T 34315f64b89148efbf1f20a43d5c5efb P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: 34315f64b89148efbf1f20a43d5c5efb
I20251025 07:52:43.104749 2508 raft_consensus.cc:493] T ce1ebb1ec14b4801930a72765bca1a7d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:43.104882 2508 raft_consensus.cc:515] T ce1ebb1ec14b4801930a72765bca1a7d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.105492 2508 leader_election.cc:290] T ce1ebb1ec14b4801930a72765bca1a7d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:43.106052 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce1ebb1ec14b4801930a72765bca1a7d" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:43.106518 710 leader_election.cc:343] T ce1ebb1ec14b4801930a72765bca1a7d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:43.108834 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f]). Expected version 1 got 0
I20251025 07:52:43.109213 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 4c155aa7fa5b4d68bbab1918b8432372 (table default.Table_6c5a24747ba14352b0285a41ec2985a4 [id=28bbaad474164a8c8c97a76bb8081791]). Expected version 2 got 0
I20251025 07:52:43.109598 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce1ebb1ec14b4801930a72765bca1a7d" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:43.109598 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34315f64b89148efbf1f20a43d5c5efb" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:43.110093 1045 raft_consensus.cc:2468] T 34315f64b89148efbf1f20a43d5c5efb P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:43.110066 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7]). Expected version 1 got 0
I20251025 07:52:43.110400 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 04d27d54fdb7483b869408af65ff85c9 (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]). Expected version 2 got 0
I20251025 07:52:43.110514 850 leader_election.cc:304] T 34315f64b89148efbf1f20a43d5c5efb P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b
W20251025 07:52:43.110647 712 leader_election.cc:343] T ce1ebb1ec14b4801930a72765bca1a7d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:43.110746 712 leader_election.cc:304] T ce1ebb1ec14b4801930a72765bca1a7d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:43.110703 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet cb829649f73d4d3f8862d268c953bd86 (table default.Table_ba4b8cc7b930482690b7e192e0edff12 [id=7944225d3ce7439b881bfb9289d9f040]). Expected version 1 got 0
I20251025 07:52:43.110989 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 872cb7512d104827b1f1f186d33a4a44 (table default.Table_d5c65dce8b1b4214ba519d964012bfc0 [id=d4be274b36a845a99a4b96b8846248fd]). Expected version 1 got 0
I20251025 07:52:43.111218 2420 raft_consensus.cc:2749] T ce1ebb1ec14b4801930a72765bca1a7d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:43.111795 2471 raft_consensus.cc:2804] T 34315f64b89148efbf1f20a43d5c5efb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:43.111876 2471 raft_consensus.cc:493] T 34315f64b89148efbf1f20a43d5c5efb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:43.111936 2471 raft_consensus.cc:3060] T 34315f64b89148efbf1f20a43d5c5efb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:43.111994 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet aa9d9ab8073945d28d52fb02213f201a (table default.Table_4791672658f043f6b97b9320a531ebb5 [id=dfd20cf9c3e541c4abe23d59e0aa7f2a]). Expected version 1 got 0
I20251025 07:52:43.112345 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]). Expected version 1 got 0
I20251025 07:52:43.112541 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet ef9a2c9f2ccf432ab835396020872208 (table default.Table_2b9b772d4c234ac286bef3658482e490 [id=409fab60f9d744798e6f92b868f0f7c2]). Expected version 1 got 0
I20251025 07:52:43.112892 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]). Expected version 1 got 0
I20251025 07:52:43.113009 2314 raft_consensus.cc:493] T 050a5d3b5f934dd19bf2e221f2fe0780 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:43.113140 2314 raft_consensus.cc:515] T 050a5d3b5f934dd19bf2e221f2fe0780 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.113224 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_a6c869949b744bc98efb735b0e73e7aa [id=bf77b5ff6f1048dd996d948437f9ff5e]). Expected version 2 got 0
I20251025 07:52:43.113641 1166 tablet_bootstrap.cc:492] T ab42909afc524f2fa3c1041495d13a59 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:43.113692 2314 leader_election.cc:290] T 050a5d3b5f934dd19bf2e221f2fe0780 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:43.113677 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet bc2d16b33ca8426abbd7bd38339c9f1d (table default.Table_4ea985df4c5b415c9570c64506a98cc4 [id=98d2b7797e33485e8872054aa2eb8ec9]). Expected version 1 got 0
I20251025 07:52:43.113930 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e]). Expected version 1 got 0
I20251025 07:52:43.114174 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet ab42909afc524f2fa3c1041495d13a59 (table default.Table_7eb8424d5192485da1a2d67bfdb2bbbf [id=40c29df99590408cbd5c583d4fc741eb]). Expected version 2 got 0
I20251025 07:52:43.114442 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "050a5d3b5f934dd19bf2e221f2fe0780" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:43.114617 774 raft_consensus.cc:2468] T 050a5d3b5f934dd19bf2e221f2fe0780 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:43.115061 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 5dab439bc50d4e869548154e1d222ad6 (table default.Table_bd82d871f84943f4baa967f5c0ec74a3 [id=45325c188eac4d34bb016d01de091f87]). Expected version 2 got 0
I20251025 07:52:43.115384 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_87d38832f85e4965bfab1a077990114f [id=ad812862698d421184ebf501ce79b7be]). Expected version 2 got 0
I20251025 07:52:43.115720 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet f70091024c034c118fb86578db19a31c (table default.Table_332b308220f64c6dbb5f29711d8463b3 [id=e041946c093b4c789ce175c736cc230b]). Expected version 1 got 0
I20251025 07:52:43.116027 1169 tablet_bootstrap.cc:654] T 1912e59548874e0e8e989050ec0064c5 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:43.116605 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]). Expected version 3 got 0
I20251025 07:52:43.116891 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]). Expected version 1 got 0
I20251025 07:52:43.117117 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c]). Expected version 1 got 0
I20251025 07:52:43.117225 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "050a5d3b5f934dd19bf2e221f2fe0780" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:43.117457 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 755f0d6c87044e20a8020f8410940123 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 4 got 0
I20251025 07:52:43.117646 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 22f7db660616457b83b24e947a0b0bae (table default.Table_4a86191163f2408f9bcc795cd8d1bbf6 [id=bb316a7e000748ca902a9bfb494802d7]). Expected version 1 got 0
I20251025 07:52:43.117861 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236]). Expected version 1 got 0
I20251025 07:52:43.117853 847 leader_election.cc:304] T 050a5d3b5f934dd19bf2e221f2fe0780 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:43.118513 2443 raft_consensus.cc:2804] T 050a5d3b5f934dd19bf2e221f2fe0780 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:43.118597 2443 raft_consensus.cc:493] T 050a5d3b5f934dd19bf2e221f2fe0780 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:43.118543 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84]). Expected version 1 got 0
I20251025 07:52:43.118659 2443 raft_consensus.cc:3060] T 050a5d3b5f934dd19bf2e221f2fe0780 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:43.119186 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 0325d462441349ef81841f1b0cab2a20 (table default.Table_750ecb905839434fa9a1aa72f87394be [id=9ce3ba77c56a4ef09cb3b40ea0e488f6]). Expected version 1 got 0
I20251025 07:52:43.119565 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5]). Expected version 1 got 0
I20251025 07:52:43.119886 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525]). Expected version 1 got 0
I20251025 07:52:43.120093 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_e90d7f5ff13b40878e5b625c97792c7d [id=fc8180c09d7b4e74b2e161f7acb8fe0e]). Expected version 1 got 0
I20251025 07:52:43.120958 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet b56209e4934e4407a8b4518ca4af0d46 (table default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de]). Expected version 1 got 0
I20251025 07:52:43.121235 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 830fdbc6812e416690d0db42a73af503 (table default.Table_92824b3f2a774564b3aea04aaa25d204 [id=fb21a45b86ab446e8c3dae4512a9598f]). Expected version 1 got 0
I20251025 07:52:43.121282 2443 raft_consensus.cc:515] T 050a5d3b5f934dd19bf2e221f2fe0780 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.121594 1166 tablet_bootstrap.cc:654] T ab42909afc524f2fa3c1041495d13a59 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:43.121829 2443 leader_election.cc:290] T 050a5d3b5f934dd19bf2e221f2fe0780 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:43.122546 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3]). Expected version 2 got 0
I20251025 07:52:43.122949 1167 tablet_bootstrap.cc:492] T 36c414bc700341188b7d85c14173c3e0 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
W20251025 07:52:43.124545 850 leader_election.cc:343] T 050a5d3b5f934dd19bf2e221f2fe0780 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:43.124603 1167 tablet_bootstrap.cc:654] T 36c414bc700341188b7d85c14173c3e0 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:43.125527 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "050a5d3b5f934dd19bf2e221f2fe0780" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:43.125892 2169 ts_tablet_manager.cc:1929] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:43.126107 2169 log.cc:1199] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/5fca357d7e24494fb6af4bd73d98833c
I20251025 07:52:43.126665 2169 ts_tablet_manager.cc:1950] T 5fca357d7e24494fb6af4bd73d98833c P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:43.128533 2314 consensus_queue.cc:1048] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:43.129148 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "050a5d3b5f934dd19bf2e221f2fe0780" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:43.129307 774 raft_consensus.cc:3060] T 050a5d3b5f934dd19bf2e221f2fe0780 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:43.129578 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 112) took 4354 ms. Trace:
I20251025 07:52:43.129978 2169 rpcz_store.cc:276] 1025 07:52:38.775325 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.775704 (+ 379us) service_pool.cc:225] Handling call
1025 07:52:43.084326 (+4308622us) ts_tablet_manager.cc:1151] Deleting tablet 5fca357d7e24494fb6af4bd73d98833c
1025 07:52:43.084339 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:43.120734 (+ 36395us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:43.125700 (+ 4966us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:43.129543 (+ 3843us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4305567}
I20251025 07:52:43.130756 2166 ts_tablet_manager.cc:1929] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:43.130741 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 113) took 4353 ms. Trace:
I20251025 07:52:43.131268 2169 rpcz_store.cc:276] 1025 07:52:38.778164 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.778426 (+ 262us) service_pool.cc:225] Handling call
1025 07:52:43.130612 (+4352186us) ts_tablet_manager.cc:1151] Deleting tablet c4610b0c62e741eca28c21f63cc3de81
1025 07:52:43.130625 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:43.130701 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4352026}
I20251025 07:52:43.132218 1169 tablet_bootstrap.cc:492] T 1912e59548874e0e8e989050ec0064c5 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
W20251025 07:52:43.132413 850 leader_election.cc:343] T 050a5d3b5f934dd19bf2e221f2fe0780 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:43.132290 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 5fca357d7e24494fb6af4bd73d98833c (table default.Table_400f06d166054eeb97890ad56453afb2 [id=5f20de141a6f452a82ed736f41c29d9b]) successfully deleted
I20251025 07:52:43.132715 1169 ts_tablet_manager.cc:1403] T 1912e59548874e0e8e989050ec0064c5 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.040s user 0.000s sys 0.004s
I20251025 07:52:43.133605 1169 raft_consensus.cc:359] T 1912e59548874e0e8e989050ec0064c5 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:43.133874 1169 raft_consensus.cc:385] T 1912e59548874e0e8e989050ec0064c5 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:43.133898 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 36262ab8c6964dd684c01b4db6219a4f (table default.Table_76158f8cf82e4cdb9a90580e46aaea48 [id=f249f30497bc450c887d659721469de8]). Expected version 1 got 0
I20251025 07:52:43.134045 1169 raft_consensus.cc:740] T 1912e59548874e0e8e989050ec0064c5 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:43.134344 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3]). Expected version 1 got 0
I20251025 07:52:43.134495 1169 consensus_queue.cc:260] T 1912e59548874e0e8e989050ec0064c5 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:43.134940 454 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 292fe04055954afd8c08ac2636b68191 (table default.Table_9fa69b10b5e842639fc0b0a58cd9b6a5 [id=a253ce0253bd4da4814741c482193d8f]). Expected version 1 got 0
I20251025 07:52:43.135119 2471 raft_consensus.cc:515] T 34315f64b89148efbf1f20a43d5c5efb P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:43.135680 2471 leader_election.cc:290] T 34315f64b89148efbf1f20a43d5c5efb P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:43.131860 774 raft_consensus.cc:2468] T 050a5d3b5f934dd19bf2e221f2fe0780 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:43.136961 2166 log.cc:1199] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/5cba9c05dd424cc788febc614a34819a
I20251025 07:52:43.137907 2166 ts_tablet_manager.cc:1950] T 5cba9c05dd424cc788febc614a34819a P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
W20251025 07:52:43.138548 847 leader_election.cc:343] T 34315f64b89148efbf1f20a43d5c5efb P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: 34315f64b89148efbf1f20a43d5c5efb
I20251025 07:52:43.138691 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 97) took 4494 ms. Trace:
I20251025 07:52:43.138727 847 leader_election.cc:304] T 050a5d3b5f934dd19bf2e221f2fe0780 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:43.138773 2166 rpcz_store.cc:276] 1025 07:52:38.643984 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.644058 (+ 74us) service_pool.cc:225] Handling call
1025 07:52:43.088744 (+4444686us) ts_tablet_manager.cc:1151] Deleting tablet 5cba9c05dd424cc788febc614a34819a
1025 07:52:43.088754 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:43.114716 (+ 25962us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:43.130664 (+ 15948us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:43.138675 (+ 8011us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":128384,"tablet-delete.queue_time_us":4444477}
I20251025 07:52:43.139163 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 98) took 4493 ms. Trace:
I20251025 07:52:43.139238 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 5cba9c05dd424cc788febc614a34819a (table default.Table_a437a2623d704e3ea3c5b8c65c88b936 [id=51b12903e6024e1a9adc2f9fa7400937]) successfully deleted
I20251025 07:52:43.139246 2166 rpcz_store.cc:276] 1025 07:52:38.645752 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.645800 (+ 48us) service_pool.cc:225] Handling call
1025 07:52:43.139058 (+4493258us) ts_tablet_manager.cc:1151] Deleting tablet dda0b26e4f1a44928bc4b84f333901c0
1025 07:52:43.139067 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:43.139150 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4493076}
I20251025 07:52:43.139500 2166 tablet_replica.cc:333] T fab67b56eaa44b54886c17181d8c52b2 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:43.139520 2314 raft_consensus.cc:2804] T 050a5d3b5f934dd19bf2e221f2fe0780 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:43.139631 2314 raft_consensus.cc:697] T 050a5d3b5f934dd19bf2e221f2fe0780 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:43.139703 2166 raft_consensus.cc:2243] T fab67b56eaa44b54886c17181d8c52b2 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
W20251025 07:52:43.139681 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet dda0b26e4f1a44928bc4b84f333901c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dda0b26e4f1a44928bc4b84f333901c0 already in progress: creating tablet
I20251025 07:52:43.139861 2314 consensus_queue.cc:237] T 050a5d3b5f934dd19bf2e221f2fe0780 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.138351 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34315f64b89148efbf1f20a43d5c5efb" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:43.140717 1042 raft_consensus.cc:3060] T 34315f64b89148efbf1f20a43d5c5efb P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:43.141302 1169 ts_tablet_manager.cc:1434] T 1912e59548874e0e8e989050ec0064c5 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251025 07:52:43.142971 823 heartbeater.cc:344] Connected to a master server at 127.0.104.61:42203
I20251025 07:52:43.144137 1042 raft_consensus.cc:2468] T 34315f64b89148efbf1f20a43d5c5efb P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:43.144345 2166 raft_consensus.cc:2272] T fab67b56eaa44b54886c17181d8c52b2 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:43.144572 850 leader_election.cc:304] T 34315f64b89148efbf1f20a43d5c5efb P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b
W20251025 07:52:43.144814 451 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5dab439bc50d4e869548154e1d222ad6 consensus state has no leader
I20251025 07:52:43.144972 2471 raft_consensus.cc:2804] T 34315f64b89148efbf1f20a43d5c5efb P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:43.145047 451 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:47610 (request call id 56) took 4759 ms. Trace:
I20251025 07:52:43.145094 2471 raft_consensus.cc:697] T 34315f64b89148efbf1f20a43d5c5efb P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:43.145087 2166 ts_tablet_manager.cc:1916] T fab67b56eaa44b54886c17181d8c52b2 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:43.145325 2471 consensus_queue.cc:237] T 34315f64b89148efbf1f20a43d5c5efb P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:43.145140 451 rpcz_store.cc:276] 1025 07:52:38.385750 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.633608 (+3247858us) service_pool.cc:225] Handling call
1025 07:52:41.641563 (+ 7955us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:41.641782 (+ 219us) catalog_manager.cc:3708] Apply alter schema
1025 07:52:41.641992 (+ 210us) catalog_manager.cc:3740] Acquired catalog manager lock
1025 07:52:41.642052 (+ 60us) catalog_manager.cc:3898] Updating metadata on disk
1025 07:52:41.649369 (+ 7317us) catalog_manager.cc:3929] Committing alterations to in-memory state
1025 07:52:43.144976 (+1495607us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:41.642626 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:41.642716 (+ 90us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:41.642723 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:41.642725 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:41.642817 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:41.642832 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:41.642837 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:41.642909 (+ 72us) tablet.cc:699] Row locks acquired
1025 07:52:41.642911 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:41.642946 (+ 35us) write_op.cc:270] Start()
1025 07:52:41.642979 (+ 33us) write_op.cc:276] Timestamp: P: 1761378761642928 usec, L: 0
1025 07:52:41.642982 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:41.643182 (+ 200us) log.cc:844] Serialized 405 byte log entry
1025 07:52:41.646664 (+ 3482us) op_driver.cc:464] REPLICATION: finished
1025 07:52:41.646772 (+ 108us) write_op.cc:301] APPLY: starting
1025 07:52:41.646788 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:41.646831 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:41.646883 (+ 52us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:41.646939 (+ 56us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:41.646977 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:41.646985 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:41.647077 (+ 92us) log.cc:844] Serialized 23 byte log entry
1025 07:52:41.647150 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:41.647173 (+ 23us) write_op.cc:454] Released schema lock
1025 07:52:41.647180 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:41.647250 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:41.647253 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":64,"apply.run_cpu_time_us":468,"apply.run_wall_time_us":468,"num_ops":1,"prepare.queue_time_us":52,"prepare.run_cpu_time_us":646,"prepare.run_wall_time_us":647,"raft.queue_time_us":385,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":273,"replication_time_us":3654}]]}
I20251025 07:52:43.157167 2146 ts_tablet_manager.cc:1916] T 71432a6848944a0da06ef9b81f160dc4 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:43.157196 450 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 288e543bb279410c91c01dd1623e110f consensus state has no leader
W20251025 07:52:43.157625 449 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5fc092c2bb2d41ac8c5b4fc07460c8ec consensus state has no leader
W20251025 07:52:43.158083 453 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cb829649f73d4d3f8862d268c953bd86 consensus state has no leader
W20251025 07:52:43.160957 448 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f70091024c034c118fb86578db19a31c consensus state has no leader
I20251025 07:52:43.162602 2166 ts_tablet_manager.cc:1929] T fab67b56eaa44b54886c17181d8c52b2 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:43.162746 2166 log.cc:1199] T fab67b56eaa44b54886c17181d8c52b2 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/fab67b56eaa44b54886c17181d8c52b2
I20251025 07:52:43.163147 2166 ts_tablet_manager.cc:1950] T fab67b56eaa44b54886c17181d8c52b2 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:43.163488 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 99) took 4517 ms. Trace:
I20251025 07:52:43.163702 1169 tablet_bootstrap.cc:492] T 4c155aa7fa5b4d68bbab1918b8432372 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:43.163690 2420 consensus_queue.cc:1048] T 04d27d54fdb7483b869408af65ff85c9 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:43.168829 2166 rpcz_store.cc:276] 1025 07:52:38.646489 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.646530 (+ 41us) service_pool.cc:225] Handling call
1025 07:52:43.139460 (+4492930us) ts_tablet_manager.cc:1151] Deleting tablet fab67b56eaa44b54886c17181d8c52b2
1025 07:52:43.139468 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:43.154546 (+ 15078us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:43.162494 (+ 7948us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:43.163471 (+ 977us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4492779}
I20251025 07:52:43.169281 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet fab67b56eaa44b54886c17181d8c52b2 (table default.Table_6ca844d1e78c4f19939985aec32a8afd [id=1e933863244e4792b4c1b4d6d5c674b7]) successfully deleted
I20251025 07:52:43.169379 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 100) took 4520 ms. Trace:
I20251025 07:52:43.169454 2166 rpcz_store.cc:276] 1025 07:52:38.648789 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.648836 (+ 47us) service_pool.cc:225] Handling call
1025 07:52:43.169257 (+4520421us) ts_tablet_manager.cc:1151] Deleting tablet c11f674385184ae6b235f1db8bf1fb91
1025 07:52:43.169267 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:43.169364 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4520231}
I20251025 07:52:43.169801 2166 tablet_replica.cc:333] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
W20251025 07:52:43.169803 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet c11f674385184ae6b235f1db8bf1fb91 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c11f674385184ae6b235f1db8bf1fb91 already in progress: creating tablet
I20251025 07:52:43.170007 2166 raft_consensus.cc:2243] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:43.170478 2166 raft_consensus.cc:2272] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:43.171196 2166 ts_tablet_manager.cc:1916] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:43.172086 1169 tablet_bootstrap.cc:654] T 4c155aa7fa5b4d68bbab1918b8432372 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:43.180048 2508 raft_consensus.cc:493] T 4a1ec8c2ed7f41abb588f592f62097d1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:43.180197 2508 raft_consensus.cc:515] T 4a1ec8c2ed7f41abb588f592f62097d1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.180895 2508 leader_election.cc:290] T 4a1ec8c2ed7f41abb588f592f62097d1 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:43.181813 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a1ec8c2ed7f41abb588f592f62097d1" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:43.182426 710 leader_election.cc:343] T 4a1ec8c2ed7f41abb588f592f62097d1 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:43.191540 2508 raft_consensus.cc:493] T 143aa3fce9ba4850b000a94054e5dc3a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:43.191695 2508 raft_consensus.cc:515] T 143aa3fce9ba4850b000a94054e5dc3a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.192387 2166 ts_tablet_manager.cc:1929] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:43.192476 2166 log.cc:1199] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/27651312cc2e42a8bfcd245f5d301679
I20251025 07:52:43.192420 2508 leader_election.cc:290] T 143aa3fce9ba4850b000a94054e5dc3a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:43.192799 2166 ts_tablet_manager.cc:1950] T 27651312cc2e42a8bfcd245f5d301679 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:43.193884 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143aa3fce9ba4850b000a94054e5dc3a" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:43.182053 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a1ec8c2ed7f41abb588f592f62097d1" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:43.193441 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143aa3fce9ba4850b000a94054e5dc3a" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:43.197681 2169 tablet_replica.cc:333] T 02e4c4d6ffb34cecaf17beb5ba197e81 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
W20251025 07:52:43.197651 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet c4610b0c62e741eca28c21f63cc3de81 because the tablet was not found. No further retry: Not found: Tablet not found: c4610b0c62e741eca28c21f63cc3de81
I20251025 07:52:43.198562 2169 raft_consensus.cc:2243] T 02e4c4d6ffb34cecaf17beb5ba197e81 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:43.199460 2169 raft_consensus.cc:2272] T 02e4c4d6ffb34cecaf17beb5ba197e81 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:43.200532 2169 ts_tablet_manager.cc:1916] T 02e4c4d6ffb34cecaf17beb5ba197e81 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:43.194437 710 leader_election.cc:343] T 143aa3fce9ba4850b000a94054e5dc3a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:43.196678 712 leader_election.cc:343] T 4a1ec8c2ed7f41abb588f592f62097d1 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:43.207072 712 leader_election.cc:304] T 4a1ec8c2ed7f41abb588f592f62097d1 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:43.207501 712 leader_election.cc:343] T 143aa3fce9ba4850b000a94054e5dc3a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:43.207597 712 leader_election.cc:304] T 143aa3fce9ba4850b000a94054e5dc3a P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:43.208261 2117 raft_consensus.cc:2749] T 4a1ec8c2ed7f41abb588f592f62097d1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:43.208483 2117 raft_consensus.cc:2749] T 143aa3fce9ba4850b000a94054e5dc3a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:43.198221 2508 raft_consensus.cc:493] T c0c6324cbae34dcebade3ef9f89a7150 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:43.208756 2508 raft_consensus.cc:515] T c0c6324cbae34dcebade3ef9f89a7150 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.209357 2508 leader_election.cc:290] T c0c6324cbae34dcebade3ef9f89a7150 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:43.209954 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0c6324cbae34dcebade3ef9f89a7150" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:43.210551 710 leader_election.cc:343] T c0c6324cbae34dcebade3ef9f89a7150 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:43.211779 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 101) took 4561 ms. Trace:
I20251025 07:52:43.214660 1169 tablet_bootstrap.cc:492] T 4c155aa7fa5b4d68bbab1918b8432372 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:43.214599 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0c6324cbae34dcebade3ef9f89a7150" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:43.214763 2166 rpcz_store.cc:276] 1025 07:52:38.650794 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.650836 (+ 42us) service_pool.cc:225] Handling call
1025 07:52:43.169760 (+4518924us) ts_tablet_manager.cc:1151] Deleting tablet 27651312cc2e42a8bfcd245f5d301679
1025 07:52:43.169768 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:43.189318 (+ 19550us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:43.192312 (+ 2994us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:43.211764 (+ 19452us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4518766}
W20251025 07:52:43.215260 712 leader_election.cc:343] T c0c6324cbae34dcebade3ef9f89a7150 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:43.215322 1169 ts_tablet_manager.cc:1403] T 4c155aa7fa5b4d68bbab1918b8432372 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.002s
I20251025 07:52:43.215308 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 27651312cc2e42a8bfcd245f5d301679 (table default.Table_dac528b081844b7c86027169c6234c27 [id=b0caa78ee8c84aa8bea4818cf9feb70c]) successfully deleted
I20251025 07:52:43.215371 712 leader_election.cc:304] T c0c6324cbae34dcebade3ef9f89a7150 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:43.215719 1042 raft_consensus.cc:3060] T 04d27d54fdb7483b869408af65ff85c9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:43.215945 2508 raft_consensus.cc:2749] T c0c6324cbae34dcebade3ef9f89a7150 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:43.216262 1169 raft_consensus.cc:359] T 4c155aa7fa5b4d68bbab1918b8432372 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:43.216502 1169 raft_consensus.cc:385] T 4c155aa7fa5b4d68bbab1918b8432372 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:43.216629 1169 raft_consensus.cc:740] T 4c155aa7fa5b4d68bbab1918b8432372 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:43.216923 1169 consensus_queue.cc:260] T 4c155aa7fa5b4d68bbab1918b8432372 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:43.217654 1169 ts_tablet_manager.cc:1434] T 4c155aa7fa5b4d68bbab1918b8432372 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:43.217978 1169 tablet_bootstrap.cc:492] T aaef24ccdfba45fe8204327723df834e P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:43.220439 2471 raft_consensus.cc:493] T f661d5909657455581dcb5e6eadd3c0b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:43.220608 2471 raft_consensus.cc:515] T f661d5909657455581dcb5e6eadd3c0b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.221581 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f661d5909657455581dcb5e6eadd3c0b" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:43.222069 850 leader_election.cc:343] T f661d5909657455581dcb5e6eadd3c0b P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:43.222332 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 102) took 4570 ms. Trace:
I20251025 07:52:43.222474 2471 leader_election.cc:290] T f661d5909657455581dcb5e6eadd3c0b P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:43.223901 2166 rpcz_store.cc:276] 1025 07:52:38.651492 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.651620 (+ 128us) service_pool.cc:225] Handling call
1025 07:52:43.216204 (+4564584us) ts_tablet_manager.cc:1151] Deleting tablet 4f7c15529be54a11961f385a953b0b03
1025 07:52:43.222162 (+ 5958us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:43.222289 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4564388}
W20251025 07:52:43.224334 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 4f7c15529be54a11961f385a953b0b03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f7c15529be54a11961f385a953b0b03 already in progress: creating tablet
I20251025 07:52:43.227203 822 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
I20251025 07:52:43.227406 446 catalog_manager.cc:6194] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: default.Table_fcddb31989ee4236a4981c12779b3f5d [id=f3794fa081364666b944c412415bdba0] alter complete (version 3)
I20251025 07:52:43.228422 2508 consensus_queue.cc:1048] T 04d27d54fdb7483b869408af65ff85c9 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 07:52:43.230420 1167 tablet_bootstrap.cc:492] T 36c414bc700341188b7d85c14173c3e0 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:43.230931 1167 ts_tablet_manager.cc:1403] T 36c414bc700341188b7d85c14173c3e0 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.108s user 0.005s sys 0.005s
I20251025 07:52:43.234321 2166 tablet_replica.cc:333] T f0bdaa8cfd764f1b83f47881353d26b6 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:43.234671 2166 raft_consensus.cc:2243] T f0bdaa8cfd764f1b83f47881353d26b6 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:43.234932 2166 raft_consensus.cc:2272] T f0bdaa8cfd764f1b83f47881353d26b6 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:43.235658 2166 ts_tablet_manager.cc:1916] T f0bdaa8cfd764f1b83f47881353d26b6 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:43.235659 712 consensus_peers.cc:597] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 23eeffd3de254ce18db5febfde971c6b -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:43.235339 1167 raft_consensus.cc:359] T 36c414bc700341188b7d85c14173c3e0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.236470 1167 raft_consensus.cc:385] T 36c414bc700341188b7d85c14173c3e0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:43.236768 1167 raft_consensus.cc:740] T 36c414bc700341188b7d85c14173c3e0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:43.238309 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f661d5909657455581dcb5e6eadd3c0b" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:43.238595 773 raft_consensus.cc:2468] T f661d5909657455581dcb5e6eadd3c0b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:43.239063 847 leader_election.cc:304] T f661d5909657455581dcb5e6eadd3c0b P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:43.239527 2471 raft_consensus.cc:2804] T f661d5909657455581dcb5e6eadd3c0b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:43.239634 2471 raft_consensus.cc:493] T f661d5909657455581dcb5e6eadd3c0b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:43.239729 2471 raft_consensus.cc:3060] T f661d5909657455581dcb5e6eadd3c0b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:43.244415 2314 raft_consensus.cc:493] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 23eeffd3de254ce18db5febfde971c6b)
I20251025 07:52:43.244555 2314 raft_consensus.cc:515] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:43.245191 2314 leader_election.cc:290] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:43.246042 2484 raft_consensus.cc:493] T 1912e59548874e0e8e989050ec0064c5 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:43.246201 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7fbae8be73bf4e18b5a5c4ec4cdebc12" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:43.246388 1045 raft_consensus.cc:2468] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:43.246749 850 leader_election.cc:304] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:43.246222 2484 raft_consensus.cc:515] T 1912e59548874e0e8e989050ec0064c5 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:43.247530 2484 leader_election.cc:290] T 1912e59548874e0e8e989050ec0064c5 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:43.248453 982 leader_election.cc:343] T 1912e59548874e0e8e989050ec0064c5 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: 1912e59548874e0e8e989050ec0064c5
I20251025 07:52:43.248728 2314 raft_consensus.cc:2804] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 07:52:43.248809 2314 raft_consensus.cc:493] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 23eeffd3de254ce18db5febfde971c6b)
I20251025 07:52:43.248876 2314 raft_consensus.cc:3060] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Advancing to term 2
I20251025 07:52:43.251724 2314 raft_consensus.cc:515] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:43.252511 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1912e59548874e0e8e989050ec0064c5" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:43.253039 847 leader_election.cc:343] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: 7fbae8be73bf4e18b5a5c4ec4cdebc12
I20251025 07:52:43.253073 983 leader_election.cc:304] T 1912e59548874e0e8e989050ec0064c5 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:43.253297 2484 raft_consensus.cc:2749] T 1912e59548874e0e8e989050ec0064c5 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:43.254515 2314 leader_election.cc:290] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 2 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
W20251025 07:52:43.254997 847 leader_election.cc:343] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: 7fbae8be73bf4e18b5a5c4ec4cdebc12
I20251025 07:52:43.255313 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7fbae8be73bf4e18b5a5c4ec4cdebc12" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:43.255508 1046 raft_consensus.cc:3060] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Advancing to term 2
I20251025 07:52:43.237536 1167 consensus_queue.cc:260] T 36c414bc700341188b7d85c14173c3e0 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.257335 960 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
I20251025 07:52:43.257269 1167 ts_tablet_manager.cc:1434] T 36c414bc700341188b7d85c14173c3e0 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.026s user 0.003s sys 0.000s
I20251025 07:52:43.258188 1167 tablet_bootstrap.cc:492] T c7691c7bdd89437c8db060254c0db25d P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
W20251025 07:52:43.259318 712 consensus_peers.cc:597] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f170424be2f643e7af499834955472b0. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:43.265239 2314 consensus_queue.cc:1048] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251025 07:52:43.270298 710 consensus_peers.cc:597] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3001314b56345f5a70314777cae0125. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:43.271335 2471 raft_consensus.cc:515] T f661d5909657455581dcb5e6eadd3c0b P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.276358 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f661d5909657455581dcb5e6eadd3c0b" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:43.276893 850 leader_election.cc:343] T f661d5909657455581dcb5e6eadd3c0b P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:43.279213 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f661d5909657455581dcb5e6eadd3c0b" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:43.279410 774 raft_consensus.cc:3060] T f661d5909657455581dcb5e6eadd3c0b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:43.280040 2166 ts_tablet_manager.cc:1929] T f0bdaa8cfd764f1b83f47881353d26b6 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:43.285166 1169 tablet_bootstrap.cc:654] T aaef24ccdfba45fe8204327723df834e P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:43.287942 2166 log.cc:1199] T f0bdaa8cfd764f1b83f47881353d26b6 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/f0bdaa8cfd764f1b83f47881353d26b6
I20251025 07:52:43.288573 2166 ts_tablet_manager.cc:1950] T f0bdaa8cfd764f1b83f47881353d26b6 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:43.291055 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 103) took 4638 ms. Trace:
I20251025 07:52:43.299513 1169 tablet_bootstrap.cc:492] T aaef24ccdfba45fe8204327723df834e P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:43.299891 2166 rpcz_store.cc:276] 1025 07:52:38.652445 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.652493 (+ 48us) service_pool.cc:225] Handling call
1025 07:52:43.224283 (+4571790us) ts_tablet_manager.cc:1151] Deleting tablet f0bdaa8cfd764f1b83f47881353d26b6
1025 07:52:43.234256 (+ 9973us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:43.249811 (+ 15555us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:43.279901 (+ 30090us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:43.291032 (+ 11131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4571628}
I20251025 07:52:43.300645 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet f0bdaa8cfd764f1b83f47881353d26b6 (table default.Table_49beae553d03426aa395b7cdb6dfec9a [id=ab3328aa6f4148bfa2877cd4f0c6d324]) successfully deleted
I20251025 07:52:43.300911 1169 ts_tablet_manager.cc:1403] T aaef24ccdfba45fe8204327723df834e P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.083s user 0.001s sys 0.003s
I20251025 07:52:43.299228 2471 leader_election.cc:290] T f661d5909657455581dcb5e6eadd3c0b P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:43.286603 2169 ts_tablet_manager.cc:1929] T 02e4c4d6ffb34cecaf17beb5ba197e81 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:43.302496 1042 raft_consensus.cc:3060] T 1912e59548874e0e8e989050ec0064c5 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:43.305181 2169 log.cc:1199] T 02e4c4d6ffb34cecaf17beb5ba197e81 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/02e4c4d6ffb34cecaf17beb5ba197e81
I20251025 07:52:43.305609 2169 ts_tablet_manager.cc:1950] T 02e4c4d6ffb34cecaf17beb5ba197e81 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:43.300719 2166 tablet_replica.cc:333] T 0b21b7da1e2f47829c3f6cdaee689e8a P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:43.306581 2166 raft_consensus.cc:2243] T 0b21b7da1e2f47829c3f6cdaee689e8a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:43.306782 2166 raft_consensus.cc:2272] T 0b21b7da1e2f47829c3f6cdaee689e8a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:43.307399 2166 ts_tablet_manager.cc:1916] T 0b21b7da1e2f47829c3f6cdaee689e8a P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:43.310902 2443 consensus_queue.cc:1048] T 1912e59548874e0e8e989050ec0064c5 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:43.311160 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 114) took 4532 ms. Trace:
I20251025 07:52:43.311246 2169 rpcz_store.cc:276] 1025 07:52:38.778378 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.778634 (+ 256us) service_pool.cc:225] Handling call
1025 07:52:43.197614 (+4418980us) ts_tablet_manager.cc:1151] Deleting tablet 02e4c4d6ffb34cecaf17beb5ba197e81
1025 07:52:43.197623 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:43.208416 (+ 10793us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:43.286460 (+ 78044us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:43.311140 (+ 24680us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4418837}
I20251025 07:52:43.312556 1166 tablet_bootstrap.cc:492] T ab42909afc524f2fa3c1041495d13a59 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:43.312783 1166 ts_tablet_manager.cc:1403] T ab42909afc524f2fa3c1041495d13a59 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.211s user 0.014s sys 0.001s
I20251025 07:52:43.313659 1166 raft_consensus.cc:359] T ab42909afc524f2fa3c1041495d13a59 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.313937 1166 raft_consensus.cc:385] T ab42909afc524f2fa3c1041495d13a59 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:43.314039 1166 raft_consensus.cc:740] T ab42909afc524f2fa3c1041495d13a59 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:43.314424 1166 consensus_queue.cc:260] T ab42909afc524f2fa3c1041495d13a59 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.315057 1166 ts_tablet_manager.cc:1434] T ab42909afc524f2fa3c1041495d13a59 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:43.318032 1166 tablet_bootstrap.cc:492] T 6dcab53a020847a1aa27930265ba0205 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:43.320091 1166 tablet_bootstrap.cc:654] T 6dcab53a020847a1aa27930265ba0205 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:43.304711 1169 raft_consensus.cc:359] T aaef24ccdfba45fe8204327723df834e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:43.325240 1169 raft_consensus.cc:385] T aaef24ccdfba45fe8204327723df834e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:43.325373 1169 raft_consensus.cc:740] T aaef24ccdfba45fe8204327723df834e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:43.325667 1169 consensus_queue.cc:260] T aaef24ccdfba45fe8204327723df834e P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:43.326261 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 02e4c4d6ffb34cecaf17beb5ba197e81 (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]) successfully deleted
I20251025 07:52:43.326051 2169 tablet_replica.cc:333] T 7c7a9d7a4fb3449bb57eafa83230a839 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:43.326611 2169 raft_consensus.cc:2243] T 7c7a9d7a4fb3449bb57eafa83230a839 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:43.327070 2169 raft_consensus.cc:2272] T 7c7a9d7a4fb3449bb57eafa83230a839 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:43.327420 847 consensus_peers.cc:597] T 1912e59548874e0e8e989050ec0064c5 P 3aea2869ff814cc7b142012faa274382 -> Peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Couldn't send request to peer 23eeffd3de254ce18db5febfde971c6b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1912e59548874e0e8e989050ec0064c5. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:43.328262 2169 ts_tablet_manager.cc:1916] T 7c7a9d7a4fb3449bb57eafa83230a839 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:43.327697 1169 ts_tablet_manager.cc:1434] T aaef24ccdfba45fe8204327723df834e P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20251025 07:52:43.331228 1169 tablet_bootstrap.cc:492] T 4b28215f61574cf48c447dd01955ac77 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:43.332654 1169 tablet_bootstrap.cc:654] T 4b28215f61574cf48c447dd01955ac77 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:43.342700 2146 ts_tablet_manager.cc:1929] T 71432a6848944a0da06ef9b81f160dc4 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:43.342967 2146 log.cc:1199] T 71432a6848944a0da06ef9b81f160dc4 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/71432a6848944a0da06ef9b81f160dc4
I20251025 07:52:43.343385 2146 ts_tablet_manager.cc:1950] T 71432a6848944a0da06ef9b81f160dc4 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:43.344051 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 121) took 4824 ms. Trace:
I20251025 07:52:43.344221 2146 rpcz_store.cc:276] 1025 07:52:38.519197 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.519248 (+ 51us) service_pool.cc:225] Handling call
1025 07:52:43.093529 (+4574281us) ts_tablet_manager.cc:1151] Deleting tablet 71432a6848944a0da06ef9b81f160dc4
1025 07:52:43.093538 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:43.191486 (+ 97948us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:43.342580 (+151094us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:43.344008 (+ 1428us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":446,"tablet-delete.queue_time_us":4569036}
I20251025 07:52:43.345011 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 71432a6848944a0da06ef9b81f160dc4 (table default.Table_79969e4ace6f4227b2748b59136027d6 [id=a1135611169142ba9ef987e3e9f95ce8]) successfully deleted
I20251025 07:52:43.345487 2146 tablet_replica.cc:333] T c4610b0c62e741eca28c21f63cc3de81 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:43.345873 2146 raft_consensus.cc:2243] T c4610b0c62e741eca28c21f63cc3de81 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:43.346251 2146 raft_consensus.cc:2272] T c4610b0c62e741eca28c21f63cc3de81 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:43.346899 1169 tablet_bootstrap.cc:492] T 4b28215f61574cf48c447dd01955ac77 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:43.347079 1169 ts_tablet_manager.cc:1403] T 4b28215f61574cf48c447dd01955ac77 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.001s
I20251025 07:52:43.347815 1169 raft_consensus.cc:359] T 4b28215f61574cf48c447dd01955ac77 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:43.348053 1169 raft_consensus.cc:385] T 4b28215f61574cf48c447dd01955ac77 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:43.348148 1169 raft_consensus.cc:740] T 4b28215f61574cf48c447dd01955ac77 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:43.348387 1169 consensus_queue.cc:260] T 4b28215f61574cf48c447dd01955ac77 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:43.348959 1169 ts_tablet_manager.cc:1434] T 4b28215f61574cf48c447dd01955ac77 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:43.350447 1167 tablet_bootstrap.cc:654] T c7691c7bdd89437c8db060254c0db25d P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:43.350762 1169 tablet_bootstrap.cc:492] T cb829649f73d4d3f8862d268c953bd86 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:43.351915 1095 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
W20251025 07:52:43.353276 850 consensus_peers.cc:597] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:43.352123 1169 tablet_bootstrap.cc:654] T cb829649f73d4d3f8862d268c953bd86 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:43.347090 1046 raft_consensus.cc:2468] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P db42be7ad3c741a0aa427f27e245993b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 2.
I20251025 07:52:43.355654 850 leader_election.cc:304] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b
I20251025 07:52:43.355952 1166 tablet_bootstrap.cc:492] T 6dcab53a020847a1aa27930265ba0205 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:43.356117 1166 ts_tablet_manager.cc:1403] T 6dcab53a020847a1aa27930265ba0205 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.038s user 0.000s sys 0.005s
I20251025 07:52:43.356189 2471 raft_consensus.cc:2804] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382 [term 2 FOLLOWER]: Leader election won for term 2
I20251025 07:52:43.356451 2471 raft_consensus.cc:697] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382 [term 2 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:43.356730 2471 consensus_queue.cc:237] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:43.357163 1166 raft_consensus.cc:359] T 6dcab53a020847a1aa27930265ba0205 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.357378 1166 raft_consensus.cc:385] T 6dcab53a020847a1aa27930265ba0205 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:43.357695 1166 raft_consensus.cc:740] T 6dcab53a020847a1aa27930265ba0205 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:43.358217 1166 consensus_queue.cc:260] T 6dcab53a020847a1aa27930265ba0205 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.359031 1166 ts_tablet_manager.cc:1434] T 6dcab53a020847a1aa27930265ba0205 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251025 07:52:43.359367 1166 tablet_bootstrap.cc:492] T 30418858bf95438686bdfe64ac733144 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:43.361284 1166 tablet_bootstrap.cc:654] T 30418858bf95438686bdfe64ac733144 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:43.362223 2169 ts_tablet_manager.cc:1929] T 7c7a9d7a4fb3449bb57eafa83230a839 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:43.364796 2169 log.cc:1199] T 7c7a9d7a4fb3449bb57eafa83230a839 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/7c7a9d7a4fb3449bb57eafa83230a839
I20251025 07:52:43.365190 2169 ts_tablet_manager.cc:1950] T 7c7a9d7a4fb3449bb57eafa83230a839 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:43.364562 2105 raft_consensus.cc:493] T 66f0d398e59a4175aa9fe90715124d1a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:43.365330 2105 raft_consensus.cc:515] T 66f0d398e59a4175aa9fe90715124d1a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.365921 2105 leader_election.cc:290] T 66f0d398e59a4175aa9fe90715124d1a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:43.366822 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66f0d398e59a4175aa9fe90715124d1a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:43.366986 770 raft_consensus.cc:2468] T 66f0d398e59a4175aa9fe90715124d1a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:43.367431 847 leader_election.cc:304] T 66f0d398e59a4175aa9fe90715124d1a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:43.367689 2314 consensus_queue.cc:1048] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:43.367841 2105 raft_consensus.cc:2804] T 66f0d398e59a4175aa9fe90715124d1a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:43.367915 2105 raft_consensus.cc:493] T 66f0d398e59a4175aa9fe90715124d1a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:43.367985 2105 raft_consensus.cc:3060] T 66f0d398e59a4175aa9fe90715124d1a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:43.371891 2105 raft_consensus.cc:515] T 66f0d398e59a4175aa9fe90715124d1a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.373831 2105 leader_election.cc:290] T 66f0d398e59a4175aa9fe90715124d1a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:43.373915 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66f0d398e59a4175aa9fe90715124d1a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:43.374082 770 raft_consensus.cc:3060] T 66f0d398e59a4175aa9fe90715124d1a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:43.374522 2420 raft_consensus.cc:493] T ebc824ec4c60411dab1a2ea4a385148e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:43.374624 2420 raft_consensus.cc:515] T ebc824ec4c60411dab1a2ea4a385148e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.375135 2420 leader_election.cc:290] T ebc824ec4c60411dab1a2ea4a385148e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:43.375702 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc824ec4c60411dab1a2ea4a385148e" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:43.375910 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 115) took 4595 ms. Trace:
I20251025 07:52:43.375998 2169 rpcz_store.cc:276] 1025 07:52:38.780719 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.780803 (+ 84us) service_pool.cc:225] Handling call
1025 07:52:43.325979 (+4545176us) ts_tablet_manager.cc:1151] Deleting tablet 7c7a9d7a4fb3449bb57eafa83230a839
1025 07:52:43.325989 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:43.346095 (+ 20106us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:43.362094 (+ 15999us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:43.375895 (+ 13801us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4545012}
W20251025 07:52:43.376155 710 leader_election.cc:343] T ebc824ec4c60411dab1a2ea4a385148e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:43.376322 2169 tablet_replica.cc:333] T 70fe9aa2f58d4abea3b3b71066a29653 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:43.376546 2169 raft_consensus.cc:2243] T 70fe9aa2f58d4abea3b3b71066a29653 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:43.376765 770 raft_consensus.cc:2468] T 66f0d398e59a4175aa9fe90715124d1a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:43.377146 847 leader_election.cc:304] T 66f0d398e59a4175aa9fe90715124d1a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:43.377431 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 7c7a9d7a4fb3449bb57eafa83230a839 (table default.Table_192549a3e67840f3aefc31029cf2ff52 [id=a37be02f8e994f82baf2f4f5ec800926]) successfully deleted
I20251025 07:52:43.366532 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66f0d398e59a4175aa9fe90715124d1a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:43.377625 2105 raft_consensus.cc:2804] T 66f0d398e59a4175aa9fe90715124d1a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:43.377920 2105 raft_consensus.cc:697] T 66f0d398e59a4175aa9fe90715124d1a P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
W20251025 07:52:43.378072 850 leader_election.cc:343] T 66f0d398e59a4175aa9fe90715124d1a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:43.373106 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66f0d398e59a4175aa9fe90715124d1a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:43.378727 850 leader_election.cc:343] T 66f0d398e59a4175aa9fe90715124d1a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:43.383584 2105 consensus_queue.cc:237] T 66f0d398e59a4175aa9fe90715124d1a P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.376937 1169 tablet_bootstrap.cc:492] T cb829649f73d4d3f8862d268c953bd86 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:43.381441 2484 consensus_queue.cc:1048] T f43ef51ee45d47a99ec732b49d50d3b4 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20251025 07:52:43.389902 1169 ts_tablet_manager.cc:1403] T cb829649f73d4d3f8862d268c953bd86 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.001s
I20251025 07:52:43.377029 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc824ec4c60411dab1a2ea4a385148e" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:43.390771 1169 raft_consensus.cc:359] T cb829649f73d4d3f8862d268c953bd86 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.390936 1169 raft_consensus.cc:385] T cb829649f73d4d3f8862d268c953bd86 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:43.390993 1169 raft_consensus.cc:740] T cb829649f73d4d3f8862d268c953bd86 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
W20251025 07:52:43.391003 712 leader_election.cc:343] T ebc824ec4c60411dab1a2ea4a385148e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:43.391127 712 leader_election.cc:304] T ebc824ec4c60411dab1a2ea4a385148e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:43.391182 1169 consensus_queue.cc:260] T cb829649f73d4d3f8862d268c953bd86 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.391505 2420 raft_consensus.cc:2749] T ebc824ec4c60411dab1a2ea4a385148e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:43.391651 1169 ts_tablet_manager.cc:1434] T cb829649f73d4d3f8862d268c953bd86 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:43.391952 1169 tablet_bootstrap.cc:492] T 26c849d376d143c6bef0c8e0e1d7d03a P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:43.393533 1169 tablet_bootstrap.cc:654] T 26c849d376d143c6bef0c8e0e1d7d03a P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:43.397768 2420 raft_consensus.cc:493] T a77eb82f26ae43b48de1e85693372d34 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:43.397924 2420 raft_consensus.cc:515] T a77eb82f26ae43b48de1e85693372d34 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.398581 2420 leader_election.cc:290] T a77eb82f26ae43b48de1e85693372d34 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:43.399788 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a77eb82f26ae43b48de1e85693372d34" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:43.400372 712 leader_election.cc:343] T a77eb82f26ae43b48de1e85693372d34 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:43.376889 2169 raft_consensus.cc:2272] T 70fe9aa2f58d4abea3b3b71066a29653 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:43.401768 2169 ts_tablet_manager.cc:1916] T 70fe9aa2f58d4abea3b3b71066a29653 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:43.401993 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a77eb82f26ae43b48de1e85693372d34" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:43.402655 710 leader_election.cc:343] T a77eb82f26ae43b48de1e85693372d34 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:43.402873 710 leader_election.cc:304] T a77eb82f26ae43b48de1e85693372d34 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:43.403604 2420 raft_consensus.cc:2749] T a77eb82f26ae43b48de1e85693372d34 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:43.405259 2443 consensus_queue.cc:1048] T 36262ab8c6964dd684c01b4db6219a4f P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:43.410424 2420 raft_consensus.cc:493] T 2e2e6893a48449969b71ee1c08d1960e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:43.410627 2420 raft_consensus.cc:515] T 2e2e6893a48449969b71ee1c08d1960e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.411504 2420 leader_election.cc:290] T 2e2e6893a48449969b71ee1c08d1960e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:43.412276 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e2e6893a48449969b71ee1c08d1960e" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:43.412277 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e2e6893a48449969b71ee1c08d1960e" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:43.413861 712 leader_election.cc:343] T 2e2e6893a48449969b71ee1c08d1960e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:43.414430 710 leader_election.cc:343] T 2e2e6893a48449969b71ee1c08d1960e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:43.414606 710 leader_election.cc:304] T 2e2e6893a48449969b71ee1c08d1960e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:43.414970 2420 raft_consensus.cc:2749] T 2e2e6893a48449969b71ee1c08d1960e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:43.419653 1167 tablet_bootstrap.cc:492] T c7691c7bdd89437c8db060254c0db25d P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:43.419808 1167 ts_tablet_manager.cc:1403] T c7691c7bdd89437c8db060254c0db25d P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.162s user 0.004s sys 0.000s
I20251025 07:52:43.420694 1167 raft_consensus.cc:359] T c7691c7bdd89437c8db060254c0db25d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.420898 1167 raft_consensus.cc:385] T c7691c7bdd89437c8db060254c0db25d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:43.420948 1167 raft_consensus.cc:740] T c7691c7bdd89437c8db060254c0db25d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:43.421181 1167 consensus_queue.cc:260] T c7691c7bdd89437c8db060254c0db25d P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.421757 1167 ts_tablet_manager.cc:1434] T c7691c7bdd89437c8db060254c0db25d P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20251025 07:52:43.423681 710 consensus_peers.cc:597] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f170424be2f643e7af499834955472b0. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:43.426743 450 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:47610 (request call id 58) took 4905 ms. Trace:
I20251025 07:52:43.426898 450 rpcz_store.cc:276] 1025 07:52:38.521552 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.660925 (+3139373us) service_pool.cc:225] Handling call
1025 07:52:41.661329 (+ 404us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:41.661558 (+ 229us) catalog_manager.cc:3708] Apply alter schema
1025 07:52:41.661744 (+ 186us) catalog_manager.cc:3740] Acquired catalog manager lock
1025 07:52:41.661818 (+ 74us) catalog_manager.cc:3898] Updating metadata on disk
1025 07:52:41.680719 (+ 18901us) catalog_manager.cc:3929] Committing alterations to in-memory state
1025 07:52:43.426690 (+1745971us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:41.662776 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:41.662891 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:41.662896 (+ 5us) write_op.cc:435] Acquired schema lock
1025 07:52:41.662898 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:41.663077 (+ 179us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:41.663091 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:41.663096 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:41.663173 (+ 77us) tablet.cc:699] Row locks acquired
1025 07:52:41.663175 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:41.663210 (+ 35us) write_op.cc:270] Start()
1025 07:52:41.663247 (+ 37us) write_op.cc:276] Timestamp: P: 1761378761663192 usec, L: 0
1025 07:52:41.663251 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:41.663430 (+ 179us) log.cc:844] Serialized 405 byte log entry
1025 07:52:41.668008 (+ 4578us) op_driver.cc:464] REPLICATION: finished
1025 07:52:41.668129 (+ 121us) write_op.cc:301] APPLY: starting
1025 07:52:41.668145 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:41.668189 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:41.668256 (+ 67us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:41.668315 (+ 59us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:41.668354 (+ 39us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:41.668363 (+ 9us) write_op.cc:312] APPLY: finished
1025 07:52:41.668441 (+ 78us) log.cc:844] Serialized 23 byte log entry
1025 07:52:41.668503 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:41.668525 (+ 22us) write_op.cc:454] Released schema lock
1025 07:52:41.668533 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:41.668603 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:41.668607 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":70,"apply.run_cpu_time_us":466,"apply.run_wall_time_us":465,"num_ops":1,"prepare.queue_time_us":328,"prepare.run_cpu_time_us":709,"prepare.run_wall_time_us":741,"raft.queue_time_us":944,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":217,"replication_time_us":4727}]]}
W20251025 07:52:43.428216 450 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:42387 (request call id 214) took 1175 ms (client timeout 1000 ms). Trace:
W20251025 07:52:43.428341 450 rpcz_store.cc:269] 1025 07:52:42.252361 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:43.427944 (+1175583us) service_pool.cc:212] Skipping call since client already timed out
1025 07:52:43.428177 (+ 233us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20251025 07:52:43.431877 2146 ts_tablet_manager.cc:1916] T c4610b0c62e741eca28c21f63cc3de81 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:43.434270 448 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:47610 (request call id 59) took 4912 ms. Trace:
I20251025 07:52:43.434441 448 rpcz_store.cc:276] 1025 07:52:38.521646 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.672673 (+3151027us) service_pool.cc:225] Handling call
1025 07:52:41.673130 (+ 457us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:41.673380 (+ 250us) catalog_manager.cc:3708] Apply alter schema
1025 07:52:41.673576 (+ 196us) catalog_manager.cc:3740] Acquired catalog manager lock
1025 07:52:41.673647 (+ 71us) catalog_manager.cc:3898] Updating metadata on disk
1025 07:52:41.700518 (+ 26871us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:41.700526 (+ 8us) write_op.cc:454] Released schema lock
1025 07:52:41.700623 (+ 97us) catalog_manager.cc:3929] Committing alterations to in-memory state
1025 07:52:43.434227 (+1733604us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:41.674213 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:41.674323 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:41.674329 (+ 6us) write_op.cc:435] Acquired schema lock
1025 07:52:41.674331 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:41.674445 (+ 114us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:41.674461 (+ 16us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:41.674467 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:41.674529 (+ 62us) tablet.cc:699] Row locks acquired
1025 07:52:41.674533 (+ 4us) write_op.cc:260] PREPARE: finished
1025 07:52:41.674567 (+ 34us) write_op.cc:270] Start()
1025 07:52:41.674597 (+ 30us) write_op.cc:276] Timestamp: P: 1761378761674549 usec, L: 0
1025 07:52:41.674601 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:41.674781 (+ 180us) log.cc:844] Serialized 405 byte log entry
1025 07:52:41.678319 (+ 3538us) op_driver.cc:464] REPLICATION: finished
1025 07:52:41.678420 (+ 101us) write_op.cc:301] APPLY: starting
1025 07:52:41.678436 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:41.678478 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:41.678513 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:41.678567 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:41.678601 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:41.678612 (+ 11us) write_op.cc:312] APPLY: finished
1025 07:52:41.678687 (+ 75us) log.cc:844] Serialized 23 byte log entry
1025 07:52:41.678750 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:41.678769 (+ 19us) write_op.cc:454] Released schema lock
1025 07:52:41.678775 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":56,"apply.run_cpu_time_us":408,"apply.run_wall_time_us":406,"num_ops":1,"prepare.queue_time_us":59,"prepare.run_cpu_time_us":670,"prepare.run_wall_time_us":671,"raft.queue_time_us":271,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":231,"replication_time_us":3692}]]}
I20251025 07:52:43.435671 448 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_9ff26a6660b84607835d929035c0b285"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:43.436502 448 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ff26a6660b84607835d929035c0b285 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:43.437767 453 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:47610 (request call id 62) took 4882 ms. Trace:
I20251025 07:52:43.437907 453 rpcz_store.cc:276] 1025 07:52:38.555276 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.841109 (+3285833us) service_pool.cc:225] Handling call
1025 07:52:41.841522 (+ 413us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:41.841761 (+ 239us) catalog_manager.cc:3708] Apply alter schema
1025 07:52:41.841970 (+ 209us) catalog_manager.cc:3740] Acquired catalog manager lock
1025 07:52:41.842101 (+ 131us) catalog_manager.cc:3898] Updating metadata on disk
1025 07:52:41.849190 (+ 7089us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:41.849195 (+ 5us) write_op.cc:454] Released schema lock
1025 07:52:41.849262 (+ 67us) catalog_manager.cc:3929] Committing alterations to in-memory state
1025 07:52:43.437725 (+1588463us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:41.842700 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:41.842793 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:41.842799 (+ 6us) write_op.cc:435] Acquired schema lock
1025 07:52:41.842801 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:41.842915 (+ 114us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:41.842930 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:41.842936 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:41.843006 (+ 70us) tablet.cc:699] Row locks acquired
1025 07:52:41.843011 (+ 5us) write_op.cc:260] PREPARE: finished
1025 07:52:41.843050 (+ 39us) write_op.cc:270] Start()
1025 07:52:41.843082 (+ 32us) write_op.cc:276] Timestamp: P: 1761378761843029 usec, L: 0
1025 07:52:41.843086 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:41.843283 (+ 197us) log.cc:844] Serialized 405 byte log entry
1025 07:52:41.846565 (+ 3282us) op_driver.cc:464] REPLICATION: finished
1025 07:52:41.846668 (+ 103us) write_op.cc:301] APPLY: starting
1025 07:52:41.846682 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:41.846759 (+ 77us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:41.846789 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:41.846860 (+ 71us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:41.846922 (+ 62us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:41.846932 (+ 10us) write_op.cc:312] APPLY: finished
1025 07:52:41.847008 (+ 76us) log.cc:844] Serialized 23 byte log entry
1025 07:52:41.847075 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:41.847095 (+ 20us) write_op.cc:454] Released schema lock
1025 07:52:41.847101 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":56,"apply.run_cpu_time_us":487,"apply.run_wall_time_us":486,"num_ops":1,"prepare.queue_time_us":43,"prepare.run_cpu_time_us":680,"prepare.run_wall_time_us":679,"raft.queue_time_us":57,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":215,"replication_time_us":3448}]]}
I20251025 07:52:43.438978 453 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_10089cff605d4c26aabf807af44b50ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:43.439796 453 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10089cff605d4c26aabf807af44b50ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:43.440989 449 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:47610 (request call id 57) took 5030 ms. Trace:
I20251025 07:52:43.441125 449 rpcz_store.cc:276] 1025 07:52:38.410474 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.637309 (+3226835us) service_pool.cc:225] Handling call
1025 07:52:41.650016 (+ 12707us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:41.650267 (+ 251us) catalog_manager.cc:3708] Apply alter schema
1025 07:52:41.650456 (+ 189us) catalog_manager.cc:3740] Acquired catalog manager lock
1025 07:52:41.650535 (+ 79us) catalog_manager.cc:3898] Updating metadata on disk
1025 07:52:41.657103 (+ 6568us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:41.657109 (+ 6us) write_op.cc:454] Released schema lock
1025 07:52:41.657186 (+ 77us) catalog_manager.cc:3929] Committing alterations to in-memory state
1025 07:52:43.440926 (+1783740us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:41.651022 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:41.651128 (+ 106us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:41.651133 (+ 5us) write_op.cc:435] Acquired schema lock
1025 07:52:41.651136 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:41.651220 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:41.651231 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:41.651236 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:41.651284 (+ 48us) tablet.cc:699] Row locks acquired
1025 07:52:41.651286 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:41.651317 (+ 31us) write_op.cc:270] Start()
1025 07:52:41.651343 (+ 26us) write_op.cc:276] Timestamp: P: 1761378761651299 usec, L: 0
1025 07:52:41.651346 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:41.651497 (+ 151us) log.cc:844] Serialized 405 byte log entry
1025 07:52:41.654735 (+ 3238us) op_driver.cc:464] REPLICATION: finished
1025 07:52:41.654826 (+ 91us) write_op.cc:301] APPLY: starting
1025 07:52:41.654850 (+ 24us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:41.654891 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:41.654914 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:41.654958 (+ 44us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:41.654988 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:41.654995 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:41.655066 (+ 71us) log.cc:844] Serialized 23 byte log entry
1025 07:52:41.655120 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:41.655138 (+ 18us) write_op.cc:454] Released schema lock
1025 07:52:41.655143 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":51,"apply.run_cpu_time_us":366,"apply.run_wall_time_us":364,"num_ops":1,"prepare.queue_time_us":46,"prepare.run_cpu_time_us":562,"prepare.run_wall_time_us":563,"raft.queue_time_us":58,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":190,"replication_time_us":3361}]]}
I20251025 07:52:43.442420 449 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_2afb6a7dc14f48cdb5c7e02852a40cf8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:43.443002 449 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2afb6a7dc14f48cdb5c7e02852a40cf8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:43.453253 2169 ts_tablet_manager.cc:1929] T 70fe9aa2f58d4abea3b3b71066a29653 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:43.455242 2169 log.cc:1199] T 70fe9aa2f58d4abea3b3b71066a29653 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/70fe9aa2f58d4abea3b3b71066a29653
I20251025 07:52:43.456125 2166 ts_tablet_manager.cc:1929] T 0b21b7da1e2f47829c3f6cdaee689e8a P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:43.456353 2166 log.cc:1199] T 0b21b7da1e2f47829c3f6cdaee689e8a P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/0b21b7da1e2f47829c3f6cdaee689e8a
I20251025 07:52:43.456736 2166 ts_tablet_manager.cc:1950] T 0b21b7da1e2f47829c3f6cdaee689e8a P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
W20251025 07:52:43.457588 712 consensus_peers.cc:597] T 943ec703e65f4fd3b0c690e08c0bd226 P 23eeffd3de254ce18db5febfde971c6b -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 943ec703e65f4fd3b0c690e08c0bd226. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:43.461367 2420 raft_consensus.cc:493] T 627a9cfd398346f2aa34c8932a11b5a8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:43.461485 2420 raft_consensus.cc:515] T 627a9cfd398346f2aa34c8932a11b5a8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.462661 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627a9cfd398346f2aa34c8932a11b5a8" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:43.462074 2420 leader_election.cc:290] T 627a9cfd398346f2aa34c8932a11b5a8 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:43.463204 710 leader_election.cc:343] T 627a9cfd398346f2aa34c8932a11b5a8 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:43.464112 448 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 68) took 4768 ms. Trace:
I20251025 07:52:43.464241 448 rpcz_store.cc:276] 1025 07:52:38.696033 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:43.435443 (+4739410us) service_pool.cc:225] Handling call
1025 07:52:43.436648 (+ 1205us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:43.437173 (+ 525us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:43.463751 (+ 26578us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:43.463774 (+ 23us) write_op.cc:454] Released schema lock
1025 07:52:43.463850 (+ 76us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:43.463964 (+ 114us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:43.464067 (+ 103us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:43.457335 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:43.457460 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:43.457466 (+ 6us) write_op.cc:435] Acquired schema lock
1025 07:52:43.457469 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:43.457560 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:43.457573 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:43.457578 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:43.457659 (+ 81us) tablet.cc:699] Row locks acquired
1025 07:52:43.457662 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:43.457706 (+ 44us) write_op.cc:270] Start()
1025 07:52:43.457755 (+ 49us) write_op.cc:276] Timestamp: P: 1761378763457685 usec, L: 0
1025 07:52:43.457759 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:43.457973 (+ 214us) log.cc:844] Serialized 433 byte log entry
1025 07:52:43.461224 (+ 3251us) op_driver.cc:464] REPLICATION: finished
1025 07:52:43.461349 (+ 125us) write_op.cc:301] APPLY: starting
1025 07:52:43.461365 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:43.461413 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:43.461438 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:43.461538 (+ 100us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:43.461573 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:43.461580 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:43.461651 (+ 71us) log.cc:844] Serialized 29 byte log entry
1025 07:52:43.461706 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:43.461724 (+ 18us) write_op.cc:454] Released schema lock
1025 07:52:43.461730 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":77,"apply.run_cpu_time_us":437,"apply.run_wall_time_us":437,"num_ops":2,"prepare.queue_time_us":19641,"prepare.run_cpu_time_us":753,"prepare.run_wall_time_us":753,"raft.queue_time_us":371,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":235,"replication_time_us":3439}]]}
I20251025 07:52:43.465168 448 master_service.cc:954] ReplaceTablet: received request to replace tablet 4c155aa7fa5b4d68bbab1918b8432372 from {username='slave'} at 127.0.0.1:47610
I20251025 07:52:43.463630 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627a9cfd398346f2aa34c8932a11b5a8" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:43.467715 712 leader_election.cc:343] T 627a9cfd398346f2aa34c8932a11b5a8 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:43.467869 712 leader_election.cc:304] T 627a9cfd398346f2aa34c8932a11b5a8 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:43.468295 2420 raft_consensus.cc:2749] T 627a9cfd398346f2aa34c8932a11b5a8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:43.469681 2169 ts_tablet_manager.cc:1950] T 70fe9aa2f58d4abea3b3b71066a29653 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:43.470062 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 117) took 4665 ms. Trace:
I20251025 07:52:43.470213 2169 rpcz_store.cc:276] 1025 07:52:38.804215 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.808756 (+ 4541us) service_pool.cc:225] Handling call
1025 07:52:43.376269 (+4567513us) ts_tablet_manager.cc:1151] Deleting tablet 70fe9aa2f58d4abea3b3b71066a29653
1025 07:52:43.376280 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:43.419366 (+ 43086us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:43.453120 (+ 33754us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:43.470047 (+ 16927us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4567243}
I20251025 07:52:43.470569 2314 raft_consensus.cc:493] T 36c8fcafd5464b4e999276a8062a812a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:43.470722 2169 tablet_replica.cc:333] T f4578ce69a0d4322944e5ce7c8a59831 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:43.470690 2314 raft_consensus.cc:515] T 36c8fcafd5464b4e999276a8062a812a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.470917 2169 raft_consensus.cc:2243] T f4578ce69a0d4322944e5ce7c8a59831 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:43.471002 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 70fe9aa2f58d4abea3b3b71066a29653 (table default.Table_517752d265854c7cbd706e2742536620 [id=aa0642dd1fa34cf8bd1fe2070037a55a]) successfully deleted
I20251025 07:52:43.471087 2169 raft_consensus.cc:2272] T f4578ce69a0d4322944e5ce7c8a59831 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:43.471240 2314 leader_election.cc:290] T 36c8fcafd5464b4e999276a8062a812a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:43.471614 2169 ts_tablet_manager.cc:1916] T f4578ce69a0d4322944e5ce7c8a59831 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:43.472119 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36c8fcafd5464b4e999276a8062a812a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:43.472545 850 leader_election.cc:343] T 36c8fcafd5464b4e999276a8062a812a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:43.486120 2169 ts_tablet_manager.cc:1929] T f4578ce69a0d4322944e5ce7c8a59831 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:43.487800 2169 log.cc:1199] T f4578ce69a0d4322944e5ce7c8a59831 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/f4578ce69a0d4322944e5ce7c8a59831
I20251025 07:52:43.487917 449 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 70) took 4729 ms. Trace:
I20251025 07:52:43.488121 449 rpcz_store.cc:276] 1025 07:52:38.758494 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:43.442226 (+4683732us) service_pool.cc:225] Handling call
1025 07:52:43.443127 (+ 901us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:43.443536 (+ 409us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:43.487599 (+ 44063us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:43.487729 (+ 130us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:43.487847 (+ 118us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:43.467085 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:43.467183 (+ 98us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:43.467188 (+ 5us) write_op.cc:435] Acquired schema lock
1025 07:52:43.467190 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:43.467277 (+ 87us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:43.467288 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:43.467295 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:43.467359 (+ 64us) tablet.cc:699] Row locks acquired
1025 07:52:43.467361 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:43.467388 (+ 27us) write_op.cc:270] Start()
1025 07:52:43.467423 (+ 35us) write_op.cc:276] Timestamp: P: 1761378763467373 usec, L: 0
1025 07:52:43.467426 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:43.467569 (+ 143us) log.cc:844] Serialized 433 byte log entry
1025 07:52:43.472301 (+ 4732us) op_driver.cc:464] REPLICATION: finished
1025 07:52:43.473046 (+ 745us) write_op.cc:301] APPLY: starting
1025 07:52:43.473055 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:43.473087 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:43.473106 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:43.473199 (+ 93us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:43.473223 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:43.473230 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:43.473275 (+ 45us) log.cc:844] Serialized 29 byte log entry
1025 07:52:43.473308 (+ 33us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:43.473322 (+ 14us) write_op.cc:454] Released schema lock
1025 07:52:43.473327 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":715,"apply.run_cpu_time_us":318,"apply.run_wall_time_us":318,"mutex_wait_us":22385,"num_ops":2,"prepare.queue_time_us":23103,"prepare.run_cpu_time_us":558,"prepare.run_wall_time_us":557,"raft.queue_time_us":332,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":4861,"threads_started":1}]]}
I20251025 07:52:43.491562 2169 ts_tablet_manager.cc:1950] T f4578ce69a0d4322944e5ce7c8a59831 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:43.494621 774 raft_consensus.cc:2468] T f661d5909657455581dcb5e6eadd3c0b P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:43.497244 1166 tablet_bootstrap.cc:492] T 30418858bf95438686bdfe64ac733144 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:43.497432 1166 ts_tablet_manager.cc:1403] T 30418858bf95438686bdfe64ac733144 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.138s user 0.005s sys 0.007s
I20251025 07:52:43.498481 1166 raft_consensus.cc:359] T 30418858bf95438686bdfe64ac733144 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.498706 1166 raft_consensus.cc:385] T 30418858bf95438686bdfe64ac733144 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:43.498762 1166 raft_consensus.cc:740] T 30418858bf95438686bdfe64ac733144 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:43.499019 1166 consensus_queue.cc:260] T 30418858bf95438686bdfe64ac733144 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.499661 1166 ts_tablet_manager.cc:1434] T 30418858bf95438686bdfe64ac733144 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:43.502532 1166 tablet_bootstrap.cc:492] T f656945454fe4921ab819c772a089eb3 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:43.504865 2146 ts_tablet_manager.cc:1929] T c4610b0c62e741eca28c21f63cc3de81 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:43.505159 2146 log.cc:1199] T c4610b0c62e741eca28c21f63cc3de81 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/c4610b0c62e741eca28c21f63cc3de81
I20251025 07:52:43.508327 1166 tablet_bootstrap.cc:654] T f656945454fe4921ab819c772a089eb3 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:43.508917 847 leader_election.cc:304] T f661d5909657455581dcb5e6eadd3c0b P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:43.509176 2105 raft_consensus.cc:2804] T f661d5909657455581dcb5e6eadd3c0b P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:43.509284 2105 raft_consensus.cc:697] T f661d5909657455581dcb5e6eadd3c0b P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:43.509552 2105 consensus_queue.cc:237] T f661d5909657455581dcb5e6eadd3c0b P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.510308 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 118) took 4701 ms. Trace:
I20251025 07:52:43.510411 2169 rpcz_store.cc:276] 1025 07:52:38.809268 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.809543 (+ 275us) service_pool.cc:225] Handling call
1025 07:52:43.470678 (+4661135us) ts_tablet_manager.cc:1151] Deleting tablet f4578ce69a0d4322944e5ce7c8a59831
1025 07:52:43.470686 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:43.475059 (+ 4373us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:43.485999 (+ 10940us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:43.510267 (+ 24268us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4660975}
I20251025 07:52:43.510823 2169 tablet_replica.cc:333] T f0bdaa8cfd764f1b83f47881353d26b6 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:43.511061 2169 raft_consensus.cc:2243] T f0bdaa8cfd764f1b83f47881353d26b6 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:43.511463 2169 raft_consensus.cc:2272] T f0bdaa8cfd764f1b83f47881353d26b6 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:43.512180 2169 ts_tablet_manager.cc:1916] T f0bdaa8cfd764f1b83f47881353d26b6 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:43.514463 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet f4578ce69a0d4322944e5ce7c8a59831 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]) successfully deleted
I20251025 07:52:43.515446 894 tablet_service.cc:1505] Processing CreateTablet for tablet d158a18484264ff3938697c87be45ce9 (DEFAULT_TABLE table=default.Table_84b2dff61e654506ab500ed4e4354244 [id=5e17f837bade47e182812916a67acdd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:43.515901 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d158a18484264ff3938697c87be45ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:43.519075 2169 ts_tablet_manager.cc:1929] T f0bdaa8cfd764f1b83f47881353d26b6 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:43.519199 2169 log.cc:1199] T f0bdaa8cfd764f1b83f47881353d26b6 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/f0bdaa8cfd764f1b83f47881353d26b6
I20251025 07:52:43.519594 2169 ts_tablet_manager.cc:1950] T f0bdaa8cfd764f1b83f47881353d26b6 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:43.520430 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 119) took 4710 ms. Trace:
I20251025 07:52:43.520532 2169 rpcz_store.cc:276] 1025 07:52:38.809899 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.810216 (+ 317us) service_pool.cc:225] Handling call
1025 07:52:43.510743 (+4700527us) ts_tablet_manager.cc:1151] Deleting tablet f0bdaa8cfd764f1b83f47881353d26b6
1025 07:52:43.510773 (+ 30us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:43.515918 (+ 5145us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:43.518979 (+ 3061us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:43.520405 (+ 1426us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4700303}
I20251025 07:52:43.521086 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet f0bdaa8cfd764f1b83f47881353d26b6 (table default.Table_49beae553d03426aa395b7cdb6dfec9a [id=ab3328aa6f4148bfa2877cd4f0c6d324]) successfully deleted
I20251025 07:52:43.521275 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 120) took 4710 ms. Trace:
I20251025 07:52:43.521416 2169 rpcz_store.cc:276] 1025 07:52:38.810599 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.811185 (+ 586us) service_pool.cc:225] Handling call
1025 07:52:43.521062 (+4709877us) ts_tablet_manager.cc:1151] Deleting tablet a7ab3150a35b4caca9119a1374d3f2f7
1025 07:52:43.521074 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:43.521253 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4709582}
W20251025 07:52:43.522002 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet a7ab3150a35b4caca9119a1374d3f2f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7ab3150a35b4caca9119a1374d3f2f7 already in progress: creating tablet
I20251025 07:52:43.522154 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 121) took 4710 ms. Trace:
I20251025 07:52:43.522285 2169 rpcz_store.cc:276] 1025 07:52:38.811616 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.811844 (+ 228us) service_pool.cc:225] Handling call
1025 07:52:43.521973 (+4710129us) ts_tablet_manager.cc:1151] Deleting tablet cb824bd780524832b2b60f5618a39da5
1025 07:52:43.521983 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:43.522100 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4709926}
W20251025 07:52:43.522776 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet cb824bd780524832b2b60f5618a39da5 because the tablet was not found. No further retry: Not found: Tablet not found: cb824bd780524832b2b60f5618a39da5
I20251025 07:52:43.526659 1166 tablet_bootstrap.cc:492] T f656945454fe4921ab819c772a089eb3 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:43.526837 1166 ts_tablet_manager.cc:1403] T f656945454fe4921ab819c772a089eb3 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.001s
I20251025 07:52:43.527596 1166 raft_consensus.cc:359] T f656945454fe4921ab819c772a089eb3 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.528119 1018 tablet_service.cc:1505] Processing CreateTablet for tablet d158a18484264ff3938697c87be45ce9 (DEFAULT_TABLE table=default.Table_84b2dff61e654506ab500ed4e4354244 [id=5e17f837bade47e182812916a67acdd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:43.528636 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d158a18484264ff3938697c87be45ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:43.528761 1166 raft_consensus.cc:385] T f656945454fe4921ab819c772a089eb3 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:43.528864 1166 raft_consensus.cc:740] T f656945454fe4921ab819c772a089eb3 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:43.529109 1166 consensus_queue.cc:260] T f656945454fe4921ab819c772a089eb3 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
W20251025 07:52:43.529860 440 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet cb829649f73d4d3f8862d268c953bd86 (table default.Table_a9026f5e095c4063b86d4c544c686e22 [id=7944225d3ce7439b881bfb9289d9f040], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet cb829649f73d4d3f8862d268c953bd86 consensus state has no leader
I20251025 07:52:43.530503 1166 ts_tablet_manager.cc:1434] T f656945454fe4921ab819c772a089eb3 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20251025 07:52:43.532928 2420 raft_consensus.cc:493] T 37c7938e8c5e41f08506aff6747801c9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:43.533071 2420 raft_consensus.cc:515] T 37c7938e8c5e41f08506aff6747801c9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.533656 2420 leader_election.cc:290] T 37c7938e8c5e41f08506aff6747801c9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:43.534555 2169 tablet_replica.cc:333] T 8956ad28b35d45f28c683ac055bba441 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:43.535053 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37c7938e8c5e41f08506aff6747801c9" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:43.535410 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37c7938e8c5e41f08506aff6747801c9" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:43.506366 2146 ts_tablet_manager.cc:1950] T c4610b0c62e741eca28c21f63cc3de81 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:43.535650 1166 tablet_bootstrap.cc:492] T 11bfe0c87ca941a3846462c8a4512d83 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
W20251025 07:52:43.536155 710 leader_election.cc:343] T 37c7938e8c5e41f08506aff6747801c9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:43.536294 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 110) took 5037 ms. Trace:
W20251025 07:52:43.536675 712 leader_election.cc:343] T 37c7938e8c5e41f08506aff6747801c9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:43.536804 712 leader_election.cc:304] T 37c7938e8c5e41f08506aff6747801c9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:43.537297 1166 tablet_bootstrap.cc:654] T 11bfe0c87ca941a3846462c8a4512d83 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:43.541011 2420 raft_consensus.cc:2749] T 37c7938e8c5e41f08506aff6747801c9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:43.536595 2146 rpcz_store.cc:276] 1025 07:52:38.498500 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.499000 (+ 500us) service_pool.cc:225] Handling call
1025 07:52:43.345409 (+4846409us) ts_tablet_manager.cc:1151] Deleting tablet c4610b0c62e741eca28c21f63cc3de81
1025 07:52:43.345420 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:43.478214 (+132794us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:43.504556 (+ 26342us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:43.536277 (+ 31721us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4820182}
I20251025 07:52:43.542510 961 heartbeater.cc:344] Connected to a master server at 127.0.104.61:42203
I20251025 07:52:43.542753 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet c4610b0c62e741eca28c21f63cc3de81 (table default.Table_c9bc68f9b5c54a33b854f9ee8a1ad311 [id=d8f616c04cc54b86af0af3cc073c5e2d]) successfully deleted
I20251025 07:52:43.544124 2169 raft_consensus.cc:2243] T 8956ad28b35d45f28c683ac055bba441 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:43.546013 2146 tablet_replica.cc:333] T c88c46e022ff41a58b02538f2151e260 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:43.546934 1167 tablet_bootstrap.cc:492] T 570201e7bf05406a97c81dde071d894b P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:43.548743 1167 tablet_bootstrap.cc:654] T 570201e7bf05406a97c81dde071d894b P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:43.549372 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36c8fcafd5464b4e999276a8062a812a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:43.549546 769 raft_consensus.cc:2468] T 36c8fcafd5464b4e999276a8062a812a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:43.550560 751 tablet_service.cc:1505] Processing CreateTablet for tablet d158a18484264ff3938697c87be45ce9 (DEFAULT_TABLE table=default.Table_84b2dff61e654506ab500ed4e4354244 [id=5e17f837bade47e182812916a67acdd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:43.552448 894 tablet_service.cc:1505] Processing CreateTablet for tablet 1d95e80ddc7d4cfc85e38e3c10b5883a (DEFAULT_TABLE table=default.Table_22e4370917454a7cb03920884a70a4f1 [id=f70bb4d9ff604674a32738452b00f681]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:43.552580 885 tablet_service.cc:1505] Processing CreateTablet for tablet 14ad6bd1337846fc855820cc7d28a86f (DEFAULT_TABLE table=default.Table_af96c59e9f8646c897e9651a4f457d1c [id=b539292b63ff48de809ce61c4f08c71f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:43.552891 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d95e80ddc7d4cfc85e38e3c10b5883a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:43.553376 753 tablet_service.cc:1505] Processing CreateTablet for tablet 1d95e80ddc7d4cfc85e38e3c10b5883a (DEFAULT_TABLE table=default.Table_22e4370917454a7cb03920884a70a4f1 [id=f70bb4d9ff604674a32738452b00f681]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:43.553929 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14ad6bd1337846fc855820cc7d28a86f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:43.554798 1019 tablet_service.cc:1505] Processing CreateTablet for tablet 14ad6bd1337846fc855820cc7d28a86f (DEFAULT_TABLE table=default.Table_af96c59e9f8646c897e9651a4f457d1c [id=b539292b63ff48de809ce61c4f08c71f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:43.555157 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14ad6bd1337846fc855820cc7d28a86f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:43.556231 2484 consensus_queue.cc:1048] T f43ef51ee45d47a99ec732b49d50d3b4 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:43.557293 847 leader_election.cc:304] T 36c8fcafd5464b4e999276a8062a812a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:43.557844 2105 consensus_queue.cc:1048] T 36262ab8c6964dd684c01b4db6219a4f P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:43.558362 2105 raft_consensus.cc:2804] T 36c8fcafd5464b4e999276a8062a812a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:43.558431 2105 raft_consensus.cc:493] T 36c8fcafd5464b4e999276a8062a812a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:43.558492 2105 raft_consensus.cc:3060] T 36c8fcafd5464b4e999276a8062a812a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:43.561095 2105 raft_consensus.cc:515] T 36c8fcafd5464b4e999276a8062a812a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.561823 1166 tablet_bootstrap.cc:492] T 11bfe0c87ca941a3846462c8a4512d83 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:43.561939 1166 ts_tablet_manager.cc:1403] T 11bfe0c87ca941a3846462c8a4512d83 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.002s
I20251025 07:52:43.562582 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36c8fcafd5464b4e999276a8062a812a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:43.562704 1166 raft_consensus.cc:359] T 11bfe0c87ca941a3846462c8a4512d83 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.562752 1018 tablet_service.cc:1505] Processing CreateTablet for tablet 1d95e80ddc7d4cfc85e38e3c10b5883a (DEFAULT_TABLE table=default.Table_22e4370917454a7cb03920884a70a4f1 [id=f70bb4d9ff604674a32738452b00f681]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:43.562839 1166 raft_consensus.cc:385] T 11bfe0c87ca941a3846462c8a4512d83 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:43.562904 1166 raft_consensus.cc:740] T 11bfe0c87ca941a3846462c8a4512d83 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:43.563108 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d95e80ddc7d4cfc85e38e3c10b5883a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:43.563115 1166 consensus_queue.cc:260] T 11bfe0c87ca941a3846462c8a4512d83 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
W20251025 07:52:43.563179 850 leader_election.cc:343] T 36c8fcafd5464b4e999276a8062a812a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:43.563573 1166 ts_tablet_manager.cc:1434] T 11bfe0c87ca941a3846462c8a4512d83 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251025 07:52:43.565351 2105 leader_election.cc:290] T 36c8fcafd5464b4e999276a8062a812a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:43.567251 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36c8fcafd5464b4e999276a8062a812a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:43.567425 775 raft_consensus.cc:3060] T 36c8fcafd5464b4e999276a8062a812a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:43.577428 2159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Bootstrap replayed 1/1 log segments. Stats: ops{read=667 overwritten=0 applied=664 ignored=0} inserts{seen=617 ignored=0} mutations{seen=756 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20251025 07:52:43.578346 2159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: Bootstrap complete.
I20251025 07:52:43.570123 775 raft_consensus.cc:2468] T 36c8fcafd5464b4e999276a8062a812a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:43.581835 847 leader_election.cc:304] T 36c8fcafd5464b4e999276a8062a812a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:43.586577 2105 raft_consensus.cc:2804] T 36c8fcafd5464b4e999276a8062a812a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:43.586941 2105 raft_consensus.cc:697] T 36c8fcafd5464b4e999276a8062a812a P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:43.587273 2105 consensus_queue.cc:237] T 36c8fcafd5464b4e999276a8062a812a P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.586826 2159 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [term 1 FOLLOWER]: Replica starting. Triggering 3 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } }
I20251025 07:52:43.590204 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d158a18484264ff3938697c87be45ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:43.592911 448 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet 4c155aa7fa5b4d68bbab1918b8432372
I20251025 07:52:43.593588 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 4c155aa7fa5b4d68bbab1918b8432372 with delete_type TABLET_DATA_DELETED (replaced by tablet 110a8e9e4c6c434284c0ae14dd6987f7) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:43.593930 448 catalog_manager.cc:6719] ReplaceTablet: tablet 4c155aa7fa5b4d68bbab1918b8432372 deleted and replaced by tablet 110a8e9e4c6c434284c0ae14dd6987f7
I20251025 07:52:43.594003 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 4c155aa7fa5b4d68bbab1918b8432372 with delete_type TABLET_DATA_DELETED (replaced by tablet 110a8e9e4c6c434284c0ae14dd6987f7) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:43.594044 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d95e80ddc7d4cfc85e38e3c10b5883a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:43.594079 448 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:47610 (request call id 71) took 4805 ms. Trace:
I20251025 07:52:43.594426 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 4c155aa7fa5b4d68bbab1918b8432372 with delete_type TABLET_DATA_DELETED (replaced by tablet 110a8e9e4c6c434284c0ae14dd6987f7) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:43.594204 448 rpcz_store.cc:276] 1025 07:52:38.788402 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:43.465105 (+4676703us) service_pool.cc:225] Handling call
1025 07:52:43.594028 (+128923us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:43.466403 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:43.466516 (+ 113us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:43.466521 (+ 5us) write_op.cc:435] Acquired schema lock
1025 07:52:43.466523 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:43.466620 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:43.466632 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:43.466638 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:43.466713 (+ 75us) tablet.cc:699] Row locks acquired
1025 07:52:43.466716 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:43.466747 (+ 31us) write_op.cc:270] Start()
1025 07:52:43.466779 (+ 32us) write_op.cc:276] Timestamp: P: 1761378763466730 usec, L: 0
1025 07:52:43.466784 (+ 5us) op_driver.cc:348] REPLICATION: starting
1025 07:52:43.466951 (+ 167us) log.cc:844] Serialized 622 byte log entry
1025 07:52:43.472234 (+ 5283us) op_driver.cc:464] REPLICATION: finished
1025 07:52:43.472492 (+ 258us) write_op.cc:301] APPLY: starting
1025 07:52:43.472506 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:43.472549 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:43.472583 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:43.472675 (+ 92us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:43.472740 (+ 65us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:43.472748 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:43.472832 (+ 84us) log.cc:844] Serialized 29 byte log entry
1025 07:52:43.472885 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:43.472903 (+ 18us) write_op.cc:454] Released schema lock
1025 07:52:43.472909 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:43.472971 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:43.472975 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":206,"apply.run_cpu_time_us":473,"apply.run_wall_time_us":473,"num_ops":2,"prepare.queue_time_us":80,"prepare.run_cpu_time_us":654,"prepare.run_wall_time_us":653,"raft.queue_time_us":653,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":236,"replication_time_us":5418}]]}
I20251025 07:52:43.596115 2159 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8603bddd8a084ff7a73a25641768dfa9, State: Initialized, Role: FOLLOWER
I20251025 07:52:43.597097 2159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 664, Last appended: 1.667, Last appended by leader: 667, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } }
I20251025 07:52:43.598548 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 104) took 4945 ms. Trace:
I20251025 07:52:43.598667 2166 rpcz_store.cc:276] 1025 07:52:38.653386 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.653428 (+ 42us) service_pool.cc:225] Handling call
1025 07:52:43.300671 (+4647243us) ts_tablet_manager.cc:1151] Deleting tablet 0b21b7da1e2f47829c3f6cdaee689e8a
1025 07:52:43.300682 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:43.351241 (+ 50559us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:43.455995 (+104754us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:43.598523 (+142528us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4647065}
I20251025 07:52:43.599116 2166 tablet_replica.cc:333] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:43.599191 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 0b21b7da1e2f47829c3f6cdaee689e8a (table default.Table_e61ae9065c604ad8b93af8146748cec9 [id=6d26779bac7243f19d534277dd1c986c]) successfully deleted
I20251025 07:52:43.599352 2166 raft_consensus.cc:2243] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:43.599788 2166 raft_consensus.cc:2272] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:43.600497 2166 ts_tablet_manager.cc:1916] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:43.602319 1169 tablet_bootstrap.cc:492] T 26c849d376d143c6bef0c8e0e1d7d03a P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:43.602456 1169 ts_tablet_manager.cc:1403] T 26c849d376d143c6bef0c8e0e1d7d03a P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.211s user 0.033s sys 0.041s
I20251025 07:52:43.603644 1169 raft_consensus.cc:359] T 26c849d376d143c6bef0c8e0e1d7d03a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.603937 1169 raft_consensus.cc:385] T 26c849d376d143c6bef0c8e0e1d7d03a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:43.604056 1169 raft_consensus.cc:740] T 26c849d376d143c6bef0c8e0e1d7d03a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:43.604355 1169 consensus_queue.cc:260] T 26c849d376d143c6bef0c8e0e1d7d03a P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
W20251025 07:52:43.605270 850 consensus_peers.cc:597] T bc2d16b33ca8426abbd7bd38339c9f1d P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:43.605914 1169 ts_tablet_manager.cc:1434] T 26c849d376d143c6bef0c8e0e1d7d03a P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251025 07:52:43.610973 746 tablet_service.cc:1505] Processing CreateTablet for tablet 14ad6bd1337846fc855820cc7d28a86f (DEFAULT_TABLE table=default.Table_af96c59e9f8646c897e9651a4f457d1c [id=b539292b63ff48de809ce61c4f08c71f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:43.611501 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14ad6bd1337846fc855820cc7d28a86f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:43.611784 885 tablet_service.cc:1505] Processing CreateTablet for tablet daf95cf5c23444f39e64efcdf7e2d893 (DEFAULT_TABLE table=default.Table_c39c116c2e3648fd88670672a25a5481 [id=2491f822f0f04a7a8738a868c330f302]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:43.612161 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daf95cf5c23444f39e64efcdf7e2d893. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:43.610973 752 tablet_service.cc:1505] Processing CreateTablet for tablet dc360d3fc0a34608a56e9117bd322c5b (DEFAULT_TABLE table=default.Table_f29b6c4e31ef483baa4d57209ac4968b [id=dd18259c38f7474d8533f0ad559cc80f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:43.614616 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc360d3fc0a34608a56e9117bd322c5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:43.621335 2508 raft_consensus.cc:493] T c7691c7bdd89437c8db060254c0db25d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:43.621523 2508 raft_consensus.cc:515] T c7691c7bdd89437c8db060254c0db25d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.621965 1167 tablet_bootstrap.cc:492] T 570201e7bf05406a97c81dde071d894b P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:43.618818 1019 tablet_service.cc:1505] Processing CreateTablet for tablet dc360d3fc0a34608a56e9117bd322c5b (DEFAULT_TABLE table=default.Table_f29b6c4e31ef483baa4d57209ac4968b [id=dd18259c38f7474d8533f0ad559cc80f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:43.622341 2508 leader_election.cc:290] T c7691c7bdd89437c8db060254c0db25d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:43.622442 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc360d3fc0a34608a56e9117bd322c5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:43.623176 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7691c7bdd89437c8db060254c0db25d" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:43.623414 1169 tablet_bootstrap.cc:492] T c87ab6a3d08f43e0b2fe58a6d7b11588 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
W20251025 07:52:43.623855 712 leader_election.cc:343] T c7691c7bdd89437c8db060254c0db25d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:43.624522 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7691c7bdd89437c8db060254c0db25d" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:43.625356 710 leader_election.cc:343] T c7691c7bdd89437c8db060254c0db25d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:43.625500 710 leader_election.cc:304] T c7691c7bdd89437c8db060254c0db25d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:43.625566 1169 tablet_bootstrap.cc:654] T c87ab6a3d08f43e0b2fe58a6d7b11588 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:43.625993 2508 raft_consensus.cc:2749] T c7691c7bdd89437c8db060254c0db25d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:43.634222 1048 raft_consensus.cc:3060] T 26c849d376d143c6bef0c8e0e1d7d03a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:43.635135 1169 tablet_bootstrap.cc:492] T c87ab6a3d08f43e0b2fe58a6d7b11588 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:43.635264 1169 ts_tablet_manager.cc:1403] T c87ab6a3d08f43e0b2fe58a6d7b11588 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.003s
I20251025 07:52:43.641091 1169 raft_consensus.cc:359] T c87ab6a3d08f43e0b2fe58a6d7b11588 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.641350 1169 raft_consensus.cc:385] T c87ab6a3d08f43e0b2fe58a6d7b11588 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:43.641496 1169 raft_consensus.cc:740] T c87ab6a3d08f43e0b2fe58a6d7b11588 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:43.641791 1169 consensus_queue.cc:260] T c87ab6a3d08f43e0b2fe58a6d7b11588 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.646245 1169 ts_tablet_manager.cc:1434] T c87ab6a3d08f43e0b2fe58a6d7b11588 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.011s user 0.000s sys 0.003s
I20251025 07:52:43.646754 2106 consensus_queue.cc:1048] T bc2d16b33ca8426abbd7bd38339c9f1d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:43.649808 1169 tablet_bootstrap.cc:492] T bc2d16b33ca8426abbd7bd38339c9f1d P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:43.652194 1167 ts_tablet_manager.cc:1403] T 570201e7bf05406a97c81dde071d894b P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.105s user 0.002s sys 0.002s
I20251025 07:52:43.657147 1169 tablet_bootstrap.cc:654] T bc2d16b33ca8426abbd7bd38339c9f1d P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:43.657584 2106 consensus_queue.cc:1048] T 26c849d376d143c6bef0c8e0e1d7d03a P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:43.668030 741 tablet_service.cc:1505] Processing CreateTablet for tablet daf95cf5c23444f39e64efcdf7e2d893 (DEFAULT_TABLE table=default.Table_c39c116c2e3648fd88670672a25a5481 [id=2491f822f0f04a7a8738a868c330f302]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:43.668251 885 tablet_service.cc:1505] Processing CreateTablet for tablet f69cdaa7ee094283890036c0189e2c81 (DEFAULT_TABLE table=default.Table_d5f74e259f754aa49c24a74e75cdd23c [id=c8b52dd6af1e43f19543fb6ffb2bee98]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:43.668479 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daf95cf5c23444f39e64efcdf7e2d893. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:43.668565 1017 tablet_service.cc:1505] Processing CreateTablet for tablet f69cdaa7ee094283890036c0189e2c81 (DEFAULT_TABLE table=default.Table_d5f74e259f754aa49c24a74e75cdd23c [id=c8b52dd6af1e43f19543fb6ffb2bee98]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:43.668650 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f69cdaa7ee094283890036c0189e2c81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:43.668908 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f69cdaa7ee094283890036c0189e2c81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:43.669538 1167 raft_consensus.cc:359] T 570201e7bf05406a97c81dde071d894b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.669699 1167 raft_consensus.cc:385] T 570201e7bf05406a97c81dde071d894b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:43.669934 1167 raft_consensus.cc:740] T 570201e7bf05406a97c81dde071d894b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:43.670255 1167 consensus_queue.cc:260] T 570201e7bf05406a97c81dde071d894b P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.670718 755 tablet_service.cc:1505] Processing CreateTablet for tablet f69cdaa7ee094283890036c0189e2c81 (DEFAULT_TABLE table=default.Table_d5f74e259f754aa49c24a74e75cdd23c [id=c8b52dd6af1e43f19543fb6ffb2bee98]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:43.671049 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f69cdaa7ee094283890036c0189e2c81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:43.671113 1167 ts_tablet_manager.cc:1434] T 570201e7bf05406a97c81dde071d894b P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20251025 07:52:43.671630 1167 tablet_bootstrap.cc:492] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:43.684286 1169 tablet_bootstrap.cc:492] T bc2d16b33ca8426abbd7bd38339c9f1d P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:43.684537 1169 ts_tablet_manager.cc:1403] T bc2d16b33ca8426abbd7bd38339c9f1d P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.005s
I20251025 07:52:43.685407 1169 raft_consensus.cc:359] T bc2d16b33ca8426abbd7bd38339c9f1d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.685657 1169 raft_consensus.cc:385] T bc2d16b33ca8426abbd7bd38339c9f1d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:43.685755 1169 raft_consensus.cc:740] T bc2d16b33ca8426abbd7bd38339c9f1d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:43.686051 1169 consensus_queue.cc:260] T bc2d16b33ca8426abbd7bd38339c9f1d P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.686345 894 tablet_service.cc:1505] Processing CreateTablet for tablet dc360d3fc0a34608a56e9117bd322c5b (DEFAULT_TABLE table=default.Table_f29b6c4e31ef483baa4d57209ac4968b [id=dd18259c38f7474d8533f0ad559cc80f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:43.686815 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc360d3fc0a34608a56e9117bd322c5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:43.691051 2469 consensus_queue.cc:1048] T 292fe04055954afd8c08ac2636b68191 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251025 07:52:43.694821 983 consensus_peers.cc:597] T 71432a6848944a0da06ef9b81f160dc4 P db42be7ad3c741a0aa427f27e245993b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71432a6848944a0da06ef9b81f160dc4. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:43.696739 1169 ts_tablet_manager.cc:1434] T bc2d16b33ca8426abbd7bd38339c9f1d P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20251025 07:52:43.697283 1169 tablet_bootstrap.cc:492] T 66f0d398e59a4175aa9fe90715124d1a P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:43.697674 1019 tablet_service.cc:1505] Processing CreateTablet for tablet daf95cf5c23444f39e64efcdf7e2d893 (DEFAULT_TABLE table=default.Table_c39c116c2e3648fd88670672a25a5481 [id=2491f822f0f04a7a8738a868c330f302]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:43.698045 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daf95cf5c23444f39e64efcdf7e2d893. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:43.699258 1169 tablet_bootstrap.cc:654] T 66f0d398e59a4175aa9fe90715124d1a P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:43.545830 2169 raft_consensus.cc:2272] T 8956ad28b35d45f28c683ac055bba441 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:43.719712 2169 ts_tablet_manager.cc:1916] T 8956ad28b35d45f28c683ac055bba441 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:43.722791 2469 consensus_queue.cc:1048] T 04449cc97f404a19b6a1b20dea074f95 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:43.726269 1167 tablet_bootstrap.cc:654] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:43.728606 2146 raft_consensus.cc:2243] T c88c46e022ff41a58b02538f2151e260 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:43.728893 2146 raft_consensus.cc:2272] T c88c46e022ff41a58b02538f2151e260 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:43.729281 1166 tablet_bootstrap.cc:492] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:43.731003 1166 tablet_bootstrap.cc:654] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:43.732560 2469 raft_consensus.cc:493] T aaef24ccdfba45fe8204327723df834e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:43.732760 2469 raft_consensus.cc:515] T aaef24ccdfba45fe8204327723df834e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:43.733594 2469 leader_election.cc:290] T aaef24ccdfba45fe8204327723df834e P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:43.734359 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aaef24ccdfba45fe8204327723df834e" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:43.735741 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aaef24ccdfba45fe8204327723df834e" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:43.736387 982 leader_election.cc:304] T aaef24ccdfba45fe8204327723df834e P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:43.736920 2469 raft_consensus.cc:2749] T aaef24ccdfba45fe8204327723df834e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:43.738193 2314 raft_consensus.cc:493] T 5361f08f50b745eb98bf02e979d7c71e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:43.738348 2314 raft_consensus.cc:515] T 5361f08f50b745eb98bf02e979d7c71e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:43.740029 850 leader_election.cc:343] T 5361f08f50b745eb98bf02e979d7c71e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 5361f08f50b745eb98bf02e979d7c71e
I20251025 07:52:43.740206 2314 leader_election.cc:290] T 5361f08f50b745eb98bf02e979d7c71e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:43.740585 453 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 69) took 4992 ms. Trace:
I20251025 07:52:43.740780 453 rpcz_store.cc:276] 1025 07:52:38.747626 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:43.438777 (+4691151us) service_pool.cc:225] Handling call
1025 07:52:43.439948 (+ 1171us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:43.440416 (+ 468us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:43.631025 (+190609us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:43.740403 (+109378us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:43.740540 (+ 137us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:43.494456 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:43.494617 (+ 161us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:43.494626 (+ 9us) write_op.cc:435] Acquired schema lock
1025 07:52:43.494630 (+ 4us) tablet.cc:660] Decoding operations
1025 07:52:43.494740 (+ 110us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:43.494754 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:43.494761 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:43.494881 (+ 120us) tablet.cc:699] Row locks acquired
1025 07:52:43.494886 (+ 5us) write_op.cc:260] PREPARE: finished
1025 07:52:43.494936 (+ 50us) write_op.cc:270] Start()
1025 07:52:43.494996 (+ 60us) write_op.cc:276] Timestamp: P: 1761378763494915 usec, L: 0
1025 07:52:43.495002 (+ 6us) op_driver.cc:348] REPLICATION: starting
1025 07:52:43.613074 (+118072us) log.cc:844] Serialized 433 byte log entry
1025 07:52:43.613368 (+ 294us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x616000041254. stack: 00007fb90f7779f6 00007fb90f7777b0 00007fb90e7533ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fb91b612862 00007fb91c9f52a5 00007fb91c9f3dc9 00007fb91c9fc1f0 00007fb91c9fbffd 0000000000a8867c 00007fb90f7bdd7f 00007fb90f7c104b 00007fb90f7c0ec8 0000000000a8867c
1025 07:52:43.629997 (+ 16629us) op_driver.cc:464] REPLICATION: finished
1025 07:52:43.630104 (+ 107us) write_op.cc:301] APPLY: starting
1025 07:52:43.630123 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:43.630209 (+ 86us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:43.630211 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:43.630426 (+ 215us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:43.630445 (+ 19us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:43.630453 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:43.630546 (+ 93us) log.cc:844] Serialized 29 byte log entry
1025 07:52:43.630907 (+ 361us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:43.630924 (+ 17us) write_op.cc:454] Released schema lock
1025 07:52:43.630935 (+ 11us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:43.631199 (+ 264us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:43.631204 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":29,"apply.run_cpu_time_us":638,"apply.run_wall_time_us":1062,"mutex_wait_us":53463,"num_ops":2,"prepare.queue_time_us":53533,"prepare.run_cpu_time_us":2361,"prepare.run_wall_time_us":118988,"raft.queue_time_us":1211,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":206,"replication_time_us":134957,"spinlock_wait_cycles":271111552}]]}
I20251025 07:52:43.742344 453 master_service.cc:954] ReplaceTablet: received request to replace tablet fce535a0c0bd45c6bf17f81add297f40 from {username='slave'} at 127.0.0.1:47610
I20251025 07:52:43.747284 2597 raft_consensus.cc:493] T bc2d16b33ca8426abbd7bd38339c9f1d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:43.747435 2597 raft_consensus.cc:515] T bc2d16b33ca8426abbd7bd38339c9f1d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.747548 2484 consensus_queue.cc:1048] T 292fe04055954afd8c08ac2636b68191 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:43.748008 2597 leader_election.cc:290] T bc2d16b33ca8426abbd7bd38339c9f1d P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:43.748811 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc2d16b33ca8426abbd7bd38339c9f1d" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:43.751091 2166 ts_tablet_manager.cc:1929] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:43.754899 2146 ts_tablet_manager.cc:1916] T c88c46e022ff41a58b02538f2151e260 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:43.772428 453 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet fce535a0c0bd45c6bf17f81add297f40
I20251025 07:52:43.773504 885 tablet_service.cc:1552] Processing DeleteTablet for tablet fce535a0c0bd45c6bf17f81add297f40 with delete_type TABLET_DATA_DELETED (replaced by tablet 079f91be4217493d8ab865f5ccce0cb1) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:43.774021 453 catalog_manager.cc:6719] ReplaceTablet: tablet fce535a0c0bd45c6bf17f81add297f40 deleted and replaced by tablet 079f91be4217493d8ab865f5ccce0cb1
I20251025 07:52:43.774227 453 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:47610 (request call id 72) took 4629 ms. Trace:
I20251025 07:52:43.774502 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet fce535a0c0bd45c6bf17f81add297f40 with delete_type TABLET_DATA_DELETED (replaced by tablet 079f91be4217493d8ab865f5ccce0cb1) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:43.774353 453 rpcz_store.cc:276] 1025 07:52:39.144516 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:43.742259 (+4597743us) service_pool.cc:225] Handling call
1025 07:52:43.759316 (+ 17057us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:43.759325 (+ 9us) write_op.cc:454] Released schema lock
1025 07:52:43.774171 (+ 14846us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:43.744216 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:43.744392 (+ 176us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:43.744401 (+ 9us) write_op.cc:435] Acquired schema lock
1025 07:52:43.744403 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:43.744518 (+ 115us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:43.744533 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:43.744538 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:43.744615 (+ 77us) tablet.cc:699] Row locks acquired
1025 07:52:43.744618 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:43.744655 (+ 37us) write_op.cc:270] Start()
1025 07:52:43.744689 (+ 34us) write_op.cc:276] Timestamp: P: 1761378763744636 usec, L: 0
1025 07:52:43.744693 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:43.744922 (+ 229us) log.cc:844] Serialized 622 byte log entry
1025 07:52:43.748645 (+ 3723us) op_driver.cc:464] REPLICATION: finished
1025 07:52:43.749664 (+ 1019us) write_op.cc:301] APPLY: starting
1025 07:52:43.749711 (+ 47us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:43.749761 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:43.749788 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:43.749902 (+ 114us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:43.749979 (+ 77us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:43.749987 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:43.750066 (+ 79us) log.cc:844] Serialized 29 byte log entry
1025 07:52:43.750784 (+ 718us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:43.750803 (+ 19us) write_op.cc:454] Released schema lock
1025 07:52:43.750810 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":954,"apply.run_cpu_time_us":538,"apply.run_wall_time_us":1201,"num_ops":2,"prepare.queue_time_us":157,"prepare.run_cpu_time_us":841,"prepare.run_wall_time_us":1245,"raft.queue_time_us":84,"raft.run_cpu_time_us":387,"raft.run_wall_time_us":4374,"replication_time_us":3922}]]}
I20251025 07:52:43.775596 453 master_service.cc:954] ReplaceTablet: received request to replace tablet 32a143a715cb485cb4501f4e28c68252 from {username='slave'} at 127.0.0.1:47610
I20251025 07:52:43.788391 1049 raft_consensus.cc:3060] T 4c155aa7fa5b4d68bbab1918b8432372 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:43.823071 2484 consensus_queue.cc:1048] T 04449cc97f404a19b6a1b20dea074f95 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:43.752264 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc2d16b33ca8426abbd7bd38339c9f1d" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:43.774293 1167 tablet_bootstrap.cc:492] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:43.827997 1167 ts_tablet_manager.cc:1403] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.156s user 0.002s sys 0.003s
I20251025 07:52:43.828321 982 leader_election.cc:304] T bc2d16b33ca8426abbd7bd38339c9f1d P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:43.828656 2484 raft_consensus.cc:2749] T bc2d16b33ca8426abbd7bd38339c9f1d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:43.828706 1167 raft_consensus.cc:359] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:43.828884 1167 raft_consensus.cc:385] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:43.828936 1167 raft_consensus.cc:740] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:43.829149 1167 consensus_queue.cc:260] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:43.829654 1167 ts_tablet_manager.cc:1434] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251025 07:52:43.829921 1167 tablet_bootstrap.cc:492] T 716fb61312e94a4d9eb8ab8336ca140c P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:43.774930 746 tablet_service.cc:1552] Processing DeleteTablet for tablet fce535a0c0bd45c6bf17f81add297f40 with delete_type TABLET_DATA_DELETED (replaced by tablet 079f91be4217493d8ab865f5ccce0cb1) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:43.783277 2420 raft_consensus.cc:493] T 5b6383c99e9e4c90a374da9ccd830a6f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:43.830993 2420 raft_consensus.cc:515] T 5b6383c99e9e4c90a374da9ccd830a6f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.831851 1167 tablet_bootstrap.cc:654] T 716fb61312e94a4d9eb8ab8336ca140c P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:43.831866 2420 leader_election.cc:290] T 5b6383c99e9e4c90a374da9ccd830a6f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:43.832808 2166 log.cc:1199] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/f3001314b56345f5a70314777cae0125
I20251025 07:52:43.832775 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b6383c99e9e4c90a374da9ccd830a6f" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:43.833288 712 leader_election.cc:343] T 5b6383c99e9e4c90a374da9ccd830a6f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:43.833665 2166 ts_tablet_manager.cc:1950] T f3001314b56345f5a70314777cae0125 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:43.834894 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 105) took 5180 ms. Trace:
I20251025 07:52:43.835031 2166 rpcz_store.cc:276] 1025 07:52:38.654694 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.654738 (+ 44us) service_pool.cc:225] Handling call
1025 07:52:43.599038 (+4944300us) ts_tablet_manager.cc:1151] Deleting tablet f3001314b56345f5a70314777cae0125
1025 07:52:43.599050 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:43.650712 (+ 51662us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:43.750945 (+100233us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:43.834872 (+ 83927us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4944139}
I20251025 07:52:43.835616 2166 tablet_replica.cc:333] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:43.835587 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet f3001314b56345f5a70314777cae0125 (table default.Table_d771d85f5f6a4016bfa426dd9dbf586a [id=0a62837e6dda43879cc075d289326c5c]) successfully deleted
I20251025 07:52:43.835896 2166 raft_consensus.cc:2243] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:43.843189 2105 raft_consensus.cc:493] T b2044ca206634eb285197af4ba2073e7 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:43.843428 2105 raft_consensus.cc:515] T b2044ca206634eb285197af4ba2073e7 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.844456 2105 leader_election.cc:290] T b2044ca206634eb285197af4ba2073e7 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:43.844961 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5]). Expected version 1 got 0
I20251025 07:52:43.845057 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2044ca206634eb285197af4ba2073e7" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:43.845237 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3]). Expected version 1 got 0
I20251025 07:52:43.845499 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_a6c869949b744bc98efb735b0e73e7aa [id=bf77b5ff6f1048dd996d948437f9ff5e]). Expected version 2 got 0
W20251025 07:52:43.845543 850 leader_election.cc:343] T b2044ca206634eb285197af4ba2073e7 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:43.846827 2469 consensus_queue.cc:1048] T b3a0af893a8947848e12093d0a7f0229 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 07:52:43.850262 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f]). Expected version 1 got 0
I20251025 07:52:43.850728 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7]). Expected version 1 got 0
I20251025 07:52:43.851006 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 872cb7512d104827b1f1f186d33a4a44 (table default.Table_d5c65dce8b1b4214ba519d964012bfc0 [id=d4be274b36a845a99a4b96b8846248fd]). Expected version 1 got 0
I20251025 07:52:43.851179 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet cb829649f73d4d3f8862d268c953bd86 (table default.Table_a9026f5e095c4063b86d4c544c686e22 [id=7944225d3ce7439b881bfb9289d9f040]). Expected version 2 got 0
I20251025 07:52:43.851377 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 04d27d54fdb7483b869408af65ff85c9 (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]). Expected version 2 got 0
I20251025 07:52:43.852610 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet aa9d9ab8073945d28d52fb02213f201a (table default.Table_4791672658f043f6b97b9320a531ebb5 [id=dfd20cf9c3e541c4abe23d59e0aa7f2a]). Expected version 1 got 0
I20251025 07:52:43.853281 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 292fe04055954afd8c08ac2636b68191 (table default.Table_9fa69b10b5e842639fc0b0a58cd9b6a5 [id=a253ce0253bd4da4814741c482193d8f]). Expected version 1 got 0
I20251025 07:52:43.853576 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]). Expected version 1 got 0
I20251025 07:52:43.853588 2469 raft_consensus.cc:493] T c87ab6a3d08f43e0b2fe58a6d7b11588 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:43.853790 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet ef9a2c9f2ccf432ab835396020872208 (table default.Table_2b9b772d4c234ac286bef3658482e490 [id=409fab60f9d744798e6f92b868f0f7c2]). Expected version 1 got 0
I20251025 07:52:43.853834 2469 raft_consensus.cc:515] T c87ab6a3d08f43e0b2fe58a6d7b11588 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.854014 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]). Expected version 1 got 0
I20251025 07:52:43.855274 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c87ab6a3d08f43e0b2fe58a6d7b11588" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:43.857952 2469 leader_election.cc:290] T c87ab6a3d08f43e0b2fe58a6d7b11588 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
W20251025 07:52:43.859592 982 consensus_peers.cc:597] T b3a0af893a8947848e12093d0a7f0229 P db42be7ad3c741a0aa427f27e245993b -> Peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Couldn't send request to peer 23eeffd3de254ce18db5febfde971c6b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b3a0af893a8947848e12093d0a7f0229. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:43.863921 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b6383c99e9e4c90a374da9ccd830a6f" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:43.864218 850 consensus_peers.cc:597] T c9f7ba3709c745d4a625449d6e1c8ba1 P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:43.866820 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 5dab439bc50d4e869548154e1d222ad6 (table default.Table_fb9def2302784c999bf0f5707c60eb4c [id=45325c188eac4d34bb016d01de091f87]). Expected version 3 got 0
I20251025 07:52:43.867159 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_87d38832f85e4965bfab1a077990114f [id=ad812862698d421184ebf501ce79b7be]). Expected version 2 got 0
I20251025 07:52:43.867400 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet f70091024c034c118fb86578db19a31c (table default.Table_b7da40b96d6c4359bed6f898e8cf6275 [id=e041946c093b4c789ce175c736cc230b]). Expected version 2 got 0
I20251025 07:52:43.868111 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 5fc092c2bb2d41ac8c5b4fc07460c8ec (table default.Table_27c62e973d74406fb00db8bcf324749c [id=630c274339664b9db18ab4ef7c29c16f]). Expected version 1 got 0
I20251025 07:52:43.868341 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]). Expected version 3 got 0
I20251025 07:52:43.868597 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 22f7db660616457b83b24e947a0b0bae (table default.Table_4a86191163f2408f9bcc795cd8d1bbf6 [id=bb316a7e000748ca902a9bfb494802d7]). Expected version 1 got 0
I20251025 07:52:43.868798 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 755f0d6c87044e20a8020f8410940123 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 4 got 0
I20251025 07:52:43.869103 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236]). Expected version 1 got 0
I20251025 07:52:43.869194 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2044ca206634eb285197af4ba2073e7" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:43.869398 773 raft_consensus.cc:2468] T b2044ca206634eb285197af4ba2073e7 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:43.872026 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c87ab6a3d08f43e0b2fe58a6d7b11588" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:43.869669 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84]). Expected version 1 got 0
I20251025 07:52:43.873320 772 raft_consensus.cc:2393] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:43.873613 1166 tablet_bootstrap.cc:492] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:43.873647 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 0325d462441349ef81841f1b0cab2a20 (table default.Table_750ecb905839434fa9a1aa72f87394be [id=9ce3ba77c56a4ef09cb3b40ea0e488f6]). Expected version 1 got 0
I20251025 07:52:43.873805 1166 ts_tablet_manager.cc:1403] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.145s user 0.022s sys 0.051s
I20251025 07:52:43.874609 1166 raft_consensus.cc:359] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.874794 1166 raft_consensus.cc:385] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:43.874863 1166 raft_consensus.cc:740] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:43.874903 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]). Expected version 1 got 0
I20251025 07:52:43.875116 1166 consensus_queue.cc:260] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.875317 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525]). Expected version 1 got 0
I20251025 07:52:43.875470 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_e90d7f5ff13b40878e5b625c97792c7d [id=fc8180c09d7b4e74b2e161f7acb8fe0e]). Expected version 1 got 0
I20251025 07:52:43.875644 1166 ts_tablet_manager.cc:1434] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:43.875708 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet b56209e4934e4407a8b4518ca4af0d46 (table default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de]). Expected version 1 got 0
I20251025 07:52:43.875862 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 830fdbc6812e416690d0db42a73af503 (table default.Table_92824b3f2a774564b3aea04aaa25d204 [id=fb21a45b86ab446e8c3dae4512a9598f]). Expected version 1 got 0
I20251025 07:52:43.875979 1166 tablet_bootstrap.cc:492] T ebc824ec4c60411dab1a2ea4a385148e P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:43.878068 1166 tablet_bootstrap.cc:654] T ebc824ec4c60411dab1a2ea4a385148e P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:43.880714 982 leader_election.cc:304] T c87ab6a3d08f43e0b2fe58a6d7b11588 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:43.881348 2469 raft_consensus.cc:3060] T c87ab6a3d08f43e0b2fe58a6d7b11588 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:43.883127 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c]). Expected version 1 got 0
I20251025 07:52:43.884876 2469 raft_consensus.cc:2749] T c87ab6a3d08f43e0b2fe58a6d7b11588 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:43.885030 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 36262ab8c6964dd684c01b4db6219a4f (table default.Table_76158f8cf82e4cdb9a90580e46aaea48 [id=f249f30497bc450c887d659721469de8]). Expected version 1 got 0
I20251025 07:52:43.886287 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3]). Expected version 2 got 0
I20251025 07:52:43.886832 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 288e543bb279410c91c01dd1623e110f (table default.Table_6e242ba55b194c23a6701f4b064fdbe5 [id=4c34b64b299f4162a53866cf01c5298f]). Expected version 1 got 0
I20251025 07:52:43.886822 2420 consensus_queue.cc:1048] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:43.887283 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet ab42909afc524f2fa3c1041495d13a59 (table default.Table_7eb8424d5192485da1a2d67bfdb2bbbf [id=40c29df99590408cbd5c583d4fc741eb]). Expected version 2 got 0
I20251025 07:52:43.887667 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet bc2d16b33ca8426abbd7bd38339c9f1d (table default.Table_4ea985df4c5b415c9570c64506a98cc4 [id=98d2b7797e33485e8872054aa2eb8ec9]). Expected version 1 got 0
I20251025 07:52:43.887881 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e]). Expected version 1 got 0
W20251025 07:52:43.888217 710 leader_election.cc:343] T 5b6383c99e9e4c90a374da9ccd830a6f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:43.888341 710 leader_election.cc:304] T 5b6383c99e9e4c90a374da9ccd830a6f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:43.888758 2420 raft_consensus.cc:2749] T 5b6383c99e9e4c90a374da9ccd830a6f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:43.890695 1167 tablet_bootstrap.cc:492] T 716fb61312e94a4d9eb8ab8336ca140c P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:43.890868 1167 ts_tablet_manager.cc:1403] T 716fb61312e94a4d9eb8ab8336ca140c P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.061s user 0.007s sys 0.000s
I20251025 07:52:43.891642 1167 raft_consensus.cc:359] T 716fb61312e94a4d9eb8ab8336ca140c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.891820 1167 raft_consensus.cc:385] T 716fb61312e94a4d9eb8ab8336ca140c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:43.891909 1167 raft_consensus.cc:740] T 716fb61312e94a4d9eb8ab8336ca140c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
W20251025 07:52:43.892015 710 consensus_peers.cc:597] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4578ce69a0d4322944e5ce7c8a59831. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:43.892438 2105 raft_consensus.cc:493] T ab42909afc524f2fa3c1041495d13a59 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:43.892627 2106 raft_consensus.cc:493] T 6dcab53a020847a1aa27930265ba0205 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:43.892592 2105 raft_consensus.cc:515] T ab42909afc524f2fa3c1041495d13a59 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.892753 2106 raft_consensus.cc:515] T 6dcab53a020847a1aa27930265ba0205 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.893236 2105 leader_election.cc:290] T ab42909afc524f2fa3c1041495d13a59 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:43.893329 2106 leader_election.cc:290] T 6dcab53a020847a1aa27930265ba0205 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:43.892091 1167 consensus_queue.cc:260] T 716fb61312e94a4d9eb8ab8336ca140c P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.893893 1167 ts_tablet_manager.cc:1434] T 716fb61312e94a4d9eb8ab8336ca140c P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251025 07:52:43.893993 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ab42909afc524f2fa3c1041495d13a59" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:43.894385 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dcab53a020847a1aa27930265ba0205" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:43.895001 850 leader_election.cc:343] T ab42909afc524f2fa3c1041495d13a59 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:43.895228 850 leader_election.cc:343] T 6dcab53a020847a1aa27930265ba0205 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:43.895731 1167 tablet_bootstrap.cc:492] T 7e6544202d4f489d82f22d6073ef240d P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:43.902393 1167 tablet_bootstrap.cc:654] T 7e6544202d4f489d82f22d6073ef240d P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:43.906944 1169 tablet_bootstrap.cc:492] T 66f0d398e59a4175aa9fe90715124d1a P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:43.907102 1169 ts_tablet_manager.cc:1403] T 66f0d398e59a4175aa9fe90715124d1a P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.210s user 0.019s sys 0.020s
W20251025 07:52:43.907243 712 consensus_peers.cc:597] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4578ce69a0d4322944e5ce7c8a59831. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:43.907859 1169 raft_consensus.cc:359] T 66f0d398e59a4175aa9fe90715124d1a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.908067 1169 raft_consensus.cc:385] T 66f0d398e59a4175aa9fe90715124d1a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:43.908139 1169 raft_consensus.cc:740] T 66f0d398e59a4175aa9fe90715124d1a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:43.908336 1169 consensus_queue.cc:260] T 66f0d398e59a4175aa9fe90715124d1a P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.908807 1169 ts_tablet_manager.cc:1434] T 66f0d398e59a4175aa9fe90715124d1a P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 07:52:43.909257 1169 tablet_bootstrap.cc:492] T 09898735796541b0a8f3be22cdb6cc30 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:43.909838 453 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet 32a143a715cb485cb4501f4e28c68252
I20251025 07:52:43.909955 1166 tablet_bootstrap.cc:492] T ebc824ec4c60411dab1a2ea4a385148e P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:43.910120 1166 ts_tablet_manager.cc:1403] T ebc824ec4c60411dab1a2ea4a385148e P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.002s
I20251025 07:52:43.910367 453 catalog_manager.cc:6719] ReplaceTablet: tablet 32a143a715cb485cb4501f4e28c68252 deleted and replaced by tablet 9c389d798cb94d8a972683c57b1f9f1f
I20251025 07:52:43.910480 453 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:47610 (request call id 73) took 4468 ms. Trace:
I20251025 07:52:43.910965 1169 tablet_bootstrap.cc:654] T 09898735796541b0a8f3be22cdb6cc30 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:43.910907 1166 raft_consensus.cc:359] T ebc824ec4c60411dab1a2ea4a385148e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.910591 453 rpcz_store.cc:276] 1025 07:52:39.442158 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:43.775512 (+4333354us) service_pool.cc:225] Handling call
1025 07:52:43.910449 (+134937us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:43.902627 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:43.902762 (+ 135us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:43.902769 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:43.902772 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:43.902896 (+ 124us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:43.902912 (+ 16us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:43.902917 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:43.903001 (+ 84us) tablet.cc:699] Row locks acquired
1025 07:52:43.903004 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:43.903046 (+ 42us) write_op.cc:270] Start()
1025 07:52:43.903082 (+ 36us) write_op.cc:276] Timestamp: P: 1761378763903019 usec, L: 0
1025 07:52:43.903086 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:43.903278 (+ 192us) log.cc:844] Serialized 622 byte log entry
1025 07:52:43.906681 (+ 3403us) op_driver.cc:464] REPLICATION: finished
1025 07:52:43.907053 (+ 372us) write_op.cc:301] APPLY: starting
1025 07:52:43.907071 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:43.907118 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:43.907146 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:43.907260 (+ 114us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:43.907328 (+ 68us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:43.907336 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:43.907414 (+ 78us) log.cc:844] Serialized 29 byte log entry
1025 07:52:43.907475 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:43.907497 (+ 22us) write_op.cc:454] Released schema lock
1025 07:52:43.907505 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:43.907574 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:43.907578 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":308,"apply.run_cpu_time_us":518,"apply.run_wall_time_us":518,"num_ops":2,"prepare.queue_time_us":59,"prepare.run_cpu_time_us":764,"prepare.run_wall_time_us":766,"raft.queue_time_us":69,"raft.run_cpu_time_us":295,"raft.run_wall_time_us":295,"replication_time_us":3564}]]}
I20251025 07:52:43.911089 1166 raft_consensus.cc:385] T ebc824ec4c60411dab1a2ea4a385148e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:43.911195 1166 raft_consensus.cc:740] T ebc824ec4c60411dab1a2ea4a385148e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:43.911207 453 master_service.cc:954] ReplaceTablet: received request to replace tablet 48ff4ce7846b4e56a37eee9bcb17bd5e from {username='slave'} at 127.0.0.1:47610
I20251025 07:52:43.911914 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 32a143a715cb485cb4501f4e28c68252 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c389d798cb94d8a972683c57b1f9f1f) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:43.912570 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 32a143a715cb485cb4501f4e28c68252 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c389d798cb94d8a972683c57b1f9f1f) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:43.914187 453 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:47610 (request call id 74) took 4379 ms. Trace:
I20251025 07:52:43.914299 453 rpcz_store.cc:276] 1025 07:52:39.534493 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:43.911149 (+4376656us) service_pool.cc:225] Handling call
1025 07:52:43.914115 (+ 2966us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 07:52:43.914430 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 32a143a715cb485cb4501f4e28c68252 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c389d798cb94d8a972683c57b1f9f1f) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:43.914590 453 master_service.cc:954] ReplaceTablet: received request to replace tablet 0540d6a10cf04fa89756623e9aaaeb68 from {username='slave'} at 127.0.0.1:47610
I20251025 07:52:43.915185 2508 raft_consensus.cc:493] T 570201e7bf05406a97c81dde071d894b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:43.915328 2508 raft_consensus.cc:515] T 570201e7bf05406a97c81dde071d894b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.915385 2602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } } }
I20251025 07:52:43.916134 2508 leader_election.cc:290] T 570201e7bf05406a97c81dde071d894b P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:43.916859 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "570201e7bf05406a97c81dde071d894b" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:43.917367 712 leader_election.cc:343] T 570201e7bf05406a97c81dde071d894b P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:43.922255 2602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 07:52:43.918702 2159 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [sys.catalog]: configured and running, proceeding with master startup.
I20251025 07:52:43.922816 2090 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [term 1 FOLLOWER]: Advancing to term 2
I20251025 07:52:43.924129 1166 consensus_queue.cc:260] T ebc824ec4c60411dab1a2ea4a385148e P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.924741 1166 ts_tablet_manager.cc:1434] T ebc824ec4c60411dab1a2ea4a385148e P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20251025 07:52:43.925123 1166 tablet_bootstrap.cc:492] T 4678f7eab5e8497db308e8a38e387618 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
W20251025 07:52:43.927419 712 consensus_peers.cc:597] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:43.927587 2508 raft_consensus.cc:493] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:43.927699 2508 raft_consensus.cc:515] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:43.928277 2508 leader_election.cc:290] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:43.928927 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f0c4f0e8f014e59be9e46d2afe0be9d" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:43.929764 712 leader_election.cc:343] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:43.931582 1166 tablet_bootstrap.cc:654] T 4678f7eab5e8497db308e8a38e387618 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:43.932425 1167 tablet_bootstrap.cc:492] T 7e6544202d4f489d82f22d6073ef240d P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:43.932626 1167 ts_tablet_manager.cc:1403] T 7e6544202d4f489d82f22d6073ef240d P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.000s
I20251025 07:52:43.933993 1167 raft_consensus.cc:359] T 7e6544202d4f489d82f22d6073ef240d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.934324 1167 raft_consensus.cc:385] T 7e6544202d4f489d82f22d6073ef240d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:43.934438 1167 raft_consensus.cc:740] T 7e6544202d4f489d82f22d6073ef240d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:43.934707 1167 consensus_queue.cc:260] T 7e6544202d4f489d82f22d6073ef240d P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.935290 1167 ts_tablet_manager.cc:1434] T 7e6544202d4f489d82f22d6073ef240d P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251025 07:52:43.935665 1167 tablet_bootstrap.cc:492] T 1b5ac7e845a24b2eb5b2429d47361232 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:43.936110 2166 raft_consensus.cc:2272] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:43.936761 2166 ts_tablet_manager.cc:1916] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:43.937012 1169 tablet_bootstrap.cc:492] T 09898735796541b0a8f3be22cdb6cc30 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:43.937194 1169 ts_tablet_manager.cc:1403] T 09898735796541b0a8f3be22cdb6cc30 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.003s
I20251025 07:52:43.937996 1169 raft_consensus.cc:359] T 09898735796541b0a8f3be22cdb6cc30 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.938210 1169 raft_consensus.cc:385] T 09898735796541b0a8f3be22cdb6cc30 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:43.938315 1169 raft_consensus.cc:740] T 09898735796541b0a8f3be22cdb6cc30 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:43.938546 1169 consensus_queue.cc:260] T 09898735796541b0a8f3be22cdb6cc30 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.938688 1167 tablet_bootstrap.cc:654] T 1b5ac7e845a24b2eb5b2429d47361232 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:43.939191 1169 ts_tablet_manager.cc:1434] T 09898735796541b0a8f3be22cdb6cc30 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251025 07:52:43.941815 1169 tablet_bootstrap.cc:492] T 30418858bf95438686bdfe64ac733144 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:43.935323 2314 raft_consensus.cc:493] T 09898735796541b0a8f3be22cdb6cc30 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:43.942405 2314 raft_consensus.cc:515] T 09898735796541b0a8f3be22cdb6cc30 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.943132 2314 leader_election.cc:290] T 09898735796541b0a8f3be22cdb6cc30 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:43.938964 2106 consensus_queue.cc:1048] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:43.944182 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "09898735796541b0a8f3be22cdb6cc30" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:43.944361 1048 raft_consensus.cc:2468] T 09898735796541b0a8f3be22cdb6cc30 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:43.944809 850 leader_election.cc:304] T 09898735796541b0a8f3be22cdb6cc30 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:43.945124 2106 raft_consensus.cc:2804] T 09898735796541b0a8f3be22cdb6cc30 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:43.945206 2106 raft_consensus.cc:493] T 09898735796541b0a8f3be22cdb6cc30 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:43.945310 2106 raft_consensus.cc:3060] T 09898735796541b0a8f3be22cdb6cc30 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:43.945618 1169 tablet_bootstrap.cc:654] T 30418858bf95438686bdfe64ac733144 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:43.949069 2106 raft_consensus.cc:515] T 09898735796541b0a8f3be22cdb6cc30 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.958379 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "09898735796541b0a8f3be22cdb6cc30" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:43.958578 1048 raft_consensus.cc:3060] T 09898735796541b0a8f3be22cdb6cc30 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:43.961421 1048 raft_consensus.cc:2468] T 09898735796541b0a8f3be22cdb6cc30 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:43.961949 850 leader_election.cc:304] T 09898735796541b0a8f3be22cdb6cc30 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:43.962381 2314 raft_consensus.cc:2804] T 09898735796541b0a8f3be22cdb6cc30 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:43.965452 453 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet 0540d6a10cf04fa89756623e9aaaeb68
I20251025 07:52:43.966073 453 catalog_manager.cc:6719] ReplaceTablet: tablet 0540d6a10cf04fa89756623e9aaaeb68 deleted and replaced by tablet 55a4a524676b4b9aa9464e934b662f30
I20251025 07:52:43.966223 453 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:47610 (request call id 75) took 4389 ms. Trace:
I20251025 07:52:43.966325 453 rpcz_store.cc:276] 1025 07:52:39.577273 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:43.914508 (+4337235us) service_pool.cc:225] Handling call
1025 07:52:43.966192 (+ 51684us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:43.958321 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:43.958502 (+ 181us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:43.958512 (+ 10us) write_op.cc:435] Acquired schema lock
1025 07:52:43.958516 (+ 4us) tablet.cc:660] Decoding operations
1025 07:52:43.958683 (+ 167us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:43.958702 (+ 19us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:43.958710 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:43.958817 (+ 107us) tablet.cc:699] Row locks acquired
1025 07:52:43.958821 (+ 4us) write_op.cc:260] PREPARE: finished
1025 07:52:43.958867 (+ 46us) write_op.cc:270] Start()
1025 07:52:43.958911 (+ 44us) write_op.cc:276] Timestamp: P: 1761378763958839 usec, L: 0
1025 07:52:43.958917 (+ 6us) op_driver.cc:348] REPLICATION: starting
1025 07:52:43.959197 (+ 280us) log.cc:844] Serialized 622 byte log entry
1025 07:52:43.963567 (+ 4370us) op_driver.cc:464] REPLICATION: finished
1025 07:52:43.963739 (+ 172us) write_op.cc:301] APPLY: starting
1025 07:52:43.963761 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:43.963823 (+ 62us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:43.963857 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:43.964003 (+ 146us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:43.964087 (+ 84us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:43.964099 (+ 12us) write_op.cc:312] APPLY: finished
1025 07:52:43.964200 (+ 101us) log.cc:844] Serialized 29 byte log entry
1025 07:52:43.964274 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:43.964303 (+ 29us) write_op.cc:454] Released schema lock
1025 07:52:43.964312 (+ 9us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:43.964397 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:43.964403 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":69,"apply.run_cpu_time_us":648,"apply.run_wall_time_us":647,"num_ops":2,"prepare.queue_time_us":4531,"prepare.run_cpu_time_us":1036,"prepare.run_wall_time_us":1036,"raft.queue_time_us":98,"raft.run_cpu_time_us":286,"raft.run_wall_time_us":287,"replication_time_us":4611}]]}
I20251025 07:52:43.967288 453 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_173708dfd0274b34a7db6fa2b239a8b0" } modify_external_catalogs: true
I20251025 07:52:43.967515 453 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_173708dfd0274b34a7db6fa2b239a8b0" } modify_external_catalogs: true
I20251025 07:52:43.968500 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 0540d6a10cf04fa89756623e9aaaeb68 with delete_type TABLET_DATA_DELETED (replaced by tablet 55a4a524676b4b9aa9464e934b662f30) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:43.968994 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 0540d6a10cf04fa89756623e9aaaeb68 with delete_type TABLET_DATA_DELETED (replaced by tablet 55a4a524676b4b9aa9464e934b662f30) from {username='slave'} at 127.0.0.1:58412
W20251025 07:52:43.939508 847 leader_election.cc:343] T 5361f08f50b745eb98bf02e979d7c71e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: 5361f08f50b745eb98bf02e979d7c71e
I20251025 07:52:43.985512 2314 raft_consensus.cc:697] T 09898735796541b0a8f3be22cdb6cc30 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:43.985877 2314 consensus_queue.cc:237] T 09898735796541b0a8f3be22cdb6cc30 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:43.989992 2106 leader_election.cc:290] T 09898735796541b0a8f3be22cdb6cc30 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:43.990242 847 leader_election.cc:304] T 5361f08f50b745eb98bf02e979d7c71e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:43.990598 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 0540d6a10cf04fa89756623e9aaaeb68 with delete_type TABLET_DATA_DELETED (replaced by tablet 55a4a524676b4b9aa9464e934b662f30) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:43.990594 2612 consensus_queue.cc:1048] T 34315f64b89148efbf1f20a43d5c5efb P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 07:52:43.997772 2106 raft_consensus.cc:2749] T 5361f08f50b745eb98bf02e979d7c71e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:43.997677 1019 tablet_service.cc:1505] Processing CreateTablet for tablet f9938382aab3426e9bb304859fbffae9 (DEFAULT_TABLE table=default.Table_2afb6a7dc14f48cdb5c7e02852a40cf8 [id=9130879596294c398c41a56614332f2c]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:44.002908 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:46231, user_credentials={real_user=slave}} blocked reactor thread for 63379.6us
I20251025 07:52:44.003189 894 tablet_service.cc:1505] Processing CreateTablet for tablet f9938382aab3426e9bb304859fbffae9 (DEFAULT_TABLE table=default.Table_2afb6a7dc14f48cdb5c7e02852a40cf8 [id=9130879596294c398c41a56614332f2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:44.003723 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9938382aab3426e9bb304859fbffae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:44.004230 746 tablet_service.cc:1505] Processing CreateTablet for tablet f9938382aab3426e9bb304859fbffae9 (DEFAULT_TABLE table=default.Table_2afb6a7dc14f48cdb5c7e02852a40cf8 [id=9130879596294c398c41a56614332f2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:44.004666 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9938382aab3426e9bb304859fbffae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:44.006632 1018 tablet_service.cc:1505] Processing CreateTablet for tablet a515af5846db4f8aaceda12591f713ba (DEFAULT_TABLE table=default.Table_9ff26a6660b84607835d929035c0b285 [id=fb7fdf8aa82b4f62aa7cab5c1b21fbb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:44.007822 885 tablet_service.cc:1505] Processing CreateTablet for tablet a515af5846db4f8aaceda12591f713ba (DEFAULT_TABLE table=default.Table_9ff26a6660b84607835d929035c0b285 [id=fb7fdf8aa82b4f62aa7cab5c1b21fbb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:44.008275 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a515af5846db4f8aaceda12591f713ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:44.015650 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9938382aab3426e9bb304859fbffae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:44.018405 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a515af5846db4f8aaceda12591f713ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:44.026830 751 tablet_service.cc:1505] Processing CreateTablet for tablet a515af5846db4f8aaceda12591f713ba (DEFAULT_TABLE table=default.Table_9ff26a6660b84607835d929035c0b285 [id=fb7fdf8aa82b4f62aa7cab5c1b21fbb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:44.027261 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a515af5846db4f8aaceda12591f713ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:44.032845 1018 tablet_service.cc:1505] Processing CreateTablet for tablet a7be799735dd4cfaa77965cc18d1fe2c (DEFAULT_TABLE table=default.Table_10089cff605d4c26aabf807af44b50ad [id=6e4c46926fde4edda30fea17cef3509a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:44.033278 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7be799735dd4cfaa77965cc18d1fe2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:44.033401 885 tablet_service.cc:1505] Processing CreateTablet for tablet a7be799735dd4cfaa77965cc18d1fe2c (DEFAULT_TABLE table=default.Table_10089cff605d4c26aabf807af44b50ad [id=6e4c46926fde4edda30fea17cef3509a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:44.033847 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7be799735dd4cfaa77965cc18d1fe2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:44.036423 755 tablet_service.cc:1505] Processing CreateTablet for tablet a7be799735dd4cfaa77965cc18d1fe2c (DEFAULT_TABLE table=default.Table_10089cff605d4c26aabf807af44b50ad [id=6e4c46926fde4edda30fea17cef3509a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:44.036903 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7be799735dd4cfaa77965cc18d1fe2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:44.044384 894 tablet_service.cc:1505] Processing CreateTablet for tablet 110a8e9e4c6c434284c0ae14dd6987f7 (DEFAULT_TABLE table=default.Table_6c5a24747ba14352b0285a41ec2985a4 [id=28bbaad474164a8c8c97a76bb8081791]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:44.044514 1019 tablet_service.cc:1505] Processing CreateTablet for tablet 110a8e9e4c6c434284c0ae14dd6987f7 (DEFAULT_TABLE table=default.Table_6c5a24747ba14352b0285a41ec2985a4 [id=28bbaad474164a8c8c97a76bb8081791]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:44.045051 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 110a8e9e4c6c434284c0ae14dd6987f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:44.045089 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 110a8e9e4c6c434284c0ae14dd6987f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:44.052595 753 tablet_service.cc:1505] Processing CreateTablet for tablet 110a8e9e4c6c434284c0ae14dd6987f7 (DEFAULT_TABLE table=default.Table_6c5a24747ba14352b0285a41ec2985a4 [id=28bbaad474164a8c8c97a76bb8081791]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:44.053036 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 110a8e9e4c6c434284c0ae14dd6987f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:44.073594 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_a6c869949b744bc98efb735b0e73e7aa [id=bf77b5ff6f1048dd996d948437f9ff5e]). Expected version 2 got 0
I20251025 07:52:44.074041 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3]). Expected version 1 got 0
I20251025 07:52:44.074266 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5]). Expected version 1 got 0
I20251025 07:52:44.075083 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f]). Expected version 1 got 0
I20251025 07:52:44.075419 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7]). Expected version 1 got 0
I20251025 07:52:44.075924 450 catalog_manager.cc:5649] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } }
I20251025 07:52:44.076586 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 872cb7512d104827b1f1f186d33a4a44 (table default.Table_d5c65dce8b1b4214ba519d964012bfc0 [id=d4be274b36a845a99a4b96b8846248fd]). Expected version 1 got 0
I20251025 07:52:44.077085 450 catalog_manager.cc:5649] T 04d27d54fdb7483b869408af65ff85c9 P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23eeffd3de254ce18db5febfde971c6b (127.0.104.1). New cstate: current_term: 1 leader_uuid: "23eeffd3de254ce18db5febfde971c6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } }
I20251025 07:52:44.077510 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 04d27d54fdb7483b869408af65ff85c9 (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]). Expected version 2 got 0
I20251025 07:52:44.078029 450 catalog_manager.cc:5649] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } }
I20251025 07:52:44.078502 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet cb829649f73d4d3f8862d268c953bd86 (table default.Table_a9026f5e095c4063b86d4c544c686e22 [id=7944225d3ce7439b881bfb9289d9f040]). Expected version 2 got 0
I20251025 07:52:44.079092 450 catalog_manager.cc:5649] T c2713942584e4734acbc0775ead772fa P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } }
I20251025 07:52:44.079679 450 catalog_manager.cc:5649] T 724d0b7e1b65445a9a1f056852983bdb P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } }
I20251025 07:52:44.080346 450 catalog_manager.cc:5649] T 8eaf6c97ab9b41e9ae746a9c5abe1735 P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } }
I20251025 07:52:44.081988 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 5dab439bc50d4e869548154e1d222ad6 (table default.Table_fb9def2302784c999bf0f5707c60eb4c [id=45325c188eac4d34bb016d01de091f87]). Expected version 3 got 0
I20251025 07:52:44.082396 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet aa9d9ab8073945d28d52fb02213f201a (table default.Table_4791672658f043f6b97b9320a531ebb5 [id=dfd20cf9c3e541c4abe23d59e0aa7f2a]). Expected version 1 got 0
I20251025 07:52:44.082545 1048 raft_consensus.cc:3060] T aaef24ccdfba45fe8204327723df834e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:44.082646 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_87d38832f85e4965bfab1a077990114f [id=ad812862698d421184ebf501ce79b7be]). Expected version 2 got 0
I20251025 07:52:44.083513 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet f70091024c034c118fb86578db19a31c (table default.Table_b7da40b96d6c4359bed6f898e8cf6275 [id=e041946c093b4c789ce175c736cc230b]). Expected version 2 got 0
I20251025 07:52:44.084228 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]). Expected version 3 got 0
I20251025 07:52:44.084578 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84]). Expected version 1 got 0
I20251025 07:52:44.084990 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 5fc092c2bb2d41ac8c5b4fc07460c8ec (table default.Table_27c62e973d74406fb00db8bcf324749c [id=630c274339664b9db18ab4ef7c29c16f]). Expected version 1 got 0
I20251025 07:52:44.085228 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 22f7db660616457b83b24e947a0b0bae (table default.Table_4a86191163f2408f9bcc795cd8d1bbf6 [id=bb316a7e000748ca902a9bfb494802d7]). Expected version 1 got 0
I20251025 07:52:44.085503 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet b56209e4934e4407a8b4518ca4af0d46 (table default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de]). Expected version 1 got 0
I20251025 07:52:44.085740 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 830fdbc6812e416690d0db42a73af503 (table default.Table_92824b3f2a774564b3aea04aaa25d204 [id=fb21a45b86ab446e8c3dae4512a9598f]). Expected version 1 got 0
I20251025 07:52:44.085952 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 755f0d6c87044e20a8020f8410940123 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 4 got 0
I20251025 07:52:44.086969 450 catalog_manager.cc:5649] T 0325d462441349ef81841f1b0cab2a20 P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } }
I20251025 07:52:44.087539 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 0325d462441349ef81841f1b0cab2a20 (table default.Table_750ecb905839434fa9a1aa72f87394be [id=9ce3ba77c56a4ef09cb3b40ea0e488f6]). Expected version 1 got 0
I20251025 07:52:44.102660 847 leader_election.cc:304] T b2044ca206634eb285197af4ba2073e7 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:44.102900 2105 consensus_queue.cc:1048] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:44.103390 2106 raft_consensus.cc:2804] T b2044ca206634eb285197af4ba2073e7 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:44.103474 2106 raft_consensus.cc:493] T b2044ca206634eb285197af4ba2073e7 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:44.103571 2106 raft_consensus.cc:3060] T b2044ca206634eb285197af4ba2073e7 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:44.106945 2106 raft_consensus.cc:515] T b2044ca206634eb285197af4ba2073e7 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.107779 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]). Expected version 1 got 0
I20251025 07:52:44.107932 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2044ca206634eb285197af4ba2073e7" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:44.108421 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525]). Expected version 1 got 0
W20251025 07:52:44.108511 850 leader_election.cc:343] T b2044ca206634eb285197af4ba2073e7 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:44.108676 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_e90d7f5ff13b40878e5b625c97792c7d [id=fc8180c09d7b4e74b2e161f7acb8fe0e]). Expected version 1 got 0
I20251025 07:52:44.108839 2106 leader_election.cc:290] T b2044ca206634eb285197af4ba2073e7 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:44.111829 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236]). Expected version 1 got 0
I20251025 07:52:44.112353 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c]). Expected version 1 got 0
I20251025 07:52:44.113551 2146 ts_tablet_manager.cc:1929] T c88c46e022ff41a58b02538f2151e260 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:44.120653 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3]). Expected version 2 got 0
I20251025 07:52:44.121666 450 catalog_manager.cc:5649] T 36262ab8c6964dd684c01b4db6219a4f P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } }
I20251025 07:52:44.122253 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 36262ab8c6964dd684c01b4db6219a4f (table default.Table_76158f8cf82e4cdb9a90580e46aaea48 [id=f249f30497bc450c887d659721469de8]). Expected version 1 got 0
I20251025 07:52:44.123221 450 catalog_manager.cc:5649] T c9f7ba3709c745d4a625449d6e1c8ba1 P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } }
I20251025 07:52:44.124492 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 288e543bb279410c91c01dd1623e110f (table default.Table_6e242ba55b194c23a6701f4b064fdbe5 [id=4c34b64b299f4162a53866cf01c5298f]). Expected version 1 got 0
I20251025 07:52:44.125222 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet ab42909afc524f2fa3c1041495d13a59 (table default.Table_7eb8424d5192485da1a2d67bfdb2bbbf [id=40c29df99590408cbd5c583d4fc741eb]). Expected version 2 got 0
I20251025 07:52:44.125479 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e]). Expected version 1 got 0
I20251025 07:52:44.125818 450 catalog_manager.cc:5649] T bc2d16b33ca8426abbd7bd38339c9f1d P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } }
I20251025 07:52:44.126315 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet bc2d16b33ca8426abbd7bd38339c9f1d (table default.Table_4ea985df4c5b415c9570c64506a98cc4 [id=98d2b7797e33485e8872054aa2eb8ec9]). Expected version 1 got 0
I20251025 07:52:44.126546 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ab42909afc524f2fa3c1041495d13a59" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:44.126772 772 raft_consensus.cc:2468] T ab42909afc524f2fa3c1041495d13a59 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:44.127170 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]). Expected version 1 got 0
I20251025 07:52:44.127430 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]). Expected version 1 got 0
I20251025 07:52:44.127887 450 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 292fe04055954afd8c08ac2636b68191 (table default.Table_9fa69b10b5e842639fc0b0a58cd9b6a5 [id=a253ce0253bd4da4814741c482193d8f]). Expected version 1 got 0
I20251025 07:52:44.138098 2169 ts_tablet_manager.cc:1929] T 8956ad28b35d45f28c683ac055bba441 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:44.138360 2169 log.cc:1199] T 8956ad28b35d45f28c683ac055bba441 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/8956ad28b35d45f28c683ac055bba441
I20251025 07:52:44.138825 2169 ts_tablet_manager.cc:1950] T 8956ad28b35d45f28c683ac055bba441 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:44.139618 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 122) took 5327 ms. Trace:
I20251025 07:52:44.139765 2169 rpcz_store.cc:276] 1025 07:52:38.812206 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.813276 (+ 1070us) service_pool.cc:225] Handling call
1025 07:52:43.533921 (+4720645us) ts_tablet_manager.cc:1151] Deleting tablet 8956ad28b35d45f28c683ac055bba441
1025 07:52:43.533933 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:43.788404 (+254471us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.137965 (+349561us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.139595 (+ 1630us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8734,"tablet-delete.queue_time_us":4717737}
I20251025 07:52:44.140182 2169 tablet_replica.cc:333] T 71432a6848944a0da06ef9b81f160dc4 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:44.140285 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 8956ad28b35d45f28c683ac055bba441 (table default.Table_f84af906af804c7a8e98ad602783d747 [id=39a63c8186374189ad21fbf8492d5d2b]) successfully deleted
I20251025 07:52:44.140504 2169 raft_consensus.cc:2243] T 71432a6848944a0da06ef9b81f160dc4 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:44.122162 2106 consensus_queue.cc:1048] T 26c849d376d143c6bef0c8e0e1d7d03a P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:44.148597 2146 log.cc:1199] T c88c46e022ff41a58b02538f2151e260 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/c88c46e022ff41a58b02538f2151e260
I20251025 07:52:44.149228 2146 ts_tablet_manager.cc:1950] T c88c46e022ff41a58b02538f2151e260 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:44.128221 2612 raft_consensus.cc:493] T ebc824ec4c60411dab1a2ea4a385148e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:44.150321 2612 raft_consensus.cc:515] T ebc824ec4c60411dab1a2ea4a385148e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.151057 1166 tablet_bootstrap.cc:492] T 4678f7eab5e8497db308e8a38e387618 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:44.151417 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc824ec4c60411dab1a2ea4a385148e" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:44.151464 2612 leader_election.cc:290] T ebc824ec4c60411dab1a2ea4a385148e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
W20251025 07:52:44.152117 850 leader_election.cc:343] T ebc824ec4c60411dab1a2ea4a385148e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:44.152849 1166 ts_tablet_manager.cc:1403] T 4678f7eab5e8497db308e8a38e387618 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.228s user 0.020s sys 0.013s
I20251025 07:52:44.153921 1166 raft_consensus.cc:359] T 4678f7eab5e8497db308e8a38e387618 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.162547 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "570201e7bf05406a97c81dde071d894b" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:44.150094 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 122) took 5630 ms. Trace:
I20251025 07:52:44.163656 2146 rpcz_store.cc:276] 1025 07:52:38.519298 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.527886 (+ 8588us) service_pool.cc:225] Handling call
1025 07:52:43.545155 (+5017269us) ts_tablet_manager.cc:1151] Deleting tablet c88c46e022ff41a58b02538f2151e260
1025 07:52:43.545165 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:43.932484 (+387319us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.113444 (+180960us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.150036 (+ 36592us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5016701}
I20251025 07:52:44.164774 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet c88c46e022ff41a58b02538f2151e260 (table default.Table_f6e21e14e6fb4649af45c7d10a5f993e [id=c5eda563853a4e15b846d063608d42ed]) successfully deleted
I20251025 07:52:44.166972 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "09898735796541b0a8f3be22cdb6cc30" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:44.167183 773 raft_consensus.cc:2468] T 09898735796541b0a8f3be22cdb6cc30 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:44.169317 771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "09898735796541b0a8f3be22cdb6cc30" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:44.169471 771 raft_consensus.cc:3060] T 09898735796541b0a8f3be22cdb6cc30 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:44.174486 2508 raft_consensus.cc:493] T 36c414bc700341188b7d85c14173c3e0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:44.174661 2508 raft_consensus.cc:515] T 36c414bc700341188b7d85c14173c3e0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.175539 2508 leader_election.cc:290] T 36c414bc700341188b7d85c14173c3e0 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:44.178696 1169 tablet_bootstrap.cc:492] T 30418858bf95438686bdfe64ac733144 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
W20251025 07:52:44.179996 710 leader_election.cc:343] T 570201e7bf05406a97c81dde071d894b P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:44.180124 710 leader_election.cc:304] T 570201e7bf05406a97c81dde071d894b P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:44.180356 2508 raft_consensus.cc:2749] T 570201e7bf05406a97c81dde071d894b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:44.180510 1166 raft_consensus.cc:385] T 4678f7eab5e8497db308e8a38e387618 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:44.180631 1166 raft_consensus.cc:740] T 4678f7eab5e8497db308e8a38e387618 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:44.180936 1166 consensus_queue.cc:260] T 4678f7eab5e8497db308e8a38e387618 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
W20251025 07:52:44.182607 710 consensus_peers.cc:597] T c4610b0c62e741eca28c21f63cc3de81 P 23eeffd3de254ce18db5febfde971c6b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c4610b0c62e741eca28c21f63cc3de81. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:44.183952 2106 raft_consensus.cc:493] T f656945454fe4921ab819c772a089eb3 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:44.184098 2106 raft_consensus.cc:515] T f656945454fe4921ab819c772a089eb3 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.184466 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2044ca206634eb285197af4ba2073e7" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:44.184617 775 raft_consensus.cc:3060] T b2044ca206634eb285197af4ba2073e7 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:44.184733 2106 leader_election.cc:290] T f656945454fe4921ab819c772a089eb3 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:44.185567 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f0c4f0e8f014e59be9e46d2afe0be9d" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:44.185806 1166 ts_tablet_manager.cc:1434] T 4678f7eab5e8497db308e8a38e387618 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.033s user 0.000s sys 0.002s
I20251025 07:52:44.191010 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc824ec4c60411dab1a2ea4a385148e" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:44.191210 774 raft_consensus.cc:2468] T ebc824ec4c60411dab1a2ea4a385148e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:44.193009 771 raft_consensus.cc:2468] T 09898735796541b0a8f3be22cdb6cc30 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
W20251025 07:52:44.187808 850 consensus_peers.cc:597] T 050a5d3b5f934dd19bf2e221f2fe0780 P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251025 07:52:44.185595 847 leader_election.cc:343] T 6dcab53a020847a1aa27930265ba0205 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: 6dcab53a020847a1aa27930265ba0205
I20251025 07:52:44.198542 847 leader_election.cc:304] T 6dcab53a020847a1aa27930265ba0205 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:44.199009 2612 raft_consensus.cc:2749] T 6dcab53a020847a1aa27930265ba0205 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:44.197760 2146 tablet_replica.cc:333] T 9d6a9937d10a4cc2813671dd1bfa1664 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:44.186113 1166 tablet_bootstrap.cc:492] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:44.205359 2146 raft_consensus.cc:2243] T 9d6a9937d10a4cc2813671dd1bfa1664 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:44.187304 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f656945454fe4921ab819c772a089eb3" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:44.214882 1169 ts_tablet_manager.cc:1403] T 30418858bf95438686bdfe64ac733144 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.273s user 0.008s sys 0.069s
W20251025 07:52:44.215255 850 leader_election.cc:343] T f656945454fe4921ab819c772a089eb3 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:44.216141 775 raft_consensus.cc:2468] T b2044ca206634eb285197af4ba2073e7 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:44.187858 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36c414bc700341188b7d85c14173c3e0" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:44.217130 712 leader_election.cc:343] T 36c414bc700341188b7d85c14173c3e0 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:44.218288 1169 raft_consensus.cc:359] T 30418858bf95438686bdfe64ac733144 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.222620 847 leader_election.cc:304] T ab42909afc524f2fa3c1041495d13a59 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:44.223009 2611 raft_consensus.cc:2804] T ab42909afc524f2fa3c1041495d13a59 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:44.223096 2611 raft_consensus.cc:493] T ab42909afc524f2fa3c1041495d13a59 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:44.223187 2611 raft_consensus.cc:3060] T ab42909afc524f2fa3c1041495d13a59 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:44.224586 2146 raft_consensus.cc:2272] T 9d6a9937d10a4cc2813671dd1bfa1664 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:44.225162 2146 ts_tablet_manager.cc:1916] T 9d6a9937d10a4cc2813671dd1bfa1664 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:44.225976 2611 raft_consensus.cc:515] T ab42909afc524f2fa3c1041495d13a59 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.227559 2611 leader_election.cc:290] T ab42909afc524f2fa3c1041495d13a59 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:44.228834 2508 raft_consensus.cc:493] T 716fb61312e94a4d9eb8ab8336ca140c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:44.228998 2508 raft_consensus.cc:515] T 716fb61312e94a4d9eb8ab8336ca140c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.229568 2508 leader_election.cc:290] T 716fb61312e94a4d9eb8ab8336ca140c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:44.231433 2611 consensus_queue.cc:1048] T c9f7ba3709c745d4a625449d6e1c8ba1 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251025 07:52:44.233917 847 consensus_peers.cc:597] T 34315f64b89148efbf1f20a43d5c5efb P 3aea2869ff814cc7b142012faa274382 -> Peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Couldn't send request to peer 23eeffd3de254ce18db5febfde971c6b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34315f64b89148efbf1f20a43d5c5efb. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:44.235147 847 leader_election.cc:304] T ebc824ec4c60411dab1a2ea4a385148e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:44.235244 1169 raft_consensus.cc:385] T 30418858bf95438686bdfe64ac733144 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:44.235365 1169 raft_consensus.cc:740] T 30418858bf95438686bdfe64ac733144 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:44.235656 2611 raft_consensus.cc:2804] T ebc824ec4c60411dab1a2ea4a385148e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:44.235623 1169 consensus_queue.cc:260] T 30418858bf95438686bdfe64ac733144 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.235767 2611 raft_consensus.cc:493] T ebc824ec4c60411dab1a2ea4a385148e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:44.235920 2611 raft_consensus.cc:3060] T ebc824ec4c60411dab1a2ea4a385148e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:44.236415 1169 ts_tablet_manager.cc:1434] T 30418858bf95438686bdfe64ac733144 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20251025 07:52:44.236788 1169 tablet_bootstrap.cc:492] T c9f7ba3709c745d4a625449d6e1c8ba1 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:44.238878 1169 tablet_bootstrap.cc:654] T c9f7ba3709c745d4a625449d6e1c8ba1 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:44.239537 2611 raft_consensus.cc:515] T ebc824ec4c60411dab1a2ea4a385148e P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.240274 1166 tablet_bootstrap.cc:654] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:44.240742 2611 leader_election.cc:290] T ebc824ec4c60411dab1a2ea4a385148e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:44.250301 847 leader_election.cc:304] T b2044ca206634eb285197af4ba2073e7 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:44.253897 2611 raft_consensus.cc:2804] T b2044ca206634eb285197af4ba2073e7 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:44.254117 2611 raft_consensus.cc:697] T b2044ca206634eb285197af4ba2073e7 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:44.254442 2611 consensus_queue.cc:237] T b2044ca206634eb285197af4ba2073e7 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.228292 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ab42909afc524f2fa3c1041495d13a59" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:44.230332 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "716fb61312e94a4d9eb8ab8336ca140c" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:44.255883 850 leader_election.cc:343] T ab42909afc524f2fa3c1041495d13a59 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:44.255949 712 leader_election.cc:343] T 716fb61312e94a4d9eb8ab8336ca140c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:44.224108 2169 raft_consensus.cc:2272] T 71432a6848944a0da06ef9b81f160dc4 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:44.241092 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc824ec4c60411dab1a2ea4a385148e" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:44.256616 850 leader_election.cc:343] T ebc824ec4c60411dab1a2ea4a385148e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:44.256723 2169 ts_tablet_manager.cc:1916] T 71432a6848944a0da06ef9b81f160dc4 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:44.244995 1048 raft_consensus.cc:3060] T 66f0d398e59a4175aa9fe90715124d1a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:44.257941 1166 tablet_bootstrap.cc:492] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:44.258157 1166 ts_tablet_manager.cc:1403] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.072s user 0.000s sys 0.004s
I20251025 07:52:44.258829 1166 raft_consensus.cc:359] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.259030 1166 raft_consensus.cc:385] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:44.259107 1166 raft_consensus.cc:740] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:44.259315 1166 consensus_queue.cc:260] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.259837 1166 ts_tablet_manager.cc:1434] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251025 07:52:44.267728 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f656945454fe4921ab819c772a089eb3" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:44.267935 775 raft_consensus.cc:2468] T f656945454fe4921ab819c772a089eb3 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
W20251025 07:52:44.268721 710 leader_election.cc:343] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:44.268846 710 leader_election.cc:304] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:44.278972 2597 raft_consensus.cc:493] T 66f0d398e59a4175aa9fe90715124d1a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:44.279119 2597 raft_consensus.cc:515] T 66f0d398e59a4175aa9fe90715124d1a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.279654 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5]). Expected version 1 got 0
I20251025 07:52:44.279922 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3]). Expected version 1 got 0
I20251025 07:52:44.279987 2597 leader_election.cc:290] T 66f0d398e59a4175aa9fe90715124d1a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:44.283524 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_a6c869949b744bc98efb735b0e73e7aa [id=bf77b5ff6f1048dd996d948437f9ff5e]). Expected version 2 got 0
I20251025 07:52:44.284911 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f]). Expected version 1 got 0
I20251025 07:52:44.286614 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7]). Expected version 1 got 0
I20251025 07:52:44.288038 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 872cb7512d104827b1f1f186d33a4a44 (table default.Table_d5c65dce8b1b4214ba519d964012bfc0 [id=d4be274b36a845a99a4b96b8846248fd]). Expected version 1 got 0
I20251025 07:52:44.288815 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet cb829649f73d4d3f8862d268c953bd86 (table default.Table_a9026f5e095c4063b86d4c544c686e22 [id=7944225d3ce7439b881bfb9289d9f040]). Expected version 2 got 0
I20251025 07:52:44.290112 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 04d27d54fdb7483b869408af65ff85c9 (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]). Expected version 2 got 0
I20251025 07:52:44.291795 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet aa9d9ab8073945d28d52fb02213f201a (table default.Table_4791672658f043f6b97b9320a531ebb5 [id=dfd20cf9c3e541c4abe23d59e0aa7f2a]). Expected version 1 got 0
I20251025 07:52:44.295167 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 5dab439bc50d4e869548154e1d222ad6 (table default.Table_fb9def2302784c999bf0f5707c60eb4c [id=45325c188eac4d34bb016d01de091f87]). Expected version 3 got 0
I20251025 07:52:44.260535 2612 consensus_queue.cc:1048] T 66f0d398e59a4175aa9fe90715124d1a P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:44.263857 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36c414bc700341188b7d85c14173c3e0" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:44.267196 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "716fb61312e94a4d9eb8ab8336ca140c" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:44.295925 1166 tablet_bootstrap.cc:492] T ce1ebb1ec14b4801930a72765bca1a7d P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:44.268857 2106 raft_consensus.cc:493] T 4678f7eab5e8497db308e8a38e387618 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:44.296115 2106 raft_consensus.cc:515] T 4678f7eab5e8497db308e8a38e387618 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.296680 2106 leader_election.cc:290] T 4678f7eab5e8497db308e8a38e387618 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:44.297199 1167 tablet_bootstrap.cc:492] T 1b5ac7e845a24b2eb5b2429d47361232 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:44.280588 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66f0d398e59a4175aa9fe90715124d1a" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:44.297354 1167 ts_tablet_manager.cc:1403] T 1b5ac7e845a24b2eb5b2429d47361232 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.362s user 0.033s sys 0.049s
I20251025 07:52:44.297449 2166 ts_tablet_manager.cc:1929] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:44.297540 2166 log.cc:1199] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/f170424be2f643e7af499834955472b0
I20251025 07:52:44.297545 1166 tablet_bootstrap.cc:654] T ce1ebb1ec14b4801930a72765bca1a7d P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:44.297701 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4678f7eab5e8497db308e8a38e387618" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:44.297886 2166 ts_tablet_manager.cc:1950] T f170424be2f643e7af499834955472b0 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:44.298259 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 106) took 5642 ms. Trace:
W20251025 07:52:44.298223 850 leader_election.cc:343] T 4678f7eab5e8497db308e8a38e387618 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:44.298331 2166 rpcz_store.cc:276] 1025 07:52:38.655488 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.655535 (+ 47us) service_pool.cc:225] Handling call
1025 07:52:43.835558 (+5180023us) ts_tablet_manager.cc:1151] Deleting tablet f170424be2f643e7af499834955472b0
1025 07:52:43.835571 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:44.208288 (+372717us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.297336 (+ 89048us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.298240 (+ 904us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5179832}
I20251025 07:52:44.298727 2166 tablet_replica.cc:333] T a13bf4d7eb1440f7a28b85cb1bf99667 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:44.298909 2166 raft_consensus.cc:2243] T a13bf4d7eb1440f7a28b85cb1bf99667 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:44.299065 2166 raft_consensus.cc:2272] T a13bf4d7eb1440f7a28b85cb1bf99667 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:44.299649 2166 ts_tablet_manager.cc:1916] T a13bf4d7eb1440f7a28b85cb1bf99667 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:44.300485 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet f170424be2f643e7af499834955472b0 (table default.Table_f3cbba9b75614002b5464930ccab8c7f [id=4f5379fa0372460ea325339f39289a10]) successfully deleted
I20251025 07:52:44.300990 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_87d38832f85e4965bfab1a077990114f [id=ad812862698d421184ebf501ce79b7be]). Expected version 2 got 0
I20251025 07:52:44.301537 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet f70091024c034c118fb86578db19a31c (table default.Table_b7da40b96d6c4359bed6f898e8cf6275 [id=e041946c093b4c789ce175c736cc230b]). Expected version 2 got 0
I20251025 07:52:44.301695 960 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
I20251025 07:52:44.303537 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ab42909afc524f2fa3c1041495d13a59" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:44.303715 775 raft_consensus.cc:3060] T ab42909afc524f2fa3c1041495d13a59 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:44.304160 2146 ts_tablet_manager.cc:1929] T 9d6a9937d10a4cc2813671dd1bfa1664 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:44.305437 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 5fc092c2bb2d41ac8c5b4fc07460c8ec (table default.Table_27c62e973d74406fb00db8bcf324749c [id=630c274339664b9db18ab4ef7c29c16f]). Expected version 1 got 0
I20251025 07:52:44.305743 1169 tablet_bootstrap.cc:492] T c9f7ba3709c745d4a625449d6e1c8ba1 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:44.305899 1169 ts_tablet_manager.cc:1403] T c9f7ba3709c745d4a625449d6e1c8ba1 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.069s user 0.000s sys 0.003s
I20251025 07:52:44.306252 771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66f0d398e59a4175aa9fe90715124d1a" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:44.306418 771 raft_consensus.cc:2468] T 66f0d398e59a4175aa9fe90715124d1a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:44.306589 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]). Expected version 3 got 0
I20251025 07:52:44.306833 1169 raft_consensus.cc:359] T c9f7ba3709c745d4a625449d6e1c8ba1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.307042 1169 raft_consensus.cc:385] T c9f7ba3709c745d4a625449d6e1c8ba1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:44.307020 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc824ec4c60411dab1a2ea4a385148e" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:44.307121 1169 raft_consensus.cc:740] T c9f7ba3709c745d4a625449d6e1c8ba1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:44.307155 773 raft_consensus.cc:3060] T ebc824ec4c60411dab1a2ea4a385148e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:44.307361 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 22f7db660616457b83b24e947a0b0bae (table default.Table_4a86191163f2408f9bcc795cd8d1bbf6 [id=bb316a7e000748ca902a9bfb494802d7]). Expected version 1 got 0
I20251025 07:52:44.307360 1169 consensus_queue.cc:260] T c9f7ba3709c745d4a625449d6e1c8ba1 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.307827 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 755f0d6c87044e20a8020f8410940123 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 4 got 0
I20251025 07:52:44.308156 2146 log.cc:1199] T 9d6a9937d10a4cc2813671dd1bfa1664 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/9d6a9937d10a4cc2813671dd1bfa1664
I20251025 07:52:44.308701 2146 ts_tablet_manager.cc:1950] T 9d6a9937d10a4cc2813671dd1bfa1664 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:44.309612 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236]). Expected version 1 got 0
I20251025 07:52:44.311040 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84]). Expected version 1 got 0
I20251025 07:52:44.311419 773 raft_consensus.cc:2468] T ebc824ec4c60411dab1a2ea4a385148e P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:44.311645 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 123) took 5783 ms. Trace:
I20251025 07:52:44.311772 2146 rpcz_store.cc:276] 1025 07:52:38.528272 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.528312 (+ 40us) service_pool.cc:225] Handling call
1025 07:52:44.168740 (+5640428us) ts_tablet_manager.cc:1151] Deleting tablet 9d6a9937d10a4cc2813671dd1bfa1664
1025 07:52:44.168750 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:44.228945 (+ 60195us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.304039 (+ 75094us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.311624 (+ 7585us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17386,"tablet-delete.queue_time_us":5640290}
I20251025 07:52:44.312420 2146 tablet_replica.cc:333] T d5e5e5bbcf9f43118eea502b69880af5 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:44.312672 2146 raft_consensus.cc:2243] T d5e5e5bbcf9f43118eea502b69880af5 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:44.312903 2146 raft_consensus.cc:2272] T d5e5e5bbcf9f43118eea502b69880af5 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:44.312381 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 9d6a9937d10a4cc2813671dd1bfa1664 (table default.Table_e7d32e91b9c64ce1b3fcff3d806e959f [id=bab869e5001c4d999daa22083fca128a]) successfully deleted
I20251025 07:52:44.313169 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 0325d462441349ef81841f1b0cab2a20 (table default.Table_750ecb905839434fa9a1aa72f87394be [id=9ce3ba77c56a4ef09cb3b40ea0e488f6]). Expected version 1 got 0
I20251025 07:52:44.313211 2597 raft_consensus.cc:493] T 4b28215f61574cf48c447dd01955ac77 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:44.313261 982 leader_election.cc:304] T 66f0d398e59a4175aa9fe90715124d1a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters: 3aea2869ff814cc7b142012faa274382
I20251025 07:52:44.313345 2597 raft_consensus.cc:515] T 4b28215f61574cf48c447dd01955ac77 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:44.313627 2484 raft_consensus.cc:2804] T 66f0d398e59a4175aa9fe90715124d1a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 07:52:44.313709 2484 raft_consensus.cc:493] T 66f0d398e59a4175aa9fe90715124d1a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3aea2869ff814cc7b142012faa274382)
I20251025 07:52:44.313773 2484 raft_consensus.cc:3060] T 66f0d398e59a4175aa9fe90715124d1a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Advancing to term 2
I20251025 07:52:44.314260 2597 leader_election.cc:290] T 4b28215f61574cf48c447dd01955ac77 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:44.314862 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]). Expected version 1 got 0
I20251025 07:52:44.315294 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525]). Expected version 1 got 0
I20251025 07:52:44.315474 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_e90d7f5ff13b40878e5b625c97792c7d [id=fc8180c09d7b4e74b2e161f7acb8fe0e]). Expected version 1 got 0
I20251025 07:52:44.315727 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet b56209e4934e4407a8b4518ca4af0d46 (table default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de]). Expected version 1 got 0
I20251025 07:52:44.315886 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 830fdbc6812e416690d0db42a73af503 (table default.Table_92824b3f2a774564b3aea04aaa25d204 [id=fb21a45b86ab446e8c3dae4512a9598f]). Expected version 1 got 0
I20251025 07:52:44.316488 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c]). Expected version 1 got 0
I20251025 07:52:44.316813 2484 raft_consensus.cc:515] T 66f0d398e59a4175aa9fe90715124d1a P db42be7ad3c741a0aa427f27e245993b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.317057 2146 ts_tablet_manager.cc:1916] T d5e5e5bbcf9f43118eea502b69880af5 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:44.318915 775 raft_consensus.cc:2468] T ab42909afc524f2fa3c1041495d13a59 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:44.319188 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b28215f61574cf48c447dd01955ac77" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:44.319525 910 raft_consensus.cc:2468] T 4b28215f61574cf48c447dd01955ac77 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:44.320335 2117 raft_consensus.cc:2749] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:44.322510 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 36262ab8c6964dd684c01b4db6219a4f (table default.Table_76158f8cf82e4cdb9a90580e46aaea48 [id=f249f30497bc450c887d659721469de8]). Expected version 1 got 0
I20251025 07:52:44.323462 1166 tablet_bootstrap.cc:492] T ce1ebb1ec14b4801930a72765bca1a7d P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:44.323673 1166 ts_tablet_manager.cc:1403] T ce1ebb1ec14b4801930a72765bca1a7d P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.003s
I20251025 07:52:44.324829 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3]). Expected version 2 got 0
I20251025 07:52:44.325413 1166 raft_consensus.cc:359] T ce1ebb1ec14b4801930a72765bca1a7d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.325609 1166 raft_consensus.cc:385] T ce1ebb1ec14b4801930a72765bca1a7d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:44.325681 1166 raft_consensus.cc:740] T ce1ebb1ec14b4801930a72765bca1a7d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:44.325944 1166 consensus_queue.cc:260] T ce1ebb1ec14b4801930a72765bca1a7d P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.326548 1166 ts_tablet_manager.cc:1434] T ce1ebb1ec14b4801930a72765bca1a7d P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251025 07:52:44.326891 1166 tablet_bootstrap.cc:492] T bdaeedbcf35a4aad97f9ef192991978a P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:44.328231 1167 raft_consensus.cc:359] T 1b5ac7e845a24b2eb5b2429d47361232 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.328444 1167 raft_consensus.cc:385] T 1b5ac7e845a24b2eb5b2429d47361232 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:44.328506 1167 raft_consensus.cc:740] T 1b5ac7e845a24b2eb5b2429d47361232 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:44.328373 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 288e543bb279410c91c01dd1623e110f (table default.Table_6e242ba55b194c23a6701f4b064fdbe5 [id=4c34b64b299f4162a53866cf01c5298f]). Expected version 1 got 0
I20251025 07:52:44.328711 1167 consensus_queue.cc:260] T 1b5ac7e845a24b2eb5b2429d47361232 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.328846 1166 tablet_bootstrap.cc:654] T bdaeedbcf35a4aad97f9ef192991978a P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:44.317368 1169 ts_tablet_manager.cc:1434] T c9f7ba3709c745d4a625449d6e1c8ba1 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20251025 07:52:44.320106 983 leader_election.cc:304] T 4b28215f61574cf48c447dd01955ac77 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:44.329319 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet ab42909afc524f2fa3c1041495d13a59 (table default.Table_7eb8424d5192485da1a2d67bfdb2bbbf [id=40c29df99590408cbd5c583d4fc741eb]). Expected version 2 got 0
I20251025 07:52:44.329349 2484 leader_election.cc:290] T 66f0d398e59a4175aa9fe90715124d1a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 2 election: Requested vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:44.329504 2484 raft_consensus.cc:2804] T 4b28215f61574cf48c447dd01955ac77 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:44.329560 2484 raft_consensus.cc:493] T 4b28215f61574cf48c447dd01955ac77 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:44.329615 2484 raft_consensus.cc:3060] T 4b28215f61574cf48c447dd01955ac77 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:44.329818 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66f0d398e59a4175aa9fe90715124d1a" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:44.330041 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet bc2d16b33ca8426abbd7bd38339c9f1d (table default.Table_4ea985df4c5b415c9570c64506a98cc4 [id=98d2b7797e33485e8872054aa2eb8ec9]). Expected version 1 got 0
I20251025 07:52:44.330329 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e]). Expected version 1 got 0
I20251025 07:52:44.323729 1048 raft_consensus.cc:3060] T cb829649f73d4d3f8862d268c953bd86 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:44.330686 1046 raft_consensus.cc:1240] T 66f0d398e59a4175aa9fe90715124d1a P db42be7ad3c741a0aa427f27e245993b [term 2 FOLLOWER]: Rejecting Update request from peer 3aea2869ff814cc7b142012faa274382 for earlier term 1. Current term is 2. Ops: []
I20251025 07:52:44.330931 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]). Expected version 1 got 0
I20251025 07:52:44.331332 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]). Expected version 1 got 0
I20251025 07:52:44.332340 2612 consensus_queue.cc:1059] T 66f0d398e59a4175aa9fe90715124d1a P 3aea2869ff814cc7b142012faa274382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:44.332629 2612 raft_consensus.cc:3055] T 66f0d398e59a4175aa9fe90715124d1a P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Stepping down as leader of term 1
I20251025 07:52:44.332758 2612 raft_consensus.cc:740] T 66f0d398e59a4175aa9fe90715124d1a P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:44.333083 2612 consensus_queue.cc:260] T 66f0d398e59a4175aa9fe90715124d1a P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.337852 2612 raft_consensus.cc:3060] T 66f0d398e59a4175aa9fe90715124d1a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Advancing to term 2
I20251025 07:52:44.345170 1169 tablet_bootstrap.cc:492] T 6dcab53a020847a1aa27930265ba0205 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
W20251025 07:52:44.345371 983 consensus_peers.cc:597] T f3c87a83082b421cb4be546ac4ce7ad8 P db42be7ad3c741a0aa427f27e245993b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3c87a83082b421cb4be546ac4ce7ad8. This is attempt 6: this message will repeat every 5th retry.
I20251025 07:52:44.345177 449 catalog_manager.cc:5649] T 292fe04055954afd8c08ac2636b68191 P db42be7ad3c741a0aa427f27e245993b reported cstate change: term changed from 0 to 1, leader changed from <none> to db42be7ad3c741a0aa427f27e245993b (127.0.104.3). New cstate: current_term: 1 leader_uuid: "db42be7ad3c741a0aa427f27e245993b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: UNKNOWN } } }
I20251025 07:52:44.345590 449 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 292fe04055954afd8c08ac2636b68191 (table default.Table_9fa69b10b5e842639fc0b0a58cd9b6a5 [id=a253ce0253bd4da4814741c482193d8f]). Expected version 1 got 0
I20251025 07:52:44.343219 765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66f0d398e59a4175aa9fe90715124d1a" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:44.346413 765 raft_consensus.cc:3060] T 66f0d398e59a4175aa9fe90715124d1a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Advancing to term 2
W20251025 07:52:44.347092 982 leader_election.cc:343] T 4b28215f61574cf48c447dd01955ac77 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: 4b28215f61574cf48c447dd01955ac77
I20251025 07:52:44.349306 2484 raft_consensus.cc:515] T 4b28215f61574cf48c447dd01955ac77 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:44.350509 2484 leader_election.cc:290] T 4b28215f61574cf48c447dd01955ac77 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:44.350584 765 raft_consensus.cc:2468] T 66f0d398e59a4175aa9fe90715124d1a P 23eeffd3de254ce18db5febfde971c6b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 2.
I20251025 07:52:44.351104 982 leader_election.cc:304] T 66f0d398e59a4175aa9fe90715124d1a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters: 3aea2869ff814cc7b142012faa274382
I20251025 07:52:44.351464 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b28215f61574cf48c447dd01955ac77" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:44.351624 910 raft_consensus.cc:3060] T 4b28215f61574cf48c447dd01955ac77 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:44.352103 2484 raft_consensus.cc:2804] T 66f0d398e59a4175aa9fe90715124d1a P db42be7ad3c741a0aa427f27e245993b [term 2 FOLLOWER]: Leader election won for term 2
W20251025 07:52:44.352713 982 leader_election.cc:343] T 4b28215f61574cf48c447dd01955ac77 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: 4b28215f61574cf48c447dd01955ac77
I20251025 07:52:44.352211 2484 raft_consensus.cc:697] T 66f0d398e59a4175aa9fe90715124d1a P db42be7ad3c741a0aa427f27e245993b [term 2 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:44.353106 2484 consensus_queue.cc:237] T 66f0d398e59a4175aa9fe90715124d1a P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.323335 2169 ts_tablet_manager.cc:1929] T 71432a6848944a0da06ef9b81f160dc4 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:44.354485 910 raft_consensus.cc:2468] T 4b28215f61574cf48c447dd01955ac77 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:44.354717 2169 log.cc:1199] T 71432a6848944a0da06ef9b81f160dc4 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/71432a6848944a0da06ef9b81f160dc4
I20251025 07:52:44.354861 983 leader_election.cc:304] T 4b28215f61574cf48c447dd01955ac77 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b
I20251025 07:52:44.356021 2467 raft_consensus.cc:2804] T 4b28215f61574cf48c447dd01955ac77 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:44.356132 2467 raft_consensus.cc:697] T 4b28215f61574cf48c447dd01955ac77 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:44.356391 2467 consensus_queue.cc:237] T 4b28215f61574cf48c447dd01955ac77 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:44.362617 2166 ts_tablet_manager.cc:1929] T a13bf4d7eb1440f7a28b85cb1bf99667 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:44.362802 2166 log.cc:1199] T a13bf4d7eb1440f7a28b85cb1bf99667 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/a13bf4d7eb1440f7a28b85cb1bf99667
I20251025 07:52:44.363265 2166 ts_tablet_manager.cc:1950] T a13bf4d7eb1440f7a28b85cb1bf99667 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:44.363641 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 107) took 5706 ms. Trace:
I20251025 07:52:44.363842 2166 rpcz_store.cc:276] 1025 07:52:38.656820 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.656862 (+ 42us) service_pool.cc:225] Handling call
1025 07:52:44.298676 (+5641814us) ts_tablet_manager.cc:1151] Deleting tablet a13bf4d7eb1440f7a28b85cb1bf99667
1025 07:52:44.298687 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:44.352018 (+ 53331us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.362511 (+ 10493us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.363624 (+ 1113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5641640}
I20251025 07:52:44.364491 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet a13bf4d7eb1440f7a28b85cb1bf99667 (table default.Table_434ef1ade0a14af987864563993afae8 [id=3a3af35e11944a6f97fc0144cb9d08d4]) successfully deleted
I20251025 07:52:44.364959 2166 tablet_replica.cc:333] T a6ddefa1738f4ff98586900c344359d2 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:44.365252 2166 raft_consensus.cc:2243] T a6ddefa1738f4ff98586900c344359d2 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:44.365484 2166 raft_consensus.cc:2272] T a6ddefa1738f4ff98586900c344359d2 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:44.366180 2166 ts_tablet_manager.cc:1916] T a6ddefa1738f4ff98586900c344359d2 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:44.368242 2169 ts_tablet_manager.cc:1950] T 71432a6848944a0da06ef9b81f160dc4 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:44.369112 2611 raft_consensus.cc:493] T 30418858bf95438686bdfe64ac733144 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:44.369319 2611 raft_consensus.cc:515] T 30418858bf95438686bdfe64ac733144 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.369542 1169 tablet_bootstrap.cc:654] T 6dcab53a020847a1aa27930265ba0205 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:44.370409 2611 leader_election.cc:290] T 30418858bf95438686bdfe64ac733144 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:44.289973 847 leader_election.cc:304] T f656945454fe4921ab819c772a089eb3 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:44.372414 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:46231, user_credentials={real_user=slave}} blocked reactor thread for 82472.7us
I20251025 07:52:44.370287 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 123) took 5556 ms. Trace:
I20251025 07:52:44.374266 2169 rpcz_store.cc:276] 1025 07:52:38.813570 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.813825 (+ 255us) service_pool.cc:225] Handling call
1025 07:52:44.140115 (+5326290us) ts_tablet_manager.cc:1151] Deleting tablet 71432a6848944a0da06ef9b81f160dc4
1025 07:52:44.140124 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:44.261605 (+121481us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.323203 (+ 61598us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.370271 (+ 47068us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5324136}
I20251025 07:52:44.375015 2169 tablet_replica.cc:333] T c59f6e6eb9594a8e83f109a09b034045 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:44.372661 2611 raft_consensus.cc:2804] T f656945454fe4921ab819c772a089eb3 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:44.375597 2611 raft_consensus.cc:493] T f656945454fe4921ab819c772a089eb3 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:44.375762 2611 raft_consensus.cc:3060] T f656945454fe4921ab819c772a089eb3 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:44.376232 2169 raft_consensus.cc:2243] T c59f6e6eb9594a8e83f109a09b034045 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:44.377948 1166 tablet_bootstrap.cc:492] T bdaeedbcf35a4aad97f9ef192991978a P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:44.378774 1166 ts_tablet_manager.cc:1403] T bdaeedbcf35a4aad97f9ef192991978a P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.002s
I20251025 07:52:44.378989 2169 raft_consensus.cc:2272] T c59f6e6eb9594a8e83f109a09b034045 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:44.379257 710 leader_election.cc:343] T 36c414bc700341188b7d85c14173c3e0 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:44.379400 710 leader_election.cc:304] T 36c414bc700341188b7d85c14173c3e0 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:44.379676 2665 raft_consensus.cc:2749] T 36c414bc700341188b7d85c14173c3e0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:44.380198 710 leader_election.cc:343] T 716fb61312e94a4d9eb8ab8336ca140c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:44.380295 710 leader_election.cc:304] T 716fb61312e94a4d9eb8ab8336ca140c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:44.380563 2665 raft_consensus.cc:2749] T 716fb61312e94a4d9eb8ab8336ca140c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:44.381603 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4678f7eab5e8497db308e8a38e387618" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:44.381779 776 raft_consensus.cc:2468] T 4678f7eab5e8497db308e8a38e387618 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:44.382455 1166 raft_consensus.cc:359] T bdaeedbcf35a4aad97f9ef192991978a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.383337 1166 raft_consensus.cc:385] T bdaeedbcf35a4aad97f9ef192991978a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:44.384646 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30418858bf95438686bdfe64ac733144" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:44.384860 776 raft_consensus.cc:2468] T 30418858bf95438686bdfe64ac733144 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:44.385366 1166 raft_consensus.cc:740] T bdaeedbcf35a4aad97f9ef192991978a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:44.392714 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30418858bf95438686bdfe64ac733144" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:44.392917 1046 raft_consensus.cc:2468] T 30418858bf95438686bdfe64ac733144 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:44.393332 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 71432a6848944a0da06ef9b81f160dc4 (table default.Table_79969e4ace6f4227b2748b59136027d6 [id=a1135611169142ba9ef987e3e9f95ce8]) successfully deleted
I20251025 07:52:44.393471 850 leader_election.cc:304] T 30418858bf95438686bdfe64ac733144 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:44.393939 2612 raft_consensus.cc:2804] T 30418858bf95438686bdfe64ac733144 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:44.394091 2612 raft_consensus.cc:493] T 30418858bf95438686bdfe64ac733144 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:44.394248 2612 raft_consensus.cc:3060] T 30418858bf95438686bdfe64ac733144 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:44.396973 2612 raft_consensus.cc:515] T 30418858bf95438686bdfe64ac733144 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.397729 2657 mvcc.cc:204] Tried to move back new op lower bound from 7214607388382949376 to 7214607375745744896. Current Snapshot: MvccSnapshot[applied={T|T < 7214607388382949376 or (T in {7214607388382949376})}]
I20251025 07:52:44.398099 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30418858bf95438686bdfe64ac733144" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:44.398320 1046 raft_consensus.cc:3060] T 30418858bf95438686bdfe64ac733144 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:44.401263 1046 raft_consensus.cc:2468] T 30418858bf95438686bdfe64ac733144 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:44.401644 850 leader_election.cc:304] T 30418858bf95438686bdfe64ac733144 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:44.402076 2615 raft_consensus.cc:2804] T 30418858bf95438686bdfe64ac733144 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:44.402262 2615 raft_consensus.cc:697] T 30418858bf95438686bdfe64ac733144 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:44.402518 2615 consensus_queue.cc:237] T 30418858bf95438686bdfe64ac733144 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.403522 2660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "b2dfbd0b44a441c9be9254f660974944" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } } }
I20251025 07:52:44.404623 2166 ts_tablet_manager.cc:1929] T a6ddefa1738f4ff98586900c344359d2 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:44.404211 2602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2dfbd0b44a441c9be9254f660974944. Latest consensus state: current_term: 2 leader_uuid: "b2dfbd0b44a441c9be9254f660974944" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } } }
I20251025 07:52:44.405098 2602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 07:52:44.405478 2660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9 [sys.catalog]: This master's current role is: FOLLOWER
I20251025 07:52:44.410069 2314 raft_consensus.cc:493] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:44.410254 2314 raft_consensus.cc:515] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.410524 2166 log.cc:1199] T a6ddefa1738f4ff98586900c344359d2 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/a6ddefa1738f4ff98586900c344359d2
I20251025 07:52:44.410955 2314 leader_election.cc:290] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:44.411954 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "91728987303b41fd840eb56f3a311c5f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:44.411867 2090 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41062 (request call id 0) took 5980 ms. Trace:
W20251025 07:52:44.412503 850 leader_election.cc:343] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:44.386025 2611 raft_consensus.cc:515] T f656945454fe4921ab819c772a089eb3 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.411322 2166 ts_tablet_manager.cc:1950] T a6ddefa1738f4ff98586900c344359d2 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:44.414633 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 108) took 5756 ms. Trace:
I20251025 07:52:44.414783 1166 consensus_queue.cc:260] T bdaeedbcf35a4aad97f9ef192991978a P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.415133 2166 rpcz_store.cc:276] 1025 07:52:38.658266 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.658310 (+ 44us) service_pool.cc:225] Handling call
1025 07:52:44.364903 (+5706593us) ts_tablet_manager.cc:1151] Deleting tablet a6ddefa1738f4ff98586900c344359d2
1025 07:52:44.364916 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:44.382059 (+ 17143us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.404473 (+ 22414us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.414615 (+ 10142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5706401}
I20251025 07:52:44.415856 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet a6ddefa1738f4ff98586900c344359d2 (table default.Table_c902adc24467448f9b5d4f585cf614f5 [id=56e169f79a37465d82dd1d1c3c285740]) successfully deleted
I20251025 07:52:44.415905 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f656945454fe4921ab819c772a089eb3" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:44.416369 2166 tablet_replica.cc:333] T fa77085e81c7440d9e255c44d1e80c26 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:44.416896 2166 raft_consensus.cc:2243] T fa77085e81c7440d9e255c44d1e80c26 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:44.417572 2166 raft_consensus.cc:2272] T fa77085e81c7440d9e255c44d1e80c26 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:44.418644 2166 ts_tablet_manager.cc:1916] T fa77085e81c7440d9e255c44d1e80c26 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:44.419248 1166 ts_tablet_manager.cc:1434] T bdaeedbcf35a4aad97f9ef192991978a P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.040s user 0.002s sys 0.000s
W20251025 07:52:44.416275 850 leader_election.cc:343] T f656945454fe4921ab819c772a089eb3 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:44.424206 2611 leader_election.cc:290] T f656945454fe4921ab819c772a089eb3 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:44.414788 2612 leader_election.cc:290] T 30418858bf95438686bdfe64ac733144 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:44.426041 2314 consensus_queue.cc:1048] T 050a5d3b5f934dd19bf2e221f2fe0780 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:44.426447 2675 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20251025 07:52:44.428098 850 consensus_peers.cc:597] T f661d5909657455581dcb5e6eadd3c0b P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:44.433465 1167 ts_tablet_manager.cc:1434] T 1b5ac7e845a24b2eb5b2429d47361232 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.136s user 0.008s sys 0.001s
I20251025 07:52:44.434103 2106 consensus_queue.cc:1048] T c87ab6a3d08f43e0b2fe58a6d7b11588 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:44.443742 1167 tablet_bootstrap.cc:492] T 9f039d9107ae493486d799ab6a1d731d P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:44.443993 2169 ts_tablet_manager.cc:1916] T c59f6e6eb9594a8e83f109a09b034045 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:44.450440 2680 tablet.cc:1722] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251025 07:52:44.412528 2090 rpcz_store.cc:276] 1025 07:52:38.431378 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.431437 (+ 59us) service_pool.cc:225] Handling call
1025 07:52:43.921829 (+5490392us) raft_consensus.cc:1471] Updating replica for 28 ops
1025 07:52:44.162813 (+240984us) raft_consensus.cc:1534] Early marking committed up to index 667
1025 07:52:44.242320 (+ 79507us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.242325 (+ 5us) write_op.cc:454] Released schema lock
1025 07:52:44.242595 (+ 270us) raft_consensus.cc:1539] Triggering prepare for 28 ops
1025 07:52:44.243018 (+ 423us) write_op.cc:270] Start()
1025 07:52:44.243065 (+ 47us) write_op.cc:276] Timestamp: P: 1761378756274659 usec, L: 0
1025 07:52:44.278781 (+ 35716us) write_op.cc:270] Start()
1025 07:52:44.278820 (+ 39us) write_op.cc:276] Timestamp: P: 1761378756279482 usec, L: 0
1025 07:52:44.279047 (+ 227us) write_op.cc:270] Start()
1025 07:52:44.279073 (+ 26us) write_op.cc:276] Timestamp: P: 1761378756280257 usec, L: 0
1025 07:52:44.279207 (+ 134us) write_op.cc:270] Start()
1025 07:52:44.279228 (+ 21us) write_op.cc:276] Timestamp: P: 1761378756281081 usec, L: 0
1025 07:52:44.279373 (+ 145us) write_op.cc:270] Start()
1025 07:52:44.279456 (+ 83us) write_op.cc:276] Timestamp: P: 1761378756281673 usec, L: 0
1025 07:52:44.279625 (+ 169us) write_op.cc:270] Start()
1025 07:52:44.279646 (+ 21us) write_op.cc:276] Timestamp: P: 1761378756282291 usec, L: 0
1025 07:52:44.279797 (+ 151us) write_op.cc:270] Start()
1025 07:52:44.279828 (+ 31us) write_op.cc:276] Timestamp: P: 1761378756284237 usec, L: 0
1025 07:52:44.280000 (+ 172us) write_op.cc:270] Start()
1025 07:52:44.280024 (+ 24us) write_op.cc:276] Timestamp: P: 1761378756294070 usec, L: 0
1025 07:52:44.280182 (+ 158us) write_op.cc:270] Start()
1025 07:52:44.280205 (+ 23us) write_op.cc:276] Timestamp: P: 1761378756313152 usec, L: 0
1025 07:52:44.280342 (+ 137us) write_op.cc:270] Start()
1025 07:52:44.280360 (+ 18us) write_op.cc:276] Timestamp: P: 1761378756804304 usec, L: 0
1025 07:52:44.280506 (+ 146us) write_op.cc:270] Start()
1025 07:52:44.280525 (+ 19us) write_op.cc:276] Timestamp: P: 1761378756849555 usec, L: 0
1025 07:52:44.280656 (+ 131us) write_op.cc:270] Start()
1025 07:52:44.280687 (+ 31us) write_op.cc:276] Timestamp: P: 1761378756852996 usec, L: 0
1025 07:52:44.280839 (+ 152us) write_op.cc:270] Start()
1025 07:52:44.280862 (+ 23us) write_op.cc:276] Timestamp: P: 1761378756870522 usec, L: 0
1025 07:52:44.281020 (+ 158us) write_op.cc:270] Start()
1025 07:52:44.281039 (+ 19us) write_op.cc:276] Timestamp: P: 1761378756871124 usec, L: 0
1025 07:52:44.281178 (+ 139us) write_op.cc:270] Start()
1025 07:52:44.281197 (+ 19us) write_op.cc:276] Timestamp: P: 1761378756907537 usec, L: 0
1025 07:52:44.281359 (+ 162us) write_op.cc:270] Start()
1025 07:52:44.281379 (+ 20us) write_op.cc:276] Timestamp: P: 1761378756918631 usec, L: 0
1025 07:52:44.281578 (+ 199us) write_op.cc:270] Start()
1025 07:52:44.281609 (+ 31us) write_op.cc:276] Timestamp: P: 1761378756929431 usec, L: 0
1025 07:52:44.281767 (+ 158us) write_op.cc:270] Start()
1025 07:52:44.281787 (+ 20us) write_op.cc:276] Timestamp: P: 1761378756940981 usec, L: 0
1025 07:52:44.281924 (+ 137us) write_op.cc:270] Start()
1025 07:52:44.281946 (+ 22us) write_op.cc:276] Timestamp: P: 1761378757044322 usec, L: 0
1025 07:52:44.282238 (+ 292us) write_op.cc:270] Start()
1025 07:52:44.282259 (+ 21us) write_op.cc:276] Timestamp: P: 1761378757056770 usec, L: 0
1025 07:52:44.282385 (+ 126us) write_op.cc:270] Start()
1025 07:52:44.282402 (+ 17us) write_op.cc:276] Timestamp: P: 1761378757057737 usec, L: 0
1025 07:52:44.282510 (+ 108us) write_op.cc:270] Start()
1025 07:52:44.282525 (+ 15us) write_op.cc:276] Timestamp: P: 1761378757058561 usec, L: 0
1025 07:52:44.282628 (+ 103us) write_op.cc:270] Start()
1025 07:52:44.282685 (+ 57us) write_op.cc:276] Timestamp: P: 1761378757172363 usec, L: 0
1025 07:52:44.282810 (+ 125us) write_op.cc:270] Start()
1025 07:52:44.282827 (+ 17us) write_op.cc:276] Timestamp: P: 1761378757672673 usec, L: 0
1025 07:52:44.282966 (+ 139us) write_op.cc:270] Start()
1025 07:52:44.282983 (+ 17us) write_op.cc:276] Timestamp: P: 1761378758059786 usec, L: 0
1025 07:52:44.283081 (+ 98us) write_op.cc:270] Start()
1025 07:52:44.283095 (+ 14us) write_op.cc:276] Timestamp: P: 1761378758073360 usec, L: 0
1025 07:52:44.283338 (+ 243us) write_op.cc:270] Start()
1025 07:52:44.283358 (+ 20us) write_op.cc:276] Timestamp: P: 1761378758241864 usec, L: 0
1025 07:52:44.284407 (+ 1049us) log.cc:844] Serialized 21814 byte log entry
1025 07:52:44.284530 (+ 123us) raft_consensus.cc:1645] Marking committed up to 695
1025 07:52:44.284686 (+ 156us) log.cc:844] Serialized 15 byte log entry
1025 07:52:44.316622 (+ 31936us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.316630 (+ 8us) write_op.cc:454] Released schema lock
1025 07:52:44.317446 (+ 816us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.317450 (+ 4us) write_op.cc:454] Released schema lock
1025 07:52:44.318186 (+ 736us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.318191 (+ 5us) write_op.cc:454] Released schema lock
1025 07:52:44.318734 (+ 543us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.318741 (+ 7us) write_op.cc:454] Released schema lock
1025 07:52:44.321407 (+ 2666us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.321415 (+ 8us) write_op.cc:454] Released schema lock
1025 07:52:44.391407 (+ 69992us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.391415 (+ 8us) write_op.cc:454] Released schema lock
1025 07:52:44.394305 (+ 2890us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.394313 (+ 8us) write_op.cc:454] Released schema lock
1025 07:52:44.394848 (+ 535us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.394852 (+ 4us) write_op.cc:454] Released schema lock
1025 07:52:44.395270 (+ 418us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.395273 (+ 3us) write_op.cc:454] Released schema lock
1025 07:52:44.395734 (+ 461us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.395737 (+ 3us) write_op.cc:454] Released schema lock
1025 07:52:44.396142 (+ 405us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.396145 (+ 3us) write_op.cc:454] Released schema lock
1025 07:52:44.396575 (+ 430us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.396579 (+ 4us) write_op.cc:454] Released schema lock
1025 07:52:44.397099 (+ 520us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.397102 (+ 3us) write_op.cc:454] Released schema lock
1025 07:52:44.397978 (+ 876us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.397984 (+ 6us) write_op.cc:454] Released schema lock
1025 07:52:44.398453 (+ 469us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.398457 (+ 4us) write_op.cc:454] Released schema lock
1025 07:52:44.398972 (+ 515us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.398976 (+ 4us) write_op.cc:454] Released schema lock
1025 07:52:44.399402 (+ 426us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.399406 (+ 4us) write_op.cc:454] Released schema lock
1025 07:52:44.399832 (+ 426us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.399839 (+ 7us) write_op.cc:454] Released schema lock
1025 07:52:44.400940 (+ 1101us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.400943 (+ 3us) write_op.cc:454] Released schema lock
1025 07:52:44.401775 (+ 832us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.401779 (+ 4us) write_op.cc:454] Released schema lock
1025 07:52:44.402263 (+ 484us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.402266 (+ 3us) write_op.cc:454] Released schema lock
1025 07:52:44.402402 (+ 136us) raft_consensus.cc:1697] Filling consensus response to leader.
1025 07:52:44.403468 (+ 1066us) spinlock_profiling.cc:243] Waited 88.7 ms on lock 0x616000051754. stack: 00007f81a9f519f6 00007f81a9f517b0 00007f81a8f2d3ad 0000000000983f04 0000000000983ecd 00007f81b924f6bd 00007f81b924f7b3 00007f81b5e1a441 00007f81bdcc1385 00007f81bdcd51e5 00007f81bdcd4f91 00007f81b71952fd 00007f81b5e2b24e 00007f81b5e2af9d 0000000000a8867c 00007f81a9f97d7f
1025 07:52:44.404162 (+ 694us) spinlock_profiling.cc:243] Waited 241 ms on lock 0x616000051754. stack: 00007f81a9f519f6 00007f81a9f517b0 00007f81a8f2d3ad 0000000000983f04 0000000000983ecd 00007f81b924f6bd 00007f81b924f7b3 00007f81b5e1a441 00007f81bdcc1385 00007f81bdcd51e5 00007f81bdcd4f91 00007f81b71952fd 00007f81b5e2b24e 00007f81b5e2af9d 0000000000a8867c 00007f81a9f97d7f
1025 07:52:44.411612 (+ 7450us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1025 07:52:44.411646 (+ 34us) raft_consensus.cc:1685] finished
1025 07:52:44.411652 (+ 6us) raft_consensus.cc:1691] UpdateReplicas() finished
1025 07:52:44.411827 (+ 175us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:44.276847 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:44.277006 (+ 159us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:44.277013 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:44.277016 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:44.277169 (+ 153us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:44.277186 (+ 17us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:44.277191 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:44.277268 (+ 77us) tablet.cc:699] Row locks acquired
1025 07:52:44.277270 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:44.315749 (+ 38479us) op_driver.cc:464] REPLICATION: finished
1025 07:52:44.315879 (+ 130us) write_op.cc:301] APPLY: starting
1025 07:52:44.315899 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:44.315948 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:44.315951 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:44.316074 (+ 123us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:44.316134 (+ 60us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:44.316168 (+ 34us) write_op.cc:312] APPLY: finished
1025 07:52:44.316256 (+ 88us) log.cc:844] Serialized 29 byte log entry
1025 07:52:44.316325 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.316348 (+ 23us) write_op.cc:454] Released schema lock
1025 07:52:44.316356 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
1025 07:52:44.285034 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:44.285167 (+ 133us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:44.285175 (+ 8us) write_op.cc:435] Acquired schema lock
1025 07:52:44.285178 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:44.285296 (+ 118us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:44.285311 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:44.285316 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:44.285400 (+ 84us) tablet.cc:699] Row locks acquired
1025 07:52:44.285402 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:44.316764 (+ 31362us) op_driver.cc:464] REPLICATION: finished
1025 07:52:44.316855 (+ 91us) write_op.cc:301] APPLY: starting
1025 07:52:44.316867 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:44.316901 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:44.316904 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:44.316995 (+ 91us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:44.317055 (+ 60us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:44.317076 (+ 21us) write_op.cc:312] APPLY: finished
1025 07:52:44.317144 (+ 68us) log.cc:844] Serialized 29 byte log entry
1025 07:52:44.317203 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.317218 (+ 15us) write_op.cc:454] Released schema lock
1025 07:52:44.317223 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
1025 07:52:44.285475 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:44.285564 (+ 89us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:44.285571 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:44.285574 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:44.285658 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:44.285685 (+ 27us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:44.285692 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:44.285749 (+ 57us) tablet.cc:699] Row locks acquired
1025 07:52:44.285751 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:44.317523 (+ 31772us) op_driver.cc:464] REPLICATION: finished
1025 07:52:44.317596 (+ 73us) write_op.cc:301] APPLY: starting
1025 07:52:44.317608 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:44.317653 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:44.317655 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:44.317739 (+ 84us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:44.317794 (+ 55us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:44.317821 (+ 27us) write_op.cc:312] APPLY: finished
1025 07:52:44.317880 (+ 59us) log.cc:844] Serialized 29 byte log entry
1025 07:52:44.317934 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.317948 (+ 14us) write_op.cc:454] Released schema lock
1025 07:52:44.317954 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
1025 07:52:44.285811 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:44.285887 (+ 76us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:44.285892 (+ 5us) write_op.cc:435] Acquired schema lock
1025 07:52:44.285894 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:44.285955 (+ 61us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:44.285965 (+ 10us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:44.285968 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:44.286010 (+ 42us) tablet.cc:699] Row locks acquired
1025 07:52:44.286012 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:44.318255 (+ 32243us) op_driver.cc:464] REPLICATION: finished
1025 07:52:44.318321 (+ 66us) write_op.cc:301] APPLY: starting
1025 07:52:44.318330 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:44.318355 (+ 25us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:44.318357 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:44.318386 (+ 29us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:44.318407 (+ 21us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:44.318429 (+ 22us) write_op.cc:312] APPLY: finished
1025 07:52:44.318474 (+ 45us) log.cc:844] Serialized 23 byte log entry
1025 07:52:44.318514 (+ 40us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.318525 (+ 11us) write_op.cc:454] Released schema lock
1025 07:52:44.318530 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
1025 07:52:44.286075 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:44.286181 (+ 106us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:44.286186 (+ 5us) write_op.cc:435] Acquired schema lock
1025 07:52:44.286187 (+ 1us) tablet.cc:660] Decoding operations
1025 07:52:44.286253 (+ 66us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:44.286266 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:44.286270 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:44.286323 (+ 53us) tablet.cc:699] Row locks acquired
1025 07:52:44.286325 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:44.318811 (+ 32486us) op_driver.cc:464] REPLICATION: finished
1025 07:52:44.318889 (+ 78us) write_op.cc:301] APPLY: starting
1025 07:52:44.318900 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:44.318926 (+ 26us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:44.318930 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:44.319017 (+ 87us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:44.319048 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:44.319084 (+ 36us) write_op.cc:312] APPLY: finished
1025 07:52:44.319136 (+ 52us) log.cc:844] Serialized 29 byte log entry
1025 07:52:44.319182 (+ 46us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.319194 (+ 12us) write_op.cc:454] Released schema lock
1025 07:52:44.319200 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
1025 07:52:44.286389 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:44.286468 (+ 79us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:44.286472 (+ 4us) write_op.cc:435] Acquired schema lock
1025 07:52:44.286474 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:44.286542 (+ 68us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:44.286553 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:44.286557 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:44.286614 (+ 57us) tablet.cc:699] Row locks acquired
1025 07:52:44.286618 (+ 4us) write_op.cc:260] PREPARE: finished
1025 07:52:44.321500 (+ 34882us) op_driver.cc:464] REPLICATION: finished
1025 07:52:44.321764 (+ 264us) write_op.cc:301] APPLY: starting
1025 07:52:44.321784 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:44.321830 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:44.321835 (+ 5us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:44.321973 (+ 138us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:44.322004 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:44.322039 (+ 35us) write_op.cc:312] APPLY: finished
1025 07:52:44.322110 (+ 71us) log.cc:844] Serialized 29 byte log entry
1025 07:52:44.322217 (+ 107us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.322234 (+ 17us) write_op.cc:454] Released schema lock
1025 07:52:44.322242 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:44.322293 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.322297 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
1025 07:52:44.286714 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:44.286806 (+ 92us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:44.286811 (+ 5us) write_op.cc:435] Acquired schema lock
1025 07:52:44.286815 (+ 4us) tablet.cc:660] Decoding operations
1025 07:52:44.286886 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:44.286896 (+ 10us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:44.286900 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:44.286959 (+ 59us) tablet.cc:699] Row locks acquired
1025 07:52:44.286961 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:44.321579 (+ 34618us) op_driver.cc:464] REPLICATION: finished
1025 07:52:44.321845 (+ 266us) write_op.cc:301] APPLY: starting
1025 07:52:44.321859 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:44.321890 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:44.321892 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:44.322004 (+ 112us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:44.322035 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:44.322062 (+ 27us) write_op.cc:312] APPLY: finished
1025 07:52:44.322148 (+ 86us) log.cc:844] Serialized 29 byte log entry
1025 07:52:44.322186 (+ 38us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.322200 (+ 14us) write_op.cc:454] Released schema lock
1025 07:52:44.322205 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:44.322250 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.322254 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
1025 07:52:44.287015 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:44.287081 (+ 66us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:44.287085 (+ 4us) write_op.cc:435] Acquired schema lock
1025 07:52:44.287087 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:44.287149 (+ 62us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:44.287158 (+ 9us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:44.287162 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:44.287206 (+ 44us) tablet.cc:699] Row locks acquired
1025 07:52:44.287208 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:44.321645 (+ 34437us) op_driver.cc:464] REPLICATION: finished
1025 07:52:44.322315 (+ 670us) write_op.cc:301] APPLY: starting
1025 07:52:44.322324 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:44.322351 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:44.322353 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:44.322451 (+ 98us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:44.322511 (+ 60us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:44.322538 (+ 27us) write_op.cc:312] APPLY: finished
1025 07:52:44.322588 (+ 50us) log.cc:844] Serialized 29 byte log entry
1025 07:52:44.322644 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.322658 (+ 14us) write_op.cc:454] Released schema lock
1025 07:52:44.322663 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
1025 07:52:44.287253 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:44.287312 (+ 59us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:44.287316 (+ 4us) write_op.cc:435] Acquired schema lock
1025 07:52:44.287319 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:44.287372 (+ 53us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:44.287380 (+ 8us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:44.287385 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:44.287417 (+ 32us) tablet.cc:699] Row locks acquired
1025 07:52:44.287419 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:44.391559 (+104140us) op_driver.cc:464] REPLICATION: finished
1025 07:52:44.391714 (+ 155us) write_op.cc:301] APPLY: starting
1025 07:52:44.391743 (+ 29us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:44.391793 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:44.391796 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:44.391857 (+ 61us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:44.391890 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:44.391929 (+ 39us) write_op.cc:312] APPLY: finished
1025 07:52:44.392019 (+ 90us) log.cc:844] Serialized 23 byte log entry
1025 07:52:44.392092 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.392111 (+ 19us) write_op.cc:454] Released schema lock
1025 07:52:44.392117 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:44.392171 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.392174 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
1025 07:52:44.287465 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:44.287539 (+ 74us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:44.287543 (+ 4us) write_op.cc:435] Acquired schema lock
1025 07:52:44.287545 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:44.287595 (+ 50us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:44.287604 (+ 9us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:44.287608 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:44.287652 (+ 44us) tablet.cc:699] Row locks acquired
1025 07:52:44.287654 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:44.391645 (+103991us) op_driver.cc:464] REPLICATION: finished
1025 07:52:44.392240 (+ 595us) write_op.cc:301] APPLY: starting
1025 07:52:44.392251 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:44.392285 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:44.392290 (+ 5us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:44.392394 (+ 104us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:44.392422 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:44.392462 (+ 40us) write_op.cc:312] APPLY: finished
1025 07:52:44.392518 (+ 56us) log.cc:844] Serialized 29 byte log entry
1025 07:52:44.392565 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.392590 (+ 25us) write_op.cc:454] Released schema lock
1025 07:52:44.392596 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
1025 07:52:44.287710 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:44.287772 (+ 62us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:44.287775 (+ 3us) write_op.cc:435] Acquired schema lock
1025 07:52:44.287778 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:44.287828 (+ 50us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:44.287836 (+ 8us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:44.287840 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:44.287883 (+ 43us) tablet.cc:699] Row locks acquired
1025 07:52:44.287885 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:44.394405 (+106520us) op_driver.cc:464] REPLICATION: finished
1025 07:52:44.394479 (+ 74us) write_op.cc:301] APPLY: starting
1025 07:52:44.394489 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:44.394523 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:44.394528 (+ 5us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:44.394611 (+ 83us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:44.394639 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:44.394668 (+ 29us) write_op.cc:312] APPLY: finished
1025 07:52:44.394727 (+ 59us) log.cc:844] Serialized 29 byte log entry
1025 07:52:44.394764 (+ 37us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.394775 (+ 11us) write_op.cc:454] Released schema lock
1025 07:52:44.394780 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
1025 07:52:44.287928 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:44.287995 (+ 67us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:44.287998 (+ 3us) write_op.cc:435] Acquired schema lock
1025 07:52:44.288001 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:44.288050 (+ 49us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:44.288058 (+ 8us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:44.288062 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:44.288106 (+ 44us) tablet.cc:699] Row locks acquired
1025 07:52:44.288108 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:44.394906 (+106798us) op_driver.cc:464] REPLICATION: finished
1025 07:52:44.394965 (+ 59us) write_op.cc:301] APPLY: starting
1025 07:52:44.394973 (+ 8us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:44.395000 (+ 27us) tablet.cc:1368] finished BulkCheckP
I20251025 07:52:44.454083 2146 ts_tablet_manager.cc:1929] T d5e5e5bbcf9f43118eea502b69880af5 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:44.457942 2146 log.cc:1199] T d5e5e5bbcf9f43118eea502b69880af5 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/d5e5e5bbcf9f43118eea502b69880af5
I20251025 07:52:44.458328 2146 ts_tablet_manager.cc:1950] T d5e5e5bbcf9f43118eea502b69880af5 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:44.460675 2169 ts_tablet_manager.cc:1929] T c59f6e6eb9594a8e83f109a09b034045 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:44.466827 847 leader_election.cc:304] T ebc824ec4c60411dab1a2ea4a385148e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:44.467103 2314 raft_consensus.cc:2804] T ebc824ec4c60411dab1a2ea4a385148e P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:44.467206 2314 raft_consensus.cc:697] T ebc824ec4c60411dab1a2ea4a385148e P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:44.467419 2314 consensus_queue.cc:237] T ebc824ec4c60411dab1a2ea4a385148e P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.469370 2169 log.cc:1199] T c59f6e6eb9594a8e83f109a09b034045 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/c59f6e6eb9594a8e83f109a09b034045
I20251025 07:52:44.469000 1169 tablet_bootstrap.cc:492] T 6dcab53a020847a1aa27930265ba0205 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:44.469897 1169 ts_tablet_manager.cc:1403] T 6dcab53a020847a1aa27930265ba0205 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.125s user 0.001s sys 0.004s
I20251025 07:52:44.470701 1169 raft_consensus.cc:359] T 6dcab53a020847a1aa27930265ba0205 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.471009 1169 raft_consensus.cc:385] T 6dcab53a020847a1aa27930265ba0205 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:44.471112 1169 raft_consensus.cc:740] T 6dcab53a020847a1aa27930265ba0205 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:44.471340 1169 consensus_queue.cc:260] T 6dcab53a020847a1aa27930265ba0205 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.471858 1169 ts_tablet_manager.cc:1434] T 6dcab53a020847a1aa27930265ba0205 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:44.472167 2169 ts_tablet_manager.cc:1950] T c59f6e6eb9594a8e83f109a09b034045 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:44.474577 448 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f]). Expected version 1 got 0
I20251025 07:52:44.474992 448 catalog_manager.cc:5649] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 23eeffd3de254ce18db5febfde971c6b (127.0.104.1). New cstate: current_term: 1 leader_uuid: "23eeffd3de254ce18db5febfde971c6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: HEALTHY } } }
I20251025 07:52:44.476581 2604 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "b2dfbd0b44a441c9be9254f660974944" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } } }
I20251025 07:52:44.476828 2604 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [sys.catalog]: This master's current role is: LEADER
I20251025 07:52:44.477557 2681 tablet.cc:1722] T cb829649f73d4d3f8862d268c953bd86 P db42be7ad3c741a0aa427f27e245993b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251025 07:52:44.479059 448 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7]). Expected version 1 got 0
I20251025 07:52:44.479403 448 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 04d27d54fdb7483b869408af65ff85c9 (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]). Expected version 2 got 0
I20251025 07:52:44.479743 448 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet cb829649f73d4d3f8862d268c953bd86 (table default.Table_a9026f5e095c4063b86d4c544c686e22 [id=7944225d3ce7439b881bfb9289d9f040]). Expected version 2 got 0
I20251025 07:52:44.479977 448 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 872cb7512d104827b1f1f186d33a4a44 (table default.Table_d5c65dce8b1b4214ba519d964012bfc0 [id=d4be274b36a845a99a4b96b8846248fd]). Expected version 1 got 0
I20251025 07:52:44.480877 1167 tablet_bootstrap.cc:654] T 9f039d9107ae493486d799ab6a1d731d P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:44.481096 448 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet aa9d9ab8073945d28d52fb02213f201a (table default.Table_4791672658f043f6b97b9320a531ebb5 [id=dfd20cf9c3e541c4abe23d59e0aa7f2a]). Expected version 1 got 0
I20251025 07:52:44.481346 448 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]). Expected version 1 got 0
I20251025 07:52:44.481647 448 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]). Expected version 1 got 0
I20251025 07:52:44.481928 448 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_a6c869949b744bc98efb735b0e73e7aa [id=bf77b5ff6f1048dd996d948437f9ff5e]). Expected version 2 got 0
I20251025 07:52:44.482955 448 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet bc2d16b33ca8426abbd7bd38339c9f1d (table default.Table_4ea985df4c5b415c9570c64506a98cc4 [id=98d2b7797e33485e8872054aa2eb8ec9]). Expected version 1 got 0
I20251025 07:52:44.483206 448 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e]). Expected version 1 got 0
I20251025 07:52:44.483557 448 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet ab42909afc524f2fa3c1041495d13a59 (table default.Table_7eb8424d5192485da1a2d67bfdb2bbbf [id=40c29df99590408cbd5c583d4fc741eb]). Expected version 2 got 0
I20251025 07:52:44.483974 448 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 288e543bb279410c91c01dd1623e110f (table default.Table_6e242ba55b194c23a6701f4b064fdbe5 [id=4c34b64b299f4162a53866cf01c5298f]). Expected version 1 got 0
I20251025 07:52:44.484238 448 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 5dab439bc50d4e869548154e1d222ad6 (table default.Table_fb9def2302784c999bf0f5707c60eb4c [id=45325c188eac4d34bb016d01de091f87]). Expected version 3 got 0
I20251025 07:52:44.484493 448 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_87d38832f85e4965bfab1a077990114f [id=ad812862698d421184ebf501ce79b7be]). Expected version 2 got 0
I20251025 07:52:44.484833 448 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet f70091024c034c118fb86578db19a31c (table default.Table_b7da40b96d6c4359bed6f898e8cf6275 [id=e041946c093b4c789ce175c736cc230b]). Expected version 2 got 0
I20251025 07:52:44.485481 448 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]). Expected version 3 got 0
I20251025 07:52:44.485786 448 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]). Expected version 1 got 0
I20251025 07:52:44.485994 448 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c]). Expected version 1 got 0
I20251025 07:52:44.486346 448 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 755f0d6c87044e20a8020f8410940123 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 4 got 0
I20251025 07:52:44.486567 448 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 22f7db660616457b83b24e947a0b0bae (table default.Table_4a86191163f2408f9bcc795cd8d1bbf6 [id=bb316a7e000748ca902a9bfb494802d7]). Expected version 1 got 0
I20251025 07:52:44.486783 448 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236]). Expected version 1 got 0
I20251025 07:52:44.487061 1095 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
I20251025 07:52:44.487222 448 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84]). Expected version 1 got 0
I20251025 07:52:44.487735 448 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 0325d462441349ef81841f1b0cab2a20 (table default.Table_750ecb905839434fa9a1aa72f87394be [id=9ce3ba77c56a4ef09cb3b40ea0e488f6]). Expected version 1 got 0
I20251025 07:52:44.488106 448 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5]). Expected version 1 got 0
I20251025 07:52:44.488477 448 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525]). Expected version 1 got 0
I20251025 07:52:44.488692 448 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_e90d7f5ff13b40878e5b625c97792c7d [id=fc8180c09d7b4e74b2e161f7acb8fe0e]). Expected version 1 got 0
I20251025 07:52:44.489434 448 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet b56209e4934e4407a8b4518ca4af0d46 (table default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de]). Expected version 1 got 0
I20251025 07:52:44.489662 448 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 830fdbc6812e416690d0db42a73af503 (table default.Table_92824b3f2a774564b3aea04aaa25d204 [id=fb21a45b86ab446e8c3dae4512a9598f]). Expected version 1 got 0
I20251025 07:52:44.489868 448 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 5fc092c2bb2d41ac8c5b4fc07460c8ec (table default.Table_27c62e973d74406fb00db8bcf324749c [id=630c274339664b9db18ab4ef7c29c16f]). Expected version 1 got 0
I20251025 07:52:44.491290 448 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3]). Expected version 2 got 0
I20251025 07:52:44.492486 448 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 36262ab8c6964dd684c01b4db6219a4f (table default.Table_76158f8cf82e4cdb9a90580e46aaea48 [id=f249f30497bc450c887d659721469de8]). Expected version 1 got 0
I20251025 07:52:44.492815 448 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3]). Expected version 1 got 0
I20251025 07:52:44.493491 448 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 292fe04055954afd8c08ac2636b68191 (table default.Table_9fa69b10b5e842639fc0b0a58cd9b6a5 [id=a253ce0253bd4da4814741c482193d8f]). Expected version 1 got 0
I20251025 07:52:44.494225 453 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet 60c60a9ee37f43978910bd067a098bb7
I20251025 07:52:44.494822 453 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:47610 (request call id 76) took 4148 ms. Trace:
I20251025 07:52:44.494911 453 rpcz_store.cc:276] 1025 07:52:40.346681 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:43.967095 (+3620414us) service_pool.cc:225] Handling call
1025 07:52:43.967648 (+ 553us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:43.967811 (+ 163us) catalog_manager.cc:2815] Modifying in-memory table state
1025 07:52:43.967893 (+ 82us) catalog_manager.cc:2824] Locking tablets
1025 07:52:43.983137 (+ 15244us) catalog_manager.cc:2838] Removing table and tablets from system table
1025 07:52:43.992557 (+ 9420us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:43.992564 (+ 7us) write_op.cc:454] Released schema lock
1025 07:52:43.992658 (+ 94us) catalog_manager.cc:2856] Removing table from by-name map
1025 07:52:43.992749 (+ 91us) catalog_manager.cc:2872] Committing in-memory state
1025 07:52:44.494178 (+501429us) catalog_manager.cc:2876] Aborting table tasks
1025 07:52:44.494767 (+ 589us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:43.984236 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:43.984425 (+ 189us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:43.984435 (+ 10us) write_op.cc:435] Acquired schema lock
1025 07:52:43.984439 (+ 4us) tablet.cc:660] Decoding operations
1025 07:52:43.984595 (+ 156us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:43.984615 (+ 20us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:43.984623 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:43.984756 (+ 133us) tablet.cc:699] Row locks acquired
1025 07:52:43.984759 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:43.984803 (+ 44us) write_op.cc:270] Start()
1025 07:52:43.984847 (+ 44us) write_op.cc:276] Timestamp: P: 1761378763984775 usec, L: 0
1025 07:52:43.984853 (+ 6us) op_driver.cc:348] REPLICATION: starting
1025 07:52:43.985115 (+ 262us) log.cc:844] Serialized 725 byte log entry
1025 07:52:43.989565 (+ 4450us) op_driver.cc:464] REPLICATION: finished
1025 07:52:43.989712 (+ 147us) write_op.cc:301] APPLY: starting
1025 07:52:43.989735 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:43.989798 (+ 63us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:43.989842 (+ 44us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:43.989931 (+ 89us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:43.989978 (+ 47us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1025 07:52:43.989985 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:43.990073 (+ 88us) log.cc:844] Serialized 29 byte log entry
1025 07:52:43.990163 (+ 90us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:43.990190 (+ 27us) write_op.cc:454] Released schema lock
1025 07:52:43.990199 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":88,"apply.run_cpu_time_us":550,"apply.run_wall_time_us":550,"num_ops":2,"prepare.queue_time_us":66,"prepare.run_cpu_time_us":1006,"prepare.run_wall_time_us":1006,"raft.queue_time_us":105,"raft.run_cpu_time_us":291,"raft.run_wall_time_us":291,"replication_time_us":4671}]]}
I20251025 07:52:44.495734 453 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_309913c03a4d46019d35478363f149e8" } modify_external_catalogs: true
I20251025 07:52:44.495893 453 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_309913c03a4d46019d35478363f149e8" } modify_external_catalogs: true
I20251025 07:52:44.505134 2612 raft_consensus.cc:493] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:44.505291 2612 raft_consensus.cc:515] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.506350 2612 leader_election.cc:290] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:44.506410 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a77eb82f26ae43b48de1e85693372d34" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:44.506965 850 leader_election.cc:343] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:44.507153 847 leader_election.cc:304] T ab42909afc524f2fa3c1041495d13a59 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:44.507529 2612 raft_consensus.cc:2804] T ab42909afc524f2fa3c1041495d13a59 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:44.509114 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 60c60a9ee37f43978910bd067a098bb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:43 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:44.509984 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 60c60a9ee37f43978910bd067a098bb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:43 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.510147 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 60c60a9ee37f43978910bd067a098bb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:43 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:44.510502 446 catalog_manager.cc:6194] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: default.Table_438f67f3dc62465999a917986922b3ed [id=7c7fcd265e624d7fab74661db505d9c3] alter complete (version 1)
I20251025 07:52:44.512784 1166 tablet_bootstrap.cc:492] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:44.517710 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 124) took 5989 ms. Trace:
I20251025 07:52:44.517802 2146 rpcz_store.cc:276] 1025 07:52:38.528599 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.528638 (+ 39us) service_pool.cc:225] Handling call
1025 07:52:44.312360 (+5783722us) ts_tablet_manager.cc:1151] Deleting tablet d5e5e5bbcf9f43118eea502b69880af5
1025 07:52:44.312374 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:44.436392 (+124018us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.453983 (+ 17591us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.517668 (+ 63685us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5783587}
I20251025 07:52:44.518079 2146 tablet_replica.cc:333] T 5361f08f50b745eb98bf02e979d7c71e P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:44.518258 2146 raft_consensus.cc:2243] T 5361f08f50b745eb98bf02e979d7c71e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:44.518386 2146 raft_consensus.cc:2272] T 5361f08f50b745eb98bf02e979d7c71e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:44.518894 2146 ts_tablet_manager.cc:1916] T 5361f08f50b745eb98bf02e979d7c71e P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:44.521184 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet d5e5e5bbcf9f43118eea502b69880af5 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]) successfully deleted
I20251025 07:52:44.521206 2106 raft_consensus.cc:493] T ce1ebb1ec14b4801930a72765bca1a7d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:44.521322 2106 raft_consensus.cc:515] T ce1ebb1ec14b4801930a72765bca1a7d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.521921 2106 leader_election.cc:290] T ce1ebb1ec14b4801930a72765bca1a7d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:44.522782 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce1ebb1ec14b4801930a72765bca1a7d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:44.523262 850 leader_election.cc:343] T ce1ebb1ec14b4801930a72765bca1a7d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:44.526296 2166 ts_tablet_manager.cc:1929] T fa77085e81c7440d9e255c44d1e80c26 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:44.526489 2166 log.cc:1199] T fa77085e81c7440d9e255c44d1e80c26 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/fa77085e81c7440d9e255c44d1e80c26
I20251025 07:52:44.527019 2166 ts_tablet_manager.cc:1950] T fa77085e81c7440d9e255c44d1e80c26 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:44.527840 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 109) took 5868 ms. Trace:
I20251025 07:52:44.528455 454 catalog_manager.cc:6194] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: default.Table_088aad2a46b041bfa29ed8e6929760b1 [id=834669c54dae449790818684fd8659bc] alter complete (version 2)
I20251025 07:52:44.529461 1169 tablet_bootstrap.cc:492] T b2044ca206634eb285197af4ba2073e7 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:44.531571 1169 tablet_bootstrap.cc:654] T b2044ca206634eb285197af4ba2073e7 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:44.533154 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 124) took 5719 ms. Trace:
I20251025 07:52:44.533277 2169 rpcz_store.cc:276] 1025 07:52:38.814082 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.814580 (+ 498us) service_pool.cc:225] Handling call
1025 07:52:44.374970 (+5560390us) ts_tablet_manager.cc:1151] Deleting tablet c59f6e6eb9594a8e83f109a09b034045
1025 07:52:44.374979 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:44.448180 (+ 73201us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.460531 (+ 12351us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.533135 (+ 72604us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5558849}
I20251025 07:52:44.533620 2169 tablet_replica.cc:333] T c88c46e022ff41a58b02538f2151e260 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:44.533828 2169 raft_consensus.cc:2243] T c88c46e022ff41a58b02538f2151e260 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:44.534037 2169 raft_consensus.cc:2272] T c88c46e022ff41a58b02538f2151e260 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:44.534669 2169 ts_tablet_manager.cc:1916] T c88c46e022ff41a58b02538f2151e260 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:44.534881 1167 tablet_bootstrap.cc:492] T 9f039d9107ae493486d799ab6a1d731d P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:44.535072 1167 ts_tablet_manager.cc:1403] T 9f039d9107ae493486d799ab6a1d731d P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.101s user 0.003s sys 0.002s
I20251025 07:52:44.535957 1167 raft_consensus.cc:359] T 9f039d9107ae493486d799ab6a1d731d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.536134 1167 raft_consensus.cc:385] T 9f039d9107ae493486d799ab6a1d731d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:44.536195 1167 raft_consensus.cc:740] T 9f039d9107ae493486d799ab6a1d731d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:44.536383 1167 consensus_queue.cc:260] T 9f039d9107ae493486d799ab6a1d731d P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.538759 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet c59f6e6eb9594a8e83f109a09b034045 (table default.Table_500d949c884c4644b70943543bd3e0c5 [id=71ac1bb83b444af2aede5644bf5619a3]) successfully deleted
I20251025 07:52:44.540473 1167 ts_tablet_manager.cc:1434] T 9f039d9107ae493486d799ab6a1d731d P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251025 07:52:44.540750 1167 tablet_bootstrap.cc:492] T 9a8e2658a81e436298fe2190726080d4 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:44.540793 454 catalog_manager.cc:6194] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: default.Table_3f081b7bb4e1420c8121ea4115439f3d [id=696a1285fbfe4038a8b1b38708c8ea49] alter complete (version 1)
I20251025 07:52:44.542747 1167 tablet_bootstrap.cc:654] T 9a8e2658a81e436298fe2190726080d4 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:44.543087 2169 ts_tablet_manager.cc:1929] T c88c46e022ff41a58b02538f2151e260 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:44.543220 2169 log.cc:1199] T c88c46e022ff41a58b02538f2151e260 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/c88c46e022ff41a58b02538f2151e260
I20251025 07:52:44.543661 2169 ts_tablet_manager.cc:1950] T c88c46e022ff41a58b02538f2151e260 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:44.543741 2166 rpcz_store.cc:276] 1025 07:52:38.658964 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.659006 (+ 42us) service_pool.cc:225] Handling call
1025 07:52:44.416310 (+5757304us) ts_tablet_manager.cc:1151] Deleting tablet fa77085e81c7440d9e255c44d1e80c26
1025 07:52:44.416324 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:44.453732 (+ 37408us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.506048 (+ 52316us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.527816 (+ 21768us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5757114}
I20251025 07:52:44.544028 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 125) took 5729 ms. Trace:
I20251025 07:52:44.544243 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet fa77085e81c7440d9e255c44d1e80c26 (table default.Table_936e8f1bd9344704940225361305cf5b [id=4998c2abb8fb48db83483bc0aa0170eb]) successfully deleted
I20251025 07:52:44.544144 2169 rpcz_store.cc:276] 1025 07:52:38.814821 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.815068 (+ 247us) service_pool.cc:225] Handling call
1025 07:52:44.533560 (+5718492us) ts_tablet_manager.cc:1151] Deleting tablet c88c46e022ff41a58b02538f2151e260
1025 07:52:44.533569 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:44.539483 (+ 5914us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.542977 (+ 3494us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.543967 (+ 990us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5717405}
I20251025 07:52:44.544312 2166 tablet_replica.cc:333] T 8c76d1b509f04ab1a10b83cff675c11e P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:44.544493 2166 raft_consensus.cc:2243] T 8c76d1b509f04ab1a10b83cff675c11e P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:44.544474 2611 raft_consensus.cc:493] T 11bfe0c87ca941a3846462c8a4512d83 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:44.544610 2611 raft_consensus.cc:515] T 11bfe0c87ca941a3846462c8a4512d83 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.544843 2166 raft_consensus.cc:2272] T 8c76d1b509f04ab1a10b83cff675c11e P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:44.544823 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet c88c46e022ff41a58b02538f2151e260 (table default.Table_f6e21e14e6fb4649af45c7d10a5f993e [id=c5eda563853a4e15b846d063608d42ed]) successfully deleted
I20251025 07:52:44.545454 2166 ts_tablet_manager.cc:1916] T 8c76d1b509f04ab1a10b83cff675c11e P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:44.545969 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11bfe0c87ca941a3846462c8a4512d83" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:44.546818 2611 leader_election.cc:290] T 11bfe0c87ca941a3846462c8a4512d83 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
W20251025 07:52:44.551357 452 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:42387 (request call id 212) took 4504 ms (client timeout 1000 ms). Trace:
W20251025 07:52:44.551469 452 rpcz_store.cc:269] 1025 07:52:40.046649 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:40.425518 (+378869us) service_pool.cc:225] Handling call
1025 07:52:44.551293 (+4125775us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":219022,"reported_tablets":252}
I20251025 07:52:44.553287 452 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_3b10c71d2c1b4bd2849e0cad9ec767d7" } modify_external_catalogs: true
I20251025 07:52:44.554100 452 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_3b10c71d2c1b4bd2849e0cad9ec767d7" } modify_external_catalogs: true
I20251025 07:52:44.554694 446 catalog_manager.cc:6194] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: default.Table_32b54e82cbc84d39a0020826f8737d62 [id=c28b0b6acd61490caecfdaab803a9d92] alter complete (version 2)
I20251025 07:52:44.566968 1167 tablet_bootstrap.cc:492] T 9a8e2658a81e436298fe2190726080d4 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:44.567106 1167 ts_tablet_manager.cc:1403] T 9a8e2658a81e436298fe2190726080d4 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20251025 07:52:44.567863 1167 raft_consensus.cc:359] T 9a8e2658a81e436298fe2190726080d4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.568050 1167 raft_consensus.cc:385] T 9a8e2658a81e436298fe2190726080d4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:44.568102 1167 raft_consensus.cc:740] T 9a8e2658a81e436298fe2190726080d4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:44.568272 1167 consensus_queue.cc:260] T 9a8e2658a81e436298fe2190726080d4 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.568831 1167 ts_tablet_manager.cc:1434] T 9a8e2658a81e436298fe2190726080d4 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20251025 07:52:44.569114 1167 tablet_bootstrap.cc:492] T 34c63b69904d499bb0d286941a8303f2 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:44.569746 751 tablet_service.cc:1552] Processing DeleteTablet for tablet f3001314b56345f5a70314777cae0125 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.570271 751 tablet_service.cc:1552] Processing DeleteTablet for tablet f170424be2f643e7af499834955472b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.570979 746 tablet_service.cc:1552] Processing DeleteTablet for tablet be5ab0db6ac34c2c9a370445e251eb46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.571062 751 tablet_service.cc:1552] Processing DeleteTablet for tablet a13bf4d7eb1440f7a28b85cb1bf99667 with delete_type TABLET_DATA_DELETED (replaced by tablet 74e2d78c024849b0af5c0bde1f48c36f) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.571604 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 95efd553493e4966bc48a01cac6e1e10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.571852 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 7f29b64704e147aa9b0b5f6cda860f09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.572824 2169 tablet_replica.cc:333] T 9d6a9937d10a4cc2813671dd1bfa1664 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:44.572896 753 tablet_service.cc:1552] Processing DeleteTablet for tablet c11f674385184ae6b235f1db8bf1fb91 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.573093 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e098c9c4613459295b6043048cba587 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.573122 2169 raft_consensus.cc:2243] T 9d6a9937d10a4cc2813671dd1bfa1664 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:44.573252 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f7c15529be54a11961f385a953b0b03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.573338 2169 raft_consensus.cc:2272] T 9d6a9937d10a4cc2813671dd1bfa1664 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:44.573408 741 tablet_service.cc:1552] Processing DeleteTablet for tablet 27651312cc2e42a8bfcd245f5d301679 with delete_type TABLET_DATA_DELETED (replaced by tablet a4e4b99be0434baaa75d988281797846) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.573683 746 tablet_service.cc:1552] Processing DeleteTablet for tablet fab67b56eaa44b54886c17181d8c52b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.573902 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 896bd7870d87408e8adfdb7a78998111 with delete_type TABLET_DATA_DELETED (replaced by tablet e8402fd33ba842e981c6c68787c7c548) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.573943 2169 ts_tablet_manager.cc:1916] T 9d6a9937d10a4cc2813671dd1bfa1664 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:44.574647 741 tablet_service.cc:1552] Processing DeleteTablet for tablet 1912e59548874e0e8e989050ec0064c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.574843 752 tablet_service.cc:1552] Processing DeleteTablet for tablet cc407c7eb12e4eadb41fb21385611c94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.575186 746 tablet_service.cc:1552] Processing DeleteTablet for tablet abf94c32ff2945119b710905c075d63c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.575203 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 95efd553493e4966bc48a01cac6e1e10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:44.575420 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 887e7c7a5d0e4aaf88f26246306df19c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.576009 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 6dcab53a020847a1aa27930265ba0205 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.576264 751 tablet_service.cc:1552] Processing DeleteTablet for tablet ee1c8ea694284d80a30f9c05be4cbf49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.578061 1167 tablet_bootstrap.cc:654] T 34c63b69904d499bb0d286941a8303f2 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:44.582960 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 27651312cc2e42a8bfcd245f5d301679 with delete_type TABLET_DATA_DELETED (replaced by tablet a4e4b99be0434baaa75d988281797846) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:44.583238 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e098c9c4613459295b6043048cba587 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:44.583290 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet fab67b56eaa44b54886c17181d8c52b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:44.583431 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 896bd7870d87408e8adfdb7a78998111 with delete_type TABLET_DATA_DELETED (replaced by tablet e8402fd33ba842e981c6c68787c7c548) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:44.583611 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 5fca357d7e24494fb6af4bd73d98833c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:44.583817 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet ac0ea298bc114c56a8093e12def21ce1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:44572
W20251025 07:52:44.585932 850 leader_election.cc:343] T 11bfe0c87ca941a3846462c8a4512d83 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:44.586400 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 74e2d78c024849b0af5c0bde1f48c36f with delete_type TABLET_DATA_DELETED (replaced by tablet fd265cdbf6a8440393a2b5d8240d1872) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:44.587060 883 tablet_service.cc:1552] Processing DeleteTablet for tablet c4610b0c62e741eca28c21f63cc3de81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:44.588362 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 570201e7bf05406a97c81dde071d894b with delete_type TABLET_DATA_DELETED (replaced by tablet 3a0f50c8000144a4acb5219484b01800) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:44.589440 1169 tablet_bootstrap.cc:492] T b2044ca206634eb285197af4ba2073e7 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:44.589659 1169 ts_tablet_manager.cc:1403] T b2044ca206634eb285197af4ba2073e7 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.060s user 0.004s sys 0.006s
I20251025 07:52:44.590881 1166 tablet_bootstrap.cc:654] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:44.591611 1169 raft_consensus.cc:359] T b2044ca206634eb285197af4ba2073e7 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.591794 1169 raft_consensus.cc:385] T b2044ca206634eb285197af4ba2073e7 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:44.591939 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 943ec703e65f4fd3b0c690e08c0bd226 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:44.591871 1169 raft_consensus.cc:740] T b2044ca206634eb285197af4ba2073e7 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:44.592466 1169 consensus_queue.cc:260] T b2044ca206634eb285197af4ba2073e7 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.593369 1169 ts_tablet_manager.cc:1434] T b2044ca206634eb285197af4ba2073e7 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251025 07:52:44.593819 1169 tablet_bootstrap.cc:492] T 050a5d3b5f934dd19bf2e221f2fe0780 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
W20251025 07:52:44.595501 442 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet b56209e4934e4407a8b4518ca4af0d46 (table default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet b56209e4934e4407a8b4518ca4af0d46 consensus state has no leader
I20251025 07:52:44.595708 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 943ec703e65f4fd3b0c690e08c0bd226 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:44.578646 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f25c9caa05b4da4bd56a8a21fde423c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.579016 746 tablet_service.cc:1552] Processing DeleteTablet for tablet dda0b26e4f1a44928bc4b84f333901c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.579457 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 1b5ac7e845a24b2eb5b2429d47361232 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.579764 741 tablet_service.cc:1552] Processing DeleteTablet for tablet 21e52928580c4762b0ae7152490ef955 with delete_type TABLET_DATA_DELETED (replaced by tablet 9d6a9937d10a4cc2813671dd1bfa1664) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.580183 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 570201e7bf05406a97c81dde071d894b with delete_type TABLET_DATA_DELETED (replaced by tablet 3a0f50c8000144a4acb5219484b01800) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.580619 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 5cba9c05dd424cc788febc614a34819a with delete_type TABLET_DATA_DELETED (replaced by tablet 4b28215f61574cf48c447dd01955ac77) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.581089 756 tablet_service.cc:1552] Processing DeleteTablet for tablet a7ab3150a35b4caca9119a1374d3f2f7 with delete_type TABLET_DATA_DELETED (replaced by tablet c19c97f881cb45218d4a6f6f231aa96e) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.587951 745 tablet_service.cc:1552] Processing DeleteTablet for tablet cb824bd780524832b2b60f5618a39da5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.588174 754 tablet_service.cc:1552] Processing DeleteTablet for tablet f0bdaa8cfd764f1b83f47881353d26b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.596786 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 3775a2e08d4e40bd9f57e4cda759d535 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:44.596869 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 5fca357d7e24494fb6af4bd73d98833c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:44.588315 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b21b7da1e2f47829c3f6cdaee689e8a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.588476 748 tablet_service.cc:1552] Processing DeleteTablet for tablet fa77085e81c7440d9e255c44d1e80c26 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.588605 749 tablet_service.cc:1552] Processing DeleteTablet for tablet a6ddefa1738f4ff98586900c344359d2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f0c4f0e8f014e59be9e46d2afe0be9d) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.588711 822 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
I20251025 07:52:44.597536 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c7a9d7a4fb3449bb57eafa83230a839 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:33 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:44.597635 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 02e4c4d6ffb34cecaf17beb5ba197e81 with delete_type TABLET_DATA_DELETED (replaced by tablet 3abfd52327a04ba88625f391a8658fbb) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:44.599058 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 70fe9aa2f58d4abea3b3b71066a29653 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:44.599167 878 tablet_service.cc:1552] Processing DeleteTablet for tablet c59f6e6eb9594a8e83f109a09b034045 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:44.599169 885 tablet_service.cc:1552] Processing DeleteTablet for tablet f4578ce69a0d4322944e5ce7c8a59831 with delete_type TABLET_DATA_DELETED (replaced by tablet 48b73e0c0dce4be686d05f991a21bd19) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:44.599599 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 627a10a0714f4be8bdbced503fd6388d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:33 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:44.599855 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 35ba9d23262248cda6fdf03c51be11a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 01f82980559b4662bdc09786ed429ddc) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:44.600483 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 74e2d78c024849b0af5c0bde1f48c36f with delete_type TABLET_DATA_DELETED (replaced by tablet fd265cdbf6a8440393a2b5d8240d1872) from {username='slave'} at 127.0.0.1:44572
W20251025 07:52:44.601145 446 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet b56209e4934e4407a8b4518ca4af0d46 (table default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet b56209e4934e4407a8b4518ca4af0d46 consensus state has no leader
I20251025 07:52:44.601953 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 02e4c4d6ffb34cecaf17beb5ba197e81 with delete_type TABLET_DATA_DELETED (replaced by tablet 3abfd52327a04ba88625f391a8658fbb) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:44.604550 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b594d4718344aafbdaec34040695589 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
W20251025 07:52:44.604843 446 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:42387 (request call id 213) took 3490 ms (client timeout 1000 ms). Trace:
I20251025 07:52:44.605229 878 tablet_service.cc:1552] Processing DeleteTablet for tablet f170424be2f643e7af499834955472b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:44.605458 885 tablet_service.cc:1552] Processing DeleteTablet for tablet f3001314b56345f5a70314777cae0125 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:44.605682 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 71432a6848944a0da06ef9b81f160dc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:44.607142 875 tablet_service.cc:1552] Processing DeleteTablet for tablet 570201e7bf05406a97c81dde071d894b with delete_type TABLET_DATA_DELETED (replaced by tablet 3a0f50c8000144a4acb5219484b01800) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:44.607371 894 tablet_service.cc:1552] Processing DeleteTablet for tablet a13bf4d7eb1440f7a28b85cb1bf99667 with delete_type TABLET_DATA_DELETED (replaced by tablet 74e2d78c024849b0af5c0bde1f48c36f) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:44.610497 894 tablet_service.cc:1552] Processing DeleteTablet for tablet c88c46e022ff41a58b02538f2151e260 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:59616
W20251025 07:52:44.615929 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:46231, user_credentials={real_user=slave}} blocked reactor thread for 108380us
W20251025 07:52:44.616235 847 consensus_peers.cc:597] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382 -> Peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Couldn't send request to peer 23eeffd3de254ce18db5febfde971c6b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7fbae8be73bf4e18b5a5c4ec4cdebc12. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:44.616492 847 leader_election.cc:304] T 4678f7eab5e8497db308e8a38e387618 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:44.617163 2611 raft_consensus.cc:2804] T 4678f7eab5e8497db308e8a38e387618 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:44.617292 2611 raft_consensus.cc:493] T 4678f7eab5e8497db308e8a38e387618 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:44.617398 2611 raft_consensus.cc:3060] T 4678f7eab5e8497db308e8a38e387618 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:44.618109 2146 ts_tablet_manager.cc:1929] T 5361f08f50b745eb98bf02e979d7c71e P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:44.619383 2146 log.cc:1199] T 5361f08f50b745eb98bf02e979d7c71e P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/5361f08f50b745eb98bf02e979d7c71e
I20251025 07:52:44.620457 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d6a9937d10a4cc2813671dd1bfa1664 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:44.625391 894 tablet_service.cc:1552] Processing DeleteTablet for tablet d5e5e5bbcf9f43118eea502b69880af5 with delete_type TABLET_DATA_DELETED (replaced by tablet f4578ce69a0d4322944e5ce7c8a59831) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:44.629326 2146 ts_tablet_manager.cc:1950] T 5361f08f50b745eb98bf02e979d7c71e P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:44.630468 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 125) took 6101 ms. Trace:
I20251025 07:52:44.630673 2146 rpcz_store.cc:276] 1025 07:52:38.528868 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.529312 (+ 444us) service_pool.cc:225] Handling call
1025 07:52:44.518038 (+5988726us) ts_tablet_manager.cc:1151] Deleting tablet 5361f08f50b745eb98bf02e979d7c71e
1025 07:52:44.518047 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:44.607992 (+ 89945us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.618010 (+ 10018us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.630442 (+ 12432us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5988598}
I20251025 07:52:44.631204 2146 tablet_replica.cc:333] T 53198283718d4b45b30df65b5bde96a6 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:44.632081 2146 raft_consensus.cc:2243] T 53198283718d4b45b30df65b5bde96a6 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:44.632325 2146 raft_consensus.cc:2272] T 53198283718d4b45b30df65b5bde96a6 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:44.633044 2146 ts_tablet_manager.cc:1916] T 53198283718d4b45b30df65b5bde96a6 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:44.636746 2611 raft_consensus.cc:515] T 4678f7eab5e8497db308e8a38e387618 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.638005 2611 leader_election.cc:290] T 4678f7eab5e8497db308e8a38e387618 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:44.638038 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4678f7eab5e8497db308e8a38e387618" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:44.638669 850 leader_election.cc:343] T 4678f7eab5e8497db308e8a38e387618 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:44.638928 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "91728987303b41fd840eb56f3a311c5f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:44.639163 776 raft_consensus.cc:2468] T 91728987303b41fd840eb56f3a311c5f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:44.640357 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30418858bf95438686bdfe64ac733144" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:44.640614 776 raft_consensus.cc:3060] T 30418858bf95438686bdfe64ac733144 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:44.643860 776 raft_consensus.cc:2468] T 30418858bf95438686bdfe64ac733144 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:44.644134 2597 raft_consensus.cc:493] T b94634ac5077439fadfbbce569f60e8b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:44.644315 2597 raft_consensus.cc:515] T b94634ac5077439fadfbbce569f60e8b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:44.644937 2597 leader_election.cc:290] T b94634ac5077439fadfbbce569f60e8b P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:44.646440 982 leader_election.cc:343] T b94634ac5077439fadfbbce569f60e8b P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: b94634ac5077439fadfbbce569f60e8b
I20251025 07:52:44.647606 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f656945454fe4921ab819c772a089eb3" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:44.647794 776 raft_consensus.cc:3060] T f656945454fe4921ab819c772a089eb3 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:44.648171 2467 raft_consensus.cc:493] T 6dcab53a020847a1aa27930265ba0205 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:44.648306 2467 raft_consensus.cc:515] T 6dcab53a020847a1aa27930265ba0205 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.648978 2467 leader_election.cc:290] T 6dcab53a020847a1aa27930265ba0205 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:44.650965 776 raft_consensus.cc:2468] T f656945454fe4921ab819c772a089eb3 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:44.653997 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a77eb82f26ae43b48de1e85693372d34" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:44.654285 774 raft_consensus.cc:2468] T a77eb82f26ae43b48de1e85693372d34 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:44.655897 1167 tablet_bootstrap.cc:492] T 34c63b69904d499bb0d286941a8303f2 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:44.656098 1167 ts_tablet_manager.cc:1403] T 34c63b69904d499bb0d286941a8303f2 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.087s user 0.001s sys 0.008s
I20251025 07:52:44.657248 1167 raft_consensus.cc:359] T 34c63b69904d499bb0d286941a8303f2 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.657495 1167 raft_consensus.cc:385] T 34c63b69904d499bb0d286941a8303f2 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:44.657581 1167 raft_consensus.cc:740] T 34c63b69904d499bb0d286941a8303f2 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:44.657912 1167 consensus_queue.cc:260] T 34c63b69904d499bb0d286941a8303f2 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.662554 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c7a9d7a4fb3449bb57eafa83230a839 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:33 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:44.662778 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 70fe9aa2f58d4abea3b3b71066a29653 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:44.662992 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet f4578ce69a0d4322944e5ce7c8a59831 with delete_type TABLET_DATA_DELETED (replaced by tablet 48b73e0c0dce4be686d05f991a21bd19) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:44.663216 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet f0bdaa8cfd764f1b83f47881353d26b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:44.663578 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet a7ab3150a35b4caca9119a1374d3f2f7 with delete_type TABLET_DATA_DELETED (replaced by tablet c19c97f881cb45218d4a6f6f231aa96e) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:44.663831 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 8956ad28b35d45f28c683ac055bba441 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:44.663861 1167 ts_tablet_manager.cc:1434] T 34c63b69904d499bb0d286941a8303f2 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20251025 07:52:44.664265 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 71432a6848944a0da06ef9b81f160dc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:44.664455 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet f3001314b56345f5a70314777cae0125 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:44.664686 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet f170424be2f643e7af499834955472b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:44.664866 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet a13bf4d7eb1440f7a28b85cb1bf99667 with delete_type TABLET_DATA_DELETED (replaced by tablet 74e2d78c024849b0af5c0bde1f48c36f) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:44.665024 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet c59f6e6eb9594a8e83f109a09b034045 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:44.665270 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet c88c46e022ff41a58b02538f2151e260 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:44.666263 1169 tablet_bootstrap.cc:654] T 050a5d3b5f934dd19bf2e221f2fe0780 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:44.667307 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce1ebb1ec14b4801930a72765bca1a7d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:44.667577 774 raft_consensus.cc:2468] T ce1ebb1ec14b4801930a72765bca1a7d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:44.668145 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dcab53a020847a1aa27930265ba0205" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:44.668350 909 raft_consensus.cc:2468] T 6dcab53a020847a1aa27930265ba0205 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:44.668943 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11bfe0c87ca941a3846462c8a4512d83" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:44.669126 774 raft_consensus.cc:2468] T 11bfe0c87ca941a3846462c8a4512d83 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:44.676353 2612 raft_consensus.cc:697] T ab42909afc524f2fa3c1041495d13a59 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:44.676689 2612 consensus_queue.cc:237] T ab42909afc524f2fa3c1041495d13a59 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.677803 1044 raft_consensus.cc:1275] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P db42be7ad3c741a0aa427f27e245993b [term 2 FOLLOWER]: Refusing update from remote peer 3aea2869ff814cc7b142012faa274382: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 07:52:44.681406 1167 tablet_bootstrap.cc:492] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:44.683429 1169 tablet_bootstrap.cc:492] T 050a5d3b5f934dd19bf2e221f2fe0780 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:44.612113 453 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet 4b278ce7768349e49dcd1131e404ab34
I20251025 07:52:44.666893 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 5361f08f50b745eb98bf02e979d7c71e (table default.Table_a3f50508f4b047b6837c67248417b21c [id=572bc5595be947468e24f2a9144f12a4]) successfully deleted
I20251025 07:52:44.684897 453 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:47610 (request call id 77) took 4074 ms. Trace:
I20251025 07:52:44.685032 453 rpcz_store.cc:276] 1025 07:52:40.610327 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.495535 (+3885208us) service_pool.cc:225] Handling call
1025 07:52:44.495977 (+ 442us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:44.496117 (+ 140us) catalog_manager.cc:2815] Modifying in-memory table state
1025 07:52:44.496183 (+ 66us) catalog_manager.cc:2824] Locking tablets
1025 07:52:44.606211 (+110028us) catalog_manager.cc:2838] Removing table and tablets from system table
1025 07:52:44.611906 (+ 5695us) catalog_manager.cc:2856] Removing table from by-name map
1025 07:52:44.611993 (+ 87us) catalog_manager.cc:2872] Committing in-memory state
1025 07:52:44.612047 (+ 54us) catalog_manager.cc:2876] Aborting table tasks
1025 07:52:44.684841 (+ 72794us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:44.606977 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:44.607080 (+ 103us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:44.607087 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:44.607089 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:44.607196 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:44.607210 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:44.607215 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:44.607314 (+ 99us) tablet.cc:699] Row locks acquired
1025 07:52:44.607317 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:44.607348 (+ 31us) write_op.cc:270] Start()
1025 07:52:44.607380 (+ 32us) write_op.cc:276] Timestamp: P: 1761378764607331 usec, L: 0
1025 07:52:44.607384 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:44.607577 (+ 193us) log.cc:844] Serialized 725 byte log entry
1025 07:52:44.610845 (+ 3268us) op_driver.cc:464] REPLICATION: finished
1025 07:52:44.610946 (+ 101us) write_op.cc:301] APPLY: starting
1025 07:52:44.610962 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:44.611009 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:44.611034 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:44.611100 (+ 66us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:44.611133 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1025 07:52:44.611140 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:44.611211 (+ 71us) log.cc:844] Serialized 29 byte log entry
1025 07:52:44.611293 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.611315 (+ 22us) write_op.cc:454] Released schema lock
1025 07:52:44.611321 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:44.611388 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.611391 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":55,"apply.run_cpu_time_us":434,"apply.run_wall_time_us":433,"num_ops":2,"prepare.queue_time_us":49,"prepare.run_cpu_time_us":694,"prepare.run_wall_time_us":694,"raft.queue_time_us":389,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":219,"replication_time_us":3432}]]}
I20251025 07:52:44.686517 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b278ce7768349e49dcd1131e404ab34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:44 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:44.686795 1169 ts_tablet_manager.cc:1403] T 050a5d3b5f934dd19bf2e221f2fe0780 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.093s user 0.002s sys 0.001s
I20251025 07:52:44.690299 2169 ts_tablet_manager.cc:1929] T 9d6a9937d10a4cc2813671dd1bfa1664 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:44.690407 2169 log.cc:1199] T 9d6a9937d10a4cc2813671dd1bfa1664 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/9d6a9937d10a4cc2813671dd1bfa1664
I20251025 07:52:44.690794 2169 ts_tablet_manager.cc:1950] T 9d6a9937d10a4cc2813671dd1bfa1664 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:44.691533 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 126) took 5876 ms. Trace:
I20251025 07:52:44.697057 2615 consensus_queue.cc:1048] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 07:52:44.698340 452 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet 9f039d9107ae493486d799ab6a1d731d
I20251025 07:52:44.698660 847 leader_election.cc:304] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:44.699335 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 9f039d9107ae493486d799ab6a1d731d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:44 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:44.701642 847 leader_election.cc:304] T f656945454fe4921ab819c772a089eb3 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:44.702052 2314 raft_consensus.cc:2804] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:44.702719 2314 raft_consensus.cc:493] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:44.702870 2314 raft_consensus.cc:3060] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:44.706395 2314 raft_consensus.cc:515] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.692044 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b278ce7768349e49dcd1131e404ab34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:44 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:44.693601 1169 raft_consensus.cc:359] T 050a5d3b5f934dd19bf2e221f2fe0780 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.694438 2467 raft_consensus.cc:493] T b2044ca206634eb285197af4ba2073e7 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:44.707767 2467 raft_consensus.cc:515] T b2044ca206634eb285197af4ba2073e7 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.708350 2467 leader_election.cc:290] T b2044ca206634eb285197af4ba2073e7 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:44.699013 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 9f039d9107ae493486d799ab6a1d731d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:44 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:44.708632 1169 raft_consensus.cc:385] T 050a5d3b5f934dd19bf2e221f2fe0780 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:44.708770 1169 raft_consensus.cc:740] T 050a5d3b5f934dd19bf2e221f2fe0780 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:44.709064 1169 consensus_queue.cc:260] T 050a5d3b5f934dd19bf2e221f2fe0780 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.709688 1169 ts_tablet_manager.cc:1434] T 050a5d3b5f934dd19bf2e221f2fe0780 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20251025 07:52:44.710021 1169 tablet_bootstrap.cc:492] T f661d5909657455581dcb5e6eadd3c0b P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:44.711616 1169 tablet_bootstrap.cc:654] T f661d5909657455581dcb5e6eadd3c0b P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:44.706786 2615 raft_consensus.cc:2804] T f656945454fe4921ab819c772a089eb3 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:44.712098 2615 raft_consensus.cc:697] T f656945454fe4921ab819c772a089eb3 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:44.712422 2615 consensus_queue.cc:237] T f656945454fe4921ab819c772a089eb3 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.707335 847 leader_election.cc:304] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:44.716403 2687 raft_consensus.cc:2804] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:44.716504 2687 raft_consensus.cc:493] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:44.716586 2687 raft_consensus.cc:3060] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:44.717558 2106 consensus_queue.cc:1048] T f661d5909657455581dcb5e6eadd3c0b P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:44.718075 847 leader_election.cc:304] T ce1ebb1ec14b4801930a72765bca1a7d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:44.718436 847 leader_election.cc:304] T 11bfe0c87ca941a3846462c8a4512d83 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:44.718624 452 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:47610 (request call id 78) took 4085 ms. Trace:
I20251025 07:52:44.718781 452 rpcz_store.cc:276] 1025 07:52:40.633237 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.553094 (+3919857us) service_pool.cc:225] Handling call
1025 07:52:44.554277 (+ 1183us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:44.554428 (+ 151us) catalog_manager.cc:2815] Modifying in-memory table state
1025 07:52:44.554488 (+ 60us) catalog_manager.cc:2824] Locking tablets
1025 07:52:44.672389 (+117901us) catalog_manager.cc:2838] Removing table and tablets from system table
1025 07:52:44.694461 (+ 22072us) catalog_manager.cc:2856] Removing table from by-name map
1025 07:52:44.698204 (+ 3743us) catalog_manager.cc:2872] Committing in-memory state
1025 07:52:44.698265 (+ 61us) catalog_manager.cc:2876] Aborting table tasks
1025 07:52:44.718582 (+ 20317us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:44.673132 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:44.673249 (+ 117us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:44.673256 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:44.673258 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:44.673369 (+ 111us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:44.673383 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:44.673388 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:44.673496 (+ 108us) tablet.cc:699] Row locks acquired
1025 07:52:44.673498 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:44.673532 (+ 34us) write_op.cc:270] Start()
1025 07:52:44.673566 (+ 34us) write_op.cc:276] Timestamp: P: 1761378764673513 usec, L: 0
1025 07:52:44.673569 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:44.673751 (+ 182us) log.cc:844] Serialized 725 byte log entry
1025 07:52:44.693637 (+ 19886us) op_driver.cc:464] REPLICATION: finished
1025 07:52:44.693772 (+ 135us) write_op.cc:301] APPLY: starting
1025 07:52:44.693790 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:44.693841 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:44.693888 (+ 47us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:44.693982 (+ 94us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:44.694026 (+ 44us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1025 07:52:44.694033 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:44.694126 (+ 93us) log.cc:844] Serialized 29 byte log entry
1025 07:52:44.694223 (+ 97us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.694246 (+ 23us) write_op.cc:454] Released schema lock
1025 07:52:44.694253 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:44.694331 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.694334 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":80,"apply.run_cpu_time_us":542,"apply.run_wall_time_us":549,"num_ops":2,"prepare.queue_time_us":48,"prepare.run_cpu_time_us":709,"prepare.run_wall_time_us":710,"raft.queue_time_us":438,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":204,"replication_time_us":20031}]]}
I20251025 07:52:44.719743 2106 raft_consensus.cc:2804] T ce1ebb1ec14b4801930a72765bca1a7d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:44.719666 2687 raft_consensus.cc:515] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.719797 2106 raft_consensus.cc:493] T ce1ebb1ec14b4801930a72765bca1a7d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:44.719851 2106 raft_consensus.cc:3060] T ce1ebb1ec14b4801930a72765bca1a7d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:44.720428 2105 raft_consensus.cc:2804] T 11bfe0c87ca941a3846462c8a4512d83 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:44.720499 2105 raft_consensus.cc:493] T 11bfe0c87ca941a3846462c8a4512d83 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:44.720552 2105 raft_consensus.cc:3060] T 11bfe0c87ca941a3846462c8a4512d83 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:44.722844 2106 raft_consensus.cc:515] T ce1ebb1ec14b4801930a72765bca1a7d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.723855 2687 leader_election.cc:290] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:44.724994 2105 raft_consensus.cc:515] T 11bfe0c87ca941a3846462c8a4512d83 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.725138 894 tablet_service.cc:1505] Processing CreateTablet for tablet 9c389d798cb94d8a972683c57b1f9f1f (DEFAULT_TABLE table=default.Table_e386bc02f89041418ff3ddf8ac224eea [id=e800c96844e843b7a6fd3145907e5503]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:44.725510 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c389d798cb94d8a972683c57b1f9f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:44.706414 1166 tablet_bootstrap.cc:492] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:44.726989 1166 ts_tablet_manager.cc:1403] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.214s user 0.008s sys 0.005s
I20251025 07:52:44.727790 1166 raft_consensus.cc:359] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.728009 1166 raft_consensus.cc:385] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:44.728094 1166 raft_consensus.cc:740] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:44.728294 1166 consensus_queue.cc:260] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.729094 1166 ts_tablet_manager.cc:1434] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:44.729383 1166 tablet_bootstrap.cc:492] T c0c6324cbae34dcebade3ef9f89a7150 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:44.734280 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11bfe0c87ca941a3846462c8a4512d83" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:44.734692 2105 leader_election.cc:290] T 11bfe0c87ca941a3846462c8a4512d83 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
W20251025 07:52:44.735085 850 leader_election.cc:343] T 11bfe0c87ca941a3846462c8a4512d83 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:44.735819 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce1ebb1ec14b4801930a72765bca1a7d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:44.736274 850 leader_election.cc:343] T ce1ebb1ec14b4801930a72765bca1a7d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:44.736763 2314 leader_election.cc:290] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:44.711735 2169 rpcz_store.cc:276] 1025 07:52:38.815334 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.815557 (+ 223us) service_pool.cc:225] Handling call
1025 07:52:44.572732 (+5757175us) ts_tablet_manager.cc:1151] Deleting tablet 9d6a9937d10a4cc2813671dd1bfa1664
1025 07:52:44.572742 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:44.622992 (+ 50250us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.690188 (+ 67196us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.691513 (+ 1325us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5756540}
I20251025 07:52:44.722725 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a77eb82f26ae43b48de1e85693372d34" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:44.730465 1017 tablet_service.cc:1505] Processing CreateTablet for tablet 9c389d798cb94d8a972683c57b1f9f1f (DEFAULT_TABLE table=default.Table_e386bc02f89041418ff3ddf8ac224eea [id=e800c96844e843b7a6fd3145907e5503]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:44.737358 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c389d798cb94d8a972683c57b1f9f1f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:44.737629 710 consensus_peers.cc:597] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d6a9937d10a4cc2813671dd1bfa1664. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:44.738247 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "91728987303b41fd840eb56f3a311c5f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:44.741557 446 rpcz_store.cc:269] 1025 07:52:41.114020 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.621609 (+507589us) service_pool.cc:225] Handling call
1025 07:52:42.745780 (+1124171us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:42.745786 (+ 6us) write_op.cc:454] Released schema lock
1025 07:52:43.226913 (+481127us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:43.226922 (+ 9us) write_op.cc:454] Released schema lock
1025 07:52:43.716630 (+489708us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:43.716634 (+ 4us) write_op.cc:454] Released schema lock
1025 07:52:44.604801 (+888167us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:42.728218 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:42.728344 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:42.728350 (+ 6us) write_op.cc:435] Acquired schema lock
1025 07:52:42.728353 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:42.728530 (+ 177us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:42.728542 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:42.728547 (+ 5us) tablet.cc:683] Acquiring locks for 3 operations
1025 07:52:42.728633 (+ 86us) tablet.cc:699] Row locks acquired
1025 07:52:42.728637 (+ 4us) write_op.cc:260] PREPARE: finished
1025 07:52:42.728678 (+ 41us) write_op.cc:270] Start()
1025 07:52:42.728708 (+ 30us) write_op.cc:276] Timestamp: P: 1761378762728661 usec, L: 0
1025 07:52:42.728712 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:42.728922 (+ 210us) log.cc:844] Serialized 1275 byte log entry
1025 07:52:42.732502 (+ 3580us) op_driver.cc:464] REPLICATION: finished
1025 07:52:42.732780 (+ 278us) write_op.cc:301] APPLY: starting
1025 07:52:42.732795 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:42.732866 (+ 71us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:42.732898 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:42.732976 (+ 78us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:42.733010 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
1025 07:52:42.733017 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:42.733089 (+ 72us) log.cc:844] Serialized 35 byte log entry
1025 07:52:42.733144 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:42.733163 (+ 19us) write_op.cc:454] Released schema lock
1025 07:52:42.733169 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
1025 07:52:43.203897 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:43.204017 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:43.204024 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:43.204026 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:43.204128 (+ 102us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:43.204141 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:43.204145 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:43.204207 (+ 62us) tablet.cc:699] Row locks acquired
1025 07:52:43.204210 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:43.204245 (+ 35us) write_op.cc:270] Start()
1025 07:52:43.204275 (+ 30us) write_op.cc:276] Timestamp: P: 1761378763204229 usec, L: 0
1025 07:52:43.204281 (+ 6us) op_driver.cc:348] REPLICATION: starting
1025 07:52:43.204468 (+ 187us) log.cc:844] Serialized 356 byte log entry
1025 07:52:43.208071 (+ 3603us) op_driver.cc:464] REPLICATION: finished
1025 07:52:43.208180 (+ 109us) write_op.cc:301] APPLY: starting
1025 07:52:43.208194 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:43.208238 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:43.208268 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:43.208317 (+ 49us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:43.208360 (+ 43us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:43.208368 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:43.208435 (+ 67us) log.cc:844] Serialized 23 byte log entry
1025 07:52:43.208494 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:43.208511 (+ 17us) write_op.cc:454] Released schema lock
1025 07:52:43.208517 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
1025 07:52:43.708768 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:43.708916 (+ 148us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:43.708923 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:43.708926 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:43.709045 (+ 119us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:43.709061 (+ 16us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:43.709067 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:43.709133 (+ 66us) tablet.cc:699] Row locks acquired
1025 07:52:43.709135 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:43.709178 (+ 43us) write_op.cc:270] Start()
1025 07:52:43.709214 (+ 36us) write_op.cc:276] Timestamp: P: 1761378763709160 usec, L: 0
1025 07:52:43.709218 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:43.709422 (+ 204us) log.cc:844] Serialized 356 byte log entry
1025 07:52:43.713314 (+ 3892us) op_driver.cc:464] REPLICATION: finished
1025 07:52:43.715891 (+ 2577us) write_op.cc:301] APPLY: starting
1025 07:52:43.715908 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:43.715946 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:43.716007 (+ 61us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:43.716068 (+ 61us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:43.716107 (+ 39us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:43.716119 (+ 12us) write_op.cc:312] APPLY: finished
1025 07:52:43.716198 (+ 79us) log.cc:844] Serialized 23 byte log entry
1025 07:52:43.716441 (+ 243us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:43.716460 (+ 19us) write_op.cc:454] Released schema lock
1025 07:52:43.716467 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
1025 07:52:44.519284 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:44.519400 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:44.519408 (+ 8us) write_op.cc:435] Acquired schema lock
1025 07:52:44.519412 (+ 4us) tablet.cc:660] Decoding operations
1025 07:52:44.519513 (+ 101us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:44.519526 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:44.519531 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:44.519588 (+ 57us) tablet.cc:699] Row locks acquired
1025 07:52:44.519591 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:44.519630 (+ 39us) write_op.cc:270] Start()
1025 07:52:44.519662 (+ 32us) write_op.cc:276] Timestamp: P: 1761378764519611 usec, L: 0
1025 07:52:44.519666 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:44.519881 (+ 215us) log.cc:844] Serialized 356 byte log entry
1025 07:52:44.526398 (+ 6517us) op_driver.cc:464] REPLICATION: finished
1025 07:52:44.526518 (+ 120us) write_op.cc:301] APPLY: starting
1025 07:52:44.526537 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:44.526577 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:44.526599 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:44.526660 (+ 61us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:44.526717 (+ 57us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:44.526729 (+ 12us) write_op.cc:312] APPLY: finished
1025 07:52:44.526826 (+ 97us) log.cc:844] Serialized 23 byte log entry
1025 07:52:44.526898 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.526921 (+ 23us) write_op.cc:454] Released schema lock
1025 07:52:44.526930 (+ 9us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:44.527017 (+ 87us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.527023 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":58770,"reported_tablets":248,"child_traces":[["op",{"apply.queue_time_us":215,"apply.run_cpu_time_us":452,"apply.run_wall_time_us":452,"num_ops":3,"prepare.queue_time_us":8495,"prepare.run_cpu_time_us":819,"prepare.run_wall_time_us":819,"raft.queue_time_us":380,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":227,"replication_time_us":3746}],["op",{"apply.queue_time_us":63,"apply.run_cpu_time_us":390,"apply.run_wall_time_us":389,"num_ops":1,"prepare.queue_time_us":143,"prepare.run_cpu_time_us":684,"prepare.run_wall_time_us":684,"raft.queue_time_us":443,"raft.run_cpu_time_us":269,"raft.run_wall_time_us":269,"replication_time_us":3764}],["op",{"apply.queue_time_us":2506,"apply.run_cpu_time_us":464,"apply.run_wall_time_us":632,"num_ops":1,"prepare.queue_time_us":172,"prepare.run_cpu_time_us":788,"prepare.run_wall_time_us":6120,"raft.queue_time_us":42,"raft.run_cpu_time_us":249,"raft.run_wall_time_us":249,"replication_time_us":4067}],["op",{"apply.queue_time_us":69,"apply.run_cpu_time_us":486,"apply.run_wall_time_us":485,"num_ops":1,"prepare.queue_time_us":6141,"prepare.run_cpu_time_us":718,"prepare.run_wall_time_us":718,"raft.queue_time_us":1718,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":252,"replication_time_us":6699}]]}
I20251025 07:52:44.744760 2169 tablet_replica.cc:333] T d5e5e5bbcf9f43118eea502b69880af5 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:44.744863 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 9d6a9937d10a4cc2813671dd1bfa1664 (table default.Table_e7d32e91b9c64ce1b3fcff3d806e959f [id=bab869e5001c4d999daa22083fca128a]) successfully deleted
W20251025 07:52:44.737382 850 leader_election.cc:343] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:44.746928 850 leader_election.cc:343] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:44.747045 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b278ce7768349e49dcd1131e404ab34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:44 UTC) from {username='slave'} at 127.0.0.1:58412
W20251025 07:52:44.747337 710 consensus_peers.cc:597] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac0ea298bc114c56a8093e12def21ce1. This is attempt 6: this message will repeat every 5th retry.
I20251025 07:52:44.747606 2169 raft_consensus.cc:2243] T d5e5e5bbcf9f43118eea502b69880af5 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:44.747913 2169 raft_consensus.cc:2272] T d5e5e5bbcf9f43118eea502b69880af5 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:44.748433 2169 ts_tablet_manager.cc:1916] T d5e5e5bbcf9f43118eea502b69880af5 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:44.749784 2146 ts_tablet_manager.cc:1929] T 53198283718d4b45b30df65b5bde96a6 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:44.750818 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 9f039d9107ae493486d799ab6a1d731d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:44 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.751111 765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4678f7eab5e8497db308e8a38e387618" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:44.751818 1167 tablet_bootstrap.cc:654] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:44.751853 1017 tablet_service.cc:1505] Processing CreateTablet for tablet 079f91be4217493d8ab865f5ccce0cb1 (DEFAULT_TABLE table=default.Table_1de0b151cf4c4c10aca35e45623caf2c [id=a947f21398254a53bc6668c4f8d39e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:44.752225 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 079f91be4217493d8ab865f5ccce0cb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:44.754901 765 raft_consensus.cc:3060] T 4678f7eab5e8497db308e8a38e387618 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:44.667038 447 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:38361 (request call id 211) took 3683 ms (client timeout 1000 ms). Trace:
W20251025 07:52:44.675316 455 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:35777 (request call id 193) took 3577 ms (client timeout 1000 ms). Trace:
W20251025 07:52:44.756314 455 rpcz_store.cc:269] 1025 07:52:41.097994 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.614590 (+516596us) service_pool.cc:225] Handling call
1025 07:52:44.675278 (+3060688us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9171,"reported_tablets":266}
I20251025 07:52:44.757922 455 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_c43ac1adc76d4ccf819411ec76775932" } modify_external_catalogs: true
I20251025 07:52:44.758154 455 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_c43ac1adc76d4ccf819411ec76775932" } modify_external_catalogs: true
I20251025 07:52:44.764820 2043 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" instance_seqno: 1761378743122862) as {username='slave'} at 127.0.104.1:60303; Asking this server to re-register.
I20251025 07:52:44.765128 1169 tablet_bootstrap.cc:492] T f661d5909657455581dcb5e6eadd3c0b P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:44.765254 1169 ts_tablet_manager.cc:1403] T f661d5909657455581dcb5e6eadd3c0b P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.055s user 0.003s sys 0.000s
I20251025 07:52:44.765936 455 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet 29dbaf9149134f548a974a5779f7da67
I20251025 07:52:44.765997 1169 raft_consensus.cc:359] T f661d5909657455581dcb5e6eadd3c0b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.766187 1169 raft_consensus.cc:385] T f661d5909657455581dcb5e6eadd3c0b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:44.766259 1169 raft_consensus.cc:740] T f661d5909657455581dcb5e6eadd3c0b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:44.766407 2040 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" instance_seqno: 1761378744097060) as {username='slave'} at 127.0.104.3:39189; Asking this server to re-register.
I20251025 07:52:44.766433 1169 consensus_queue.cc:260] T f661d5909657455581dcb5e6eadd3c0b P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.767089 455 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:47610 (request call id 79) took 4084 ms. Trace:
I20251025 07:52:44.767210 455 rpcz_store.cc:276] 1025 07:52:40.682623 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.757583 (+4074960us) service_pool.cc:225] Handling call
1025 07:52:44.758305 (+ 722us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:44.758465 (+ 160us) catalog_manager.cc:2815] Modifying in-memory table state
1025 07:52:44.758527 (+ 62us) catalog_manager.cc:2824] Locking tablets
1025 07:52:44.758669 (+ 142us) catalog_manager.cc:2838] Removing table and tablets from system table
1025 07:52:44.765698 (+ 7029us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.765703 (+ 5us) write_op.cc:454] Released schema lock
1025 07:52:44.765770 (+ 67us) catalog_manager.cc:2856] Removing table from by-name map
1025 07:52:44.765838 (+ 68us) catalog_manager.cc:2872] Committing in-memory state
1025 07:52:44.765873 (+ 35us) catalog_manager.cc:2876] Aborting table tasks
1025 07:52:44.767017 (+ 1144us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:44.759466 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:44.759568 (+ 102us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:44.759574 (+ 6us) write_op.cc:435] Acquired schema lock
1025 07:52:44.759577 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:44.759705 (+ 128us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:44.759718 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:44.759724 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:44.759826 (+ 102us) tablet.cc:699] Row locks acquired
1025 07:52:44.759829 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:44.759864 (+ 35us) write_op.cc:270] Start()
1025 07:52:44.759898 (+ 34us) write_op.cc:276] Timestamp: P: 1761378764759844 usec, L: 0
1025 07:52:44.759913 (+ 15us) op_driver.cc:348] REPLICATION: starting
1025 07:52:44.760083 (+ 170us) log.cc:844] Serialized 725 byte log entry
1025 07:52:44.763540 (+ 3457us) op_driver.cc:464] REPLICATION: finished
1025 07:52:44.763643 (+ 103us) write_op.cc:301] APPLY: starting
1025 07:52:44.763657 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:44.763711 (+ 54us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:44.763734 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:44.763793 (+ 59us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:44.763821 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1025 07:52:44.763828 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:44.763887 (+ 59us) log.cc:844] Serialized 29 byte log entry
1025 07:52:44.763949 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.763968 (+ 19us) write_op.cc:454] Released schema lock
1025 07:52:44.763974 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":47,"apply.run_cpu_time_us":393,"apply.run_wall_time_us":392,"num_ops":2,"prepare.queue_time_us":48,"prepare.run_cpu_time_us":711,"prepare.run_wall_time_us":711,"raft.queue_time_us":404,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":229,"replication_time_us":3599}]]}
I20251025 07:52:44.768958 455 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_00b3b394e5ed46faaac0202afe54b437" } modify_external_catalogs: true
I20251025 07:52:44.769232 455 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_00b3b394e5ed46faaac0202afe54b437" } modify_external_catalogs: true
I20251025 07:52:44.770478 2166 ts_tablet_manager.cc:1929] T 8c76d1b509f04ab1a10b83cff675c11e P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:44.770634 2166 log.cc:1199] T 8c76d1b509f04ab1a10b83cff675c11e P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/8c76d1b509f04ab1a10b83cff675c11e
I20251025 07:52:44.773758 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 29dbaf9149134f548a974a5779f7da67 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:44 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:44.783737 2169 ts_tablet_manager.cc:1929] T d5e5e5bbcf9f43118eea502b69880af5 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251025 07:52:44.755648 447 rpcz_store.cc:269] 1025 07:52:40.983682 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.362869 (+379187us) service_pool.cc:225] Handling call
1025 07:52:44.666984 (+3304115us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:42.384052 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:42.384193 (+ 141us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:42.384200 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:42.384202 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:42.384400 (+ 198us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:42.384414 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:42.384420 (+ 6us) tablet.cc:683] Acquiring locks for 5 operations
1025 07:52:42.384587 (+ 167us) tablet.cc:699] Row locks acquired
1025 07:52:42.384590 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:42.384634 (+ 44us) write_op.cc:270] Start()
1025 07:52:42.384672 (+ 38us) write_op.cc:276] Timestamp: P: 1761378762384616 usec, L: 0
1025 07:52:42.384677 (+ 5us) op_driver.cc:348] REPLICATION: starting
1025 07:52:42.384914 (+ 237us) log.cc:844] Serialized 2024 byte log entry
1025 07:52:42.388601 (+ 3687us) op_driver.cc:464] REPLICATION: finished
1025 07:52:42.388730 (+ 129us) write_op.cc:301] APPLY: starting
1025 07:52:42.388746 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:42.388808 (+ 62us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:42.388835 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:42.388979 (+ 144us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:42.389017 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
1025 07:52:42.389024 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:42.389112 (+ 88us) log.cc:844] Serialized 47 byte log entry
1025 07:52:42.389188 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:42.389215 (+ 27us) write_op.cc:454] Released schema lock
1025 07:52:42.389223 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:42.389298 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:42.389303 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":192003,"reported_tablets":264,"child_traces":[["op",{"apply.queue_time_us":74,"apply.run_cpu_time_us":560,"apply.run_wall_time_us":559,"num_ops":5,"prepare.queue_time_us":3708,"prepare.run_cpu_time_us":984,"prepare.run_wall_time_us":985,"raft.queue_time_us":436,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":277,"replication_time_us":3889}]]}
I20251025 07:52:44.789594 748 tablet_service.cc:1505] Processing CreateTablet for tablet 9c389d798cb94d8a972683c57b1f9f1f (DEFAULT_TABLE table=default.Table_e386bc02f89041418ff3ddf8ac224eea [id=e800c96844e843b7a6fd3145907e5503]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:44.790372 765 raft_consensus.cc:2468] T 4678f7eab5e8497db308e8a38e387618 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:44.790539 447 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_307630cf39c04c02a5f618b1e66a9169"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:44.791178 447 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_307630cf39c04c02a5f618b1e66a9169 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:44.792486 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 29dbaf9149134f548a974a5779f7da67 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:44 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.792832 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c389d798cb94d8a972683c57b1f9f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:44.796183 749 tablet_service.cc:1505] Processing CreateTablet for tablet 55a4a524676b4b9aa9464e934b662f30 (DEFAULT_TABLE table=default.Table_f16b7a3b7d91423f8502d73e4234ab4a [id=cc7db0a7eaea4caa8c17281a8079c372]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:44.796835 2166 ts_tablet_manager.cc:1950] T 8c76d1b509f04ab1a10b83cff675c11e P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:44.797675 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 110) took 6137 ms. Trace:
I20251025 07:52:44.798347 455 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet 11bfe0c87ca941a3846462c8a4512d83
I20251025 07:52:44.798673 743 tablet_service.cc:1505] Processing CreateTablet for tablet 079f91be4217493d8ab865f5ccce0cb1 (DEFAULT_TABLE table=default.Table_1de0b151cf4c4c10aca35e45623caf2c [id=a947f21398254a53bc6668c4f8d39e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:44.799088 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55a4a524676b4b9aa9464e934b662f30. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:44.799469 455 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:47610 (request call id 80) took 3229 ms. Trace:
I20251025 07:52:44.799882 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 11bfe0c87ca941a3846462c8a4512d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:44 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.799588 455 rpcz_store.cc:276] 1025 07:52:41.569828 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.768729 (+3198901us) service_pool.cc:225] Handling call
1025 07:52:44.769380 (+ 651us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:44.769535 (+ 155us) catalog_manager.cc:2815] Modifying in-memory table state
1025 07:52:44.769614 (+ 79us) catalog_manager.cc:2824] Locking tablets
1025 07:52:44.769828 (+ 214us) catalog_manager.cc:2838] Removing table and tablets from system table
1025 07:52:44.797977 (+ 28149us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.797985 (+ 8us) write_op.cc:454] Released schema lock
1025 07:52:44.798079 (+ 94us) catalog_manager.cc:2856] Removing table from by-name map
1025 07:52:44.798220 (+ 141us) catalog_manager.cc:2872] Committing in-memory state
1025 07:52:44.798273 (+ 53us) catalog_manager.cc:2876] Aborting table tasks
1025 07:52:44.799432 (+ 1159us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:44.770492 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:44.770609 (+ 117us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:44.770614 (+ 5us) write_op.cc:435] Acquired schema lock
1025 07:52:44.770616 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:44.770715 (+ 99us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:44.770727 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:44.770731 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:44.770829 (+ 98us) tablet.cc:699] Row locks acquired
1025 07:52:44.770832 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:44.770862 (+ 30us) write_op.cc:270] Start()
1025 07:52:44.770890 (+ 28us) write_op.cc:276] Timestamp: P: 1761378764770846 usec, L: 0
1025 07:52:44.770893 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:44.771046 (+ 153us) log.cc:844] Serialized 725 byte log entry
1025 07:52:44.776895 (+ 5849us) op_driver.cc:464] REPLICATION: finished
1025 07:52:44.777225 (+ 330us) write_op.cc:301] APPLY: starting
1025 07:52:44.777241 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:44.777283 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:44.777313 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:44.777376 (+ 63us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:44.777407 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1025 07:52:44.777415 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:44.777490 (+ 75us) log.cc:844] Serialized 29 byte log entry
1025 07:52:44.777547 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.777568 (+ 21us) write_op.cc:454] Released schema lock
1025 07:52:44.777575 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":268,"apply.run_cpu_time_us":399,"apply.run_wall_time_us":399,"num_ops":2,"prepare.queue_time_us":41,"prepare.run_cpu_time_us":635,"prepare.run_wall_time_us":636,"raft.queue_time_us":1623,"raft.run_cpu_time_us":285,"raft.run_wall_time_us":286,"replication_time_us":5944}]]}
I20251025 07:52:44.800150 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 079f91be4217493d8ab865f5ccce0cb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:44.800684 823 heartbeater.cc:461] Registering TS with master...
I20251025 07:52:44.800558 455 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_0c3a15ef44874a679e54197a65dc10c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:44.801169 455 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c3a15ef44874a679e54197a65dc10c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:44.784266 1169 ts_tablet_manager.cc:1434] T f661d5909657455581dcb5e6eadd3c0b P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20251025 07:52:44.802042 1169 tablet_bootstrap.cc:492] T 36c8fcafd5464b4e999276a8062a812a P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:44.799077 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 11bfe0c87ca941a3846462c8a4512d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:44 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:44.792883 1017 tablet_service.cc:1505] Processing CreateTablet for tablet 55a4a524676b4b9aa9464e934b662f30 (DEFAULT_TABLE table=default.Table_f16b7a3b7d91423f8502d73e4234ab4a [id=cc7db0a7eaea4caa8c17281a8079c372]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:44.802615 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55a4a524676b4b9aa9464e934b662f30. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:44.745384 2106 leader_election.cc:290] T ce1ebb1ec14b4801930a72765bca1a7d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:44.755986 894 tablet_service.cc:1505] Processing CreateTablet for tablet 079f91be4217493d8ab865f5ccce0cb1 (DEFAULT_TABLE table=default.Table_1de0b151cf4c4c10aca35e45623caf2c [id=a947f21398254a53bc6668c4f8d39e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:44.803349 2146 log.cc:1199] T 53198283718d4b45b30df65b5bde96a6 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/53198283718d4b45b30df65b5bde96a6
I20251025 07:52:44.743928 1166 tablet_bootstrap.cc:654] T c0c6324cbae34dcebade3ef9f89a7150 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:44.803668 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 079f91be4217493d8ab865f5ccce0cb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:44.809965 2420 raft_consensus.cc:493] T 34c63b69904d499bb0d286941a8303f2 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:44.810256 1169 tablet_bootstrap.cc:654] T 36c8fcafd5464b4e999276a8062a812a P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:44.810217 2420 raft_consensus.cc:515] T 34c63b69904d499bb0d286941a8303f2 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.810899 2420 leader_election.cc:290] T 34c63b69904d499bb0d286941a8303f2 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:44.814312 2169 log.cc:1199] T d5e5e5bbcf9f43118eea502b69880af5 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/d5e5e5bbcf9f43118eea502b69880af5
I20251025 07:52:44.814975 875 tablet_service.cc:1552] Processing DeleteTablet for tablet 29dbaf9149134f548a974a5779f7da67 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:44 UTC) from {username='slave'} at 127.0.0.1:59616
W20251025 07:52:44.815125 850 consensus_peers.cc:597] T 36c8fcafd5464b4e999276a8062a812a P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:44.798568 2166 rpcz_store.cc:276] 1025 07:52:38.659690 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.659743 (+ 53us) service_pool.cc:225] Handling call
1025 07:52:44.544266 (+5884523us) ts_tablet_manager.cc:1151] Deleting tablet 8c76d1b509f04ab1a10b83cff675c11e
1025 07:52:44.544277 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:44.563983 (+ 19706us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.753776 (+189793us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.797622 (+ 43846us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5884320}
I20251025 07:52:44.818198 2166 tablet_replica.cc:333] T b772025852794a49b072f37cccbd3291 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:44.818440 2166 raft_consensus.cc:2243] T b772025852794a49b072f37cccbd3291 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:44.818615 2166 raft_consensus.cc:2272] T b772025852794a49b072f37cccbd3291 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:44.819204 2166 ts_tablet_manager.cc:1916] T b772025852794a49b072f37cccbd3291 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:44.819546 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 11bfe0c87ca941a3846462c8a4512d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:44 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:44.819662 875 tablet_service.cc:1505] Processing CreateTablet for tablet 55a4a524676b4b9aa9464e934b662f30 (DEFAULT_TABLE table=default.Table_f16b7a3b7d91423f8502d73e4234ab4a [id=cc7db0a7eaea4caa8c17281a8079c372]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:44.820011 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55a4a524676b4b9aa9464e934b662f30. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:44.822438 2420 raft_consensus.cc:493] T 7e6544202d4f489d82f22d6073ef240d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:44.822602 2420 raft_consensus.cc:515] T 7e6544202d4f489d82f22d6073ef240d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.823315 2420 leader_election.cc:290] T 7e6544202d4f489d82f22d6073ef240d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:44.822436 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 8c76d1b509f04ab1a10b83cff675c11e (table default.Table_cf686e1101314426a3ab24fd46a8974f [id=13fbed1c733c42e2b45e4ca69580b06d]) successfully deleted
I20251025 07:52:44.818251 1046 raft_consensus.cc:3060] T bc2d16b33ca8426abbd7bd38339c9f1d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:44.831282 2146 ts_tablet_manager.cc:1950] T 53198283718d4b45b30df65b5bde96a6 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:44.832242 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 126) took 6302 ms. Trace:
I20251025 07:52:44.832420 2146 rpcz_store.cc:276] 1025 07:52:38.529568 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.529843 (+ 275us) service_pool.cc:225] Handling call
1025 07:52:44.631142 (+6101299us) ts_tablet_manager.cc:1151] Deleting tablet 53198283718d4b45b30df65b5bde96a6
1025 07:52:44.631157 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:44.683273 (+ 52116us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.749687 (+ 66414us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.832220 (+ 82533us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6101157}
I20251025 07:52:44.833175 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 53198283718d4b45b30df65b5bde96a6 (table default.Table_77761ef857464942be15bf6e67127d6d [id=d9d0e8bb796a45a8b17336c04b12cd01]) successfully deleted
I20251025 07:52:44.833824 2146 tablet_replica.cc:333] T 6ec421eb820b4f40afcff5a5bf80c508 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:44.834327 2146 raft_consensus.cc:2243] T 6ec421eb820b4f40afcff5a5bf80c508 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:44.834594 2146 raft_consensus.cc:2272] T 6ec421eb820b4f40afcff5a5bf80c508 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:44.835420 2146 ts_tablet_manager.cc:1916] T 6ec421eb820b4f40afcff5a5bf80c508 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:44.837910 2700 tablet.cc:1722] T cb829649f73d4d3f8862d268c953bd86 P 23eeffd3de254ce18db5febfde971c6b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251025 07:52:44.841012 2169 ts_tablet_manager.cc:1950] T d5e5e5bbcf9f43118eea502b69880af5 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:44.841806 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 127) took 6025 ms. Trace:
I20251025 07:52:44.841907 2166 ts_tablet_manager.cc:1929] T b772025852794a49b072f37cccbd3291 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:44.842016 2166 log.cc:1199] T b772025852794a49b072f37cccbd3291 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/b772025852794a49b072f37cccbd3291
I20251025 07:52:44.841885 2169 rpcz_store.cc:276] 1025 07:52:38.817036 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.817293 (+ 257us) service_pool.cc:225] Handling call
1025 07:52:44.744716 (+5927423us) ts_tablet_manager.cc:1151] Deleting tablet d5e5e5bbcf9f43118eea502b69880af5
1025 07:52:44.744724 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:44.780209 (+ 35485us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.783603 (+ 3394us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.841780 (+ 58177us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5927272}
I20251025 07:52:44.842327 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet d5e5e5bbcf9f43118eea502b69880af5 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]) successfully deleted
I20251025 07:52:44.848618 2169 tablet_replica.cc:333] T 53198283718d4b45b30df65b5bde96a6 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:44.849404 2169 raft_consensus.cc:2243] T 53198283718d4b45b30df65b5bde96a6 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:44.850410 2169 raft_consensus.cc:2272] T 53198283718d4b45b30df65b5bde96a6 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:44.853602 1048 raft_consensus.cc:3060] T c9f7ba3709c745d4a625449d6e1c8ba1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:44.855345 2166 ts_tablet_manager.cc:1950] T b772025852794a49b072f37cccbd3291 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:44.855698 2040 ts_manager.cc:194] Registered new tserver with Master: 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:44.857285 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 111) took 6189 ms. Trace:
I20251025 07:52:44.857394 2166 rpcz_store.cc:276] 1025 07:52:38.667739 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.667782 (+ 43us) service_pool.cc:225] Handling call
1025 07:52:44.818091 (+6150309us) ts_tablet_manager.cc:1151] Deleting tablet b772025852794a49b072f37cccbd3291
1025 07:52:44.818102 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:44.838700 (+ 20598us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.841784 (+ 3084us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.857255 (+ 15471us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6145080}
I20251025 07:52:44.862003 2169 ts_tablet_manager.cc:1916] T 53198283718d4b45b30df65b5bde96a6 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:44.862957 2166 tablet_replica.cc:333] T 4037dee385d04e85bb78c8e77cb00379 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:44.863492 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet b772025852794a49b072f37cccbd3291 (table default.Table_e50bf47b78d94bbc811e6f3fb3551edb [id=0827603890c8466082bc6c306c9c66af]) successfully deleted
I20251025 07:52:44.866077 2166 raft_consensus.cc:2243] T 4037dee385d04e85bb78c8e77cb00379 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:44.866345 2166 raft_consensus.cc:2272] T 4037dee385d04e85bb78c8e77cb00379 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:44.870594 2166 ts_tablet_manager.cc:1916] T 4037dee385d04e85bb78c8e77cb00379 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:44.876802 1167 tablet_bootstrap.cc:492] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:44.876901 2169 ts_tablet_manager.cc:1929] T 53198283718d4b45b30df65b5bde96a6 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:44.876967 1167 ts_tablet_manager.cc:1403] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.205s user 0.002s sys 0.005s
I20251025 07:52:44.878077 1167 raft_consensus.cc:359] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.878324 1167 raft_consensus.cc:385] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:44.878394 1167 raft_consensus.cc:740] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:44.878705 1167 consensus_queue.cc:260] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.879236 2169 log.cc:1199] T 53198283718d4b45b30df65b5bde96a6 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/53198283718d4b45b30df65b5bde96a6
I20251025 07:52:44.879336 1167 ts_tablet_manager.cc:1434] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:44.879771 2169 ts_tablet_manager.cc:1950] T 53198283718d4b45b30df65b5bde96a6 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:44.881080 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 128) took 6063 ms. Trace:
I20251025 07:52:44.882153 2169 rpcz_store.cc:276] 1025 07:52:38.818440 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.818511 (+ 71us) service_pool.cc:225] Handling call
1025 07:52:44.848561 (+6030050us) ts_tablet_manager.cc:1151] Deleting tablet 53198283718d4b45b30df65b5bde96a6
1025 07:52:44.848573 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:44.868664 (+ 20091us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.876768 (+ 8104us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.881059 (+ 4291us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6029912}
I20251025 07:52:44.882925 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 53198283718d4b45b30df65b5bde96a6 (table default.Table_77761ef857464942be15bf6e67127d6d [id=d9d0e8bb796a45a8b17336c04b12cd01]) successfully deleted
I20251025 07:52:44.882983 2169 tablet_replica.cc:333] T f3c87a83082b421cb4be546ac4ce7ad8 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:44.883168 2169 raft_consensus.cc:2243] T f3c87a83082b421cb4be546ac4ce7ad8 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:44.883970 2169 raft_consensus.cc:2272] T f3c87a83082b421cb4be546ac4ce7ad8 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:44.884968 2169 ts_tablet_manager.cc:1916] T f3c87a83082b421cb4be546ac4ce7ad8 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:44.886950 1167 tablet_bootstrap.cc:492] T dda0b26e4f1a44928bc4b84f333901c0 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:44.885982 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d6a9937d10a4cc2813671dd1bfa1664 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:44.888145 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 53198283718d4b45b30df65b5bde96a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:44.887514 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet d5e5e5bbcf9f43118eea502b69880af5 with delete_type TABLET_DATA_DELETED (replaced by tablet f4578ce69a0d4322944e5ce7c8a59831) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:44.888798 1167 tablet_bootstrap.cc:654] T dda0b26e4f1a44928bc4b84f333901c0 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:44.896443 2166 ts_tablet_manager.cc:1929] T 4037dee385d04e85bb78c8e77cb00379 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:44.898321 2169 ts_tablet_manager.cc:1929] T f3c87a83082b421cb4be546ac4ce7ad8 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:44.902693 2169 log.cc:1199] T f3c87a83082b421cb4be546ac4ce7ad8 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/f3c87a83082b421cb4be546ac4ce7ad8
I20251025 07:52:44.903185 2169 ts_tablet_manager.cc:1950] T f3c87a83082b421cb4be546ac4ce7ad8 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:44.904345 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 129) took 6085 ms. Trace:
I20251025 07:52:44.901993 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c76d1b509f04ab1a10b83cff675c11e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.905567 743 tablet_service.cc:1552] Processing DeleteTablet for tablet b772025852794a49b072f37cccbd3291 with delete_type TABLET_DATA_DELETED (replaced by tablet f97acc9a638b4d5a968dcc3412a384bc) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:44.905874 2166 log.cc:1199] T 4037dee385d04e85bb78c8e77cb00379 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/4037dee385d04e85bb78c8e77cb00379
I20251025 07:52:44.905759 2169 rpcz_store.cc:276] 1025 07:52:38.818991 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.820014 (+ 1023us) service_pool.cc:225] Handling call
1025 07:52:44.882933 (+6062919us) ts_tablet_manager.cc:1151] Deleting tablet f3c87a83082b421cb4be546ac4ce7ad8
1025 07:52:44.882944 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:44.895131 (+ 12187us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.898238 (+ 3107us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.904324 (+ 6086us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6062717}
I20251025 07:52:44.906280 2166 ts_tablet_manager.cc:1950] T 4037dee385d04e85bb78c8e77cb00379 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:44.906530 2169 tablet_replica.cc:333] T 08da9d2687704725aacd11cd92c6a9f9 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:44.906991 2169 raft_consensus.cc:2243] T 08da9d2687704725aacd11cd92c6a9f9 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:44.906993 2663 raft_consensus.cc:493] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:44.907151 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet f3c87a83082b421cb4be546ac4ce7ad8 (table default.Table_fcddb31989ee4236a4981c12779b3f5d [id=f3794fa081364666b944c412415bdba0]) successfully deleted
I20251025 07:52:44.907136 2663 raft_consensus.cc:515] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.907522 447 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 81) took 2874 ms. Trace:
I20251025 07:52:44.907627 2169 raft_consensus.cc:2272] T 08da9d2687704725aacd11cd92c6a9f9 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:44.907779 2663 leader_election.cc:290] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:44.907650 447 rpcz_store.cc:276] 1025 07:52:42.032654 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.790299 (+2757645us) service_pool.cc:225] Handling call
1025 07:52:44.791289 (+ 990us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:44.807969 (+ 16680us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:44.819842 (+ 11873us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.819849 (+ 7us) write_op.cc:454] Released schema lock
1025 07:52:44.819940 (+ 91us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:44.902873 (+ 82933us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:44.907463 (+ 4590us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:44.808499 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:44.808601 (+ 102us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:44.808607 (+ 6us) write_op.cc:435] Acquired schema lock
1025 07:52:44.808610 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:44.808698 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:44.808712 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:44.808717 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:44.808790 (+ 73us) tablet.cc:699] Row locks acquired
1025 07:52:44.808793 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:44.808828 (+ 35us) write_op.cc:270] Start()
1025 07:52:44.808859 (+ 31us) write_op.cc:276] Timestamp: P: 1761378764808809 usec, L: 0
1025 07:52:44.808863 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:44.809055 (+ 192us) log.cc:844] Serialized 433 byte log entry
1025 07:52:44.815356 (+ 6301us) op_driver.cc:464] REPLICATION: finished
1025 07:52:44.815459 (+ 103us) write_op.cc:301] APPLY: starting
1025 07:52:44.815474 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:44.815517 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:44.815561 (+ 44us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:44.815666 (+ 105us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:44.815702 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:44.815710 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:44.815784 (+ 74us) log.cc:844] Serialized 29 byte log entry
1025 07:52:44.815847 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.815868 (+ 21us) write_op.cc:454] Released schema lock
1025 07:52:44.815875 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":57,"apply.run_cpu_time_us":480,"apply.run_wall_time_us":479,"num_ops":2,"prepare.queue_time_us":50,"prepare.run_cpu_time_us":672,"prepare.run_wall_time_us":671,"raft.queue_time_us":1107,"raft.run_cpu_time_us":575,"raft.run_wall_time_us":574,"replication_time_us":6464}]]}
I20251025 07:52:44.908653 2169 ts_tablet_manager.cc:1916] T 08da9d2687704725aacd11cd92c6a9f9 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:44.908829 447 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_c4fe5959f4f242eeb407926c8f97e810"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:44.907009 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 112) took 6238 ms. Trace:
I20251025 07:52:44.909287 2166 rpcz_store.cc:276] 1025 07:52:38.668976 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.669026 (+ 50us) service_pool.cc:225] Handling call
1025 07:52:44.862899 (+6193873us) ts_tablet_manager.cc:1151] Deleting tablet 4037dee385d04e85bb78c8e77cb00379
1025 07:52:44.862911 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:44.892954 (+ 30043us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.896348 (+ 3394us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.906990 (+ 10642us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6189839}
W20251025 07:52:44.909438 447 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4fe5959f4f242eeb407926c8f97e810 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:44.912293 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 4037dee385d04e85bb78c8e77cb00379 (table default.Table_892da226f7054e61b5ffc882b1d5c35a [id=c13e40cbb3624b3997e907d4f2c1f706]) successfully deleted
I20251025 07:52:44.912730 2166 tablet_replica.cc:333] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:44.912955 2166 raft_consensus.cc:2243] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:44.916258 2166 raft_consensus.cc:2272] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:44.919551 2166 ts_tablet_manager.cc:1916] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:44.920450 447 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 83) took 2750 ms. Trace:
I20251025 07:52:44.920590 447 rpcz_store.cc:276] 1025 07:52:42.169579 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.908472 (+2738893us) service_pool.cc:225] Handling call
1025 07:52:44.909547 (+ 1075us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:44.909967 (+ 420us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:44.920120 (+ 10153us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.920127 (+ 7us) write_op.cc:454] Released schema lock
1025 07:52:44.920194 (+ 67us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:44.920301 (+ 107us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:44.920392 (+ 91us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:44.910565 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:44.910661 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:44.910668 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:44.910671 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:44.910762 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:44.910790 (+ 28us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:44.910795 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:44.910865 (+ 70us) tablet.cc:699] Row locks acquired
1025 07:52:44.910867 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:44.910901 (+ 34us) write_op.cc:270] Start()
1025 07:52:44.910942 (+ 41us) write_op.cc:276] Timestamp: P: 1761378764910886 usec, L: 0
1025 07:52:44.910945 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:44.911153 (+ 208us) log.cc:844] Serialized 433 byte log entry
1025 07:52:44.916061 (+ 4908us) op_driver.cc:464] REPLICATION: finished
1025 07:52:44.916163 (+ 102us) write_op.cc:301] APPLY: starting
1025 07:52:44.916180 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:44.916222 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:44.916246 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:44.916373 (+ 127us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:44.916414 (+ 41us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:44.916422 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:44.916499 (+ 77us) log.cc:844] Serialized 29 byte log entry
1025 07:52:44.916553 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.916574 (+ 21us) write_op.cc:454] Released schema lock
1025 07:52:44.916580 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":54,"apply.run_cpu_time_us":465,"apply.run_wall_time_us":464,"num_ops":2,"prepare.queue_time_us":43,"prepare.run_cpu_time_us":685,"prepare.run_wall_time_us":685,"raft.queue_time_us":1336,"raft.run_cpu_time_us":511,"raft.run_wall_time_us":511,"replication_time_us":5089}]]}
I20251025 07:52:44.921514 447 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_9726fef71da14752a13971fe2ddfa9e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:44.922089 447 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9726fef71da14752a13971fe2ddfa9e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:44.923238 455 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 82) took 2753 ms. Trace:
I20251025 07:52:44.923372 455 rpcz_store.cc:276] 1025 07:52:42.169349 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.800352 (+2631003us) service_pool.cc:225] Handling call
1025 07:52:44.801328 (+ 976us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:44.857982 (+ 56654us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:44.878239 (+ 20257us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.878247 (+ 8us) write_op.cc:454] Released schema lock
1025 07:52:44.878373 (+ 126us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:44.923087 (+ 44714us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:44.923164 (+ 77us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:44.858613 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:44.858733 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:44.858740 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:44.858743 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:44.858834 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:44.858848 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:44.858853 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:44.858951 (+ 98us) tablet.cc:699] Row locks acquired
1025 07:52:44.858954 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:44.858989 (+ 35us) write_op.cc:270] Start()
1025 07:52:44.859024 (+ 35us) write_op.cc:276] Timestamp: P: 1761378764858971 usec, L: 0
1025 07:52:44.859028 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:44.859197 (+ 169us) log.cc:844] Serialized 433 byte log entry
1025 07:52:44.862378 (+ 3181us) op_driver.cc:464] REPLICATION: finished
1025 07:52:44.862489 (+ 111us) write_op.cc:301] APPLY: starting
1025 07:52:44.862505 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:44.862552 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:44.862586 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:44.862703 (+ 117us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:44.862751 (+ 48us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:44.862762 (+ 11us) write_op.cc:312] APPLY: finished
1025 07:52:44.862865 (+ 103us) log.cc:844] Serialized 29 byte log entry
1025 07:52:44.862954 (+ 89us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.862977 (+ 23us) write_op.cc:454] Released schema lock
1025 07:52:44.862987 (+ 10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":59,"apply.run_cpu_time_us":560,"apply.run_wall_time_us":559,"num_ops":2,"prepare.queue_time_us":73,"prepare.run_cpu_time_us":696,"prepare.run_wall_time_us":696,"raft.queue_time_us":1046,"raft.run_cpu_time_us":452,"raft.run_wall_time_us":452,"replication_time_us":3319}]]}
I20251025 07:52:44.924506 455 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_9ce0483829814297b8bb466be9cf1ce6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:44.925087 455 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ce0483829814297b8bb466be9cf1ce6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:44.931627 451 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:38361 (request call id 212) took 2708 ms (client timeout 1000 ms). Trace:
W20251025 07:52:44.840507 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:46231, user_credentials={real_user=slave}} blocked reactor thread for 122086us
I20251025 07:52:44.938699 847 leader_election.cc:304] T 4678f7eab5e8497db308e8a38e387618 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:44.873584 1166 tablet_bootstrap.cc:492] T c0c6324cbae34dcebade3ef9f89a7150 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:44.939397 1166 ts_tablet_manager.cc:1403] T c0c6324cbae34dcebade3ef9f89a7150 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.210s user 0.004s sys 0.000s
I20251025 07:52:44.940165 1166 raft_consensus.cc:359] T c0c6324cbae34dcebade3ef9f89a7150 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.940313 1166 raft_consensus.cc:385] T c0c6324cbae34dcebade3ef9f89a7150 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:44.940353 1166 raft_consensus.cc:740] T c0c6324cbae34dcebade3ef9f89a7150 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:44.940521 1166 consensus_queue.cc:260] T c0c6324cbae34dcebade3ef9f89a7150 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.940948 2612 consensus_queue.cc:1048] T 36c8fcafd5464b4e999276a8062a812a P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:44.941555 2612 raft_consensus.cc:2804] T 4678f7eab5e8497db308e8a38e387618 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:44.941704 2612 raft_consensus.cc:697] T 4678f7eab5e8497db308e8a38e387618 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:44.941959 2612 consensus_queue.cc:237] T 4678f7eab5e8497db308e8a38e387618 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.928372 2615 raft_consensus.cc:493] T bdaeedbcf35a4aad97f9ef192991978a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:44.943246 2615 raft_consensus.cc:515] T bdaeedbcf35a4aad97f9ef192991978a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.943478 2663 raft_consensus.cc:493] T 1b5ac7e845a24b2eb5b2429d47361232 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:44.943604 2663 raft_consensus.cc:515] T 1b5ac7e845a24b2eb5b2429d47361232 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.943817 2615 leader_election.cc:290] T bdaeedbcf35a4aad97f9ef192991978a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:44.944211 2663 leader_election.cc:290] T 1b5ac7e845a24b2eb5b2429d47361232 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:44.944379 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bdaeedbcf35a4aad97f9ef192991978a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:44.944931 850 leader_election.cc:343] T bdaeedbcf35a4aad97f9ef192991978a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:44.947691 983 leader_election.cc:343] T b94634ac5077439fadfbbce569f60e8b P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: b94634ac5077439fadfbbce569f60e8b
I20251025 07:52:44.947934 983 leader_election.cc:304] T b94634ac5077439fadfbbce569f60e8b P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
W20251025 07:52:44.948189 983 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:43563, user_credentials={real_user=slave}} blocked reactor thread for 279494us
I20251025 07:52:44.949152 2671 raft_consensus.cc:2749] T b94634ac5077439fadfbbce569f60e8b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:44.951120 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2044ca206634eb285197af4ba2073e7" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:44.956280 1167 tablet_bootstrap.cc:492] T dda0b26e4f1a44928bc4b84f333901c0 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:44.956532 1167 ts_tablet_manager.cc:1403] T dda0b26e4f1a44928bc4b84f333901c0 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.070s user 0.004s sys 0.003s
I20251025 07:52:44.957551 1167 raft_consensus.cc:359] T dda0b26e4f1a44928bc4b84f333901c0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.958091 1167 raft_consensus.cc:385] T dda0b26e4f1a44928bc4b84f333901c0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:44.958225 1167 raft_consensus.cc:740] T dda0b26e4f1a44928bc4b84f333901c0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:44.958571 1167 consensus_queue.cc:260] T dda0b26e4f1a44928bc4b84f333901c0 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:44.959345 1167 ts_tablet_manager.cc:1434] T dda0b26e4f1a44928bc4b84f333901c0 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251025 07:52:44.959729 1167 tablet_bootstrap.cc:492] T c11f674385184ae6b235f1db8bf1fb91 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:44.961968 1167 tablet_bootstrap.cc:654] T c11f674385184ae6b235f1db8bf1fb91 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:44.969997 2166 ts_tablet_manager.cc:1929] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:44.970211 2166 log.cc:1199] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/ac0ea298bc114c56a8093e12def21ce1
I20251025 07:52:44.970566 2166 ts_tablet_manager.cc:1950] T ac0ea298bc114c56a8093e12def21ce1 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:44.971240 1169 tablet_bootstrap.cc:492] T 36c8fcafd5464b4e999276a8062a812a P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:44.971342 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 113) took 6298 ms. Trace:
I20251025 07:52:44.971432 1169 ts_tablet_manager.cc:1403] T 36c8fcafd5464b4e999276a8062a812a P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.169s user 0.009s sys 0.019s
W20251025 07:52:44.971758 982 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:46231, user_credentials={real_user=slave}} blocked reactor thread for 325334us
I20251025 07:52:44.972580 2166 rpcz_store.cc:276] 1025 07:52:38.672732 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.672780 (+ 48us) service_pool.cc:225] Handling call
1025 07:52:44.912677 (+6239897us) ts_tablet_manager.cc:1151] Deleting tablet ac0ea298bc114c56a8093e12def21ce1
1025 07:52:44.912689 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:44.932557 (+ 19868us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.969856 (+ 37299us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.971322 (+ 1466us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6239569}
I20251025 07:52:44.973510 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2044ca206634eb285197af4ba2073e7" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:44.973731 774 raft_consensus.cc:2393] T b2044ca206634eb285197af4ba2073e7 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:44.974295 2166 tablet_replica.cc:333] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:44.974614 2166 raft_consensus.cc:2243] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:44.978255 2166 raft_consensus.cc:2272] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:44.980051 2166 ts_tablet_manager.cc:1916] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:44.981575 2169 ts_tablet_manager.cc:1929] T 08da9d2687704725aacd11cd92c6a9f9 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:44.981747 2169 log.cc:1199] T 08da9d2687704725aacd11cd92c6a9f9 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/08da9d2687704725aacd11cd92c6a9f9
I20251025 07:52:44.982197 2169 ts_tablet_manager.cc:1950] T 08da9d2687704725aacd11cd92c6a9f9 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:44.982986 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 130) took 6161 ms. Trace:
I20251025 07:52:44.983130 2169 rpcz_store.cc:276] 1025 07:52:38.821432 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.821518 (+ 86us) service_pool.cc:225] Handling call
1025 07:52:44.906465 (+6084947us) ts_tablet_manager.cc:1151] Deleting tablet 08da9d2687704725aacd11cd92c6a9f9
1025 07:52:44.906479 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:44.917229 (+ 10750us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.981463 (+ 64234us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:44.982971 (+ 1508us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6084746}
I20251025 07:52:44.983536 2169 tablet_replica.cc:333] T 551d07a9242740dcb9366bb2d707a4a8 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:44.983783 2169 raft_consensus.cc:2243] T 551d07a9242740dcb9366bb2d707a4a8 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:44.984124 2169 raft_consensus.cc:2272] T 551d07a9242740dcb9366bb2d707a4a8 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:44.994500 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:46231, user_credentials={real_user=slave}} blocked reactor thread for 55387.1us
I20251025 07:52:45.000725 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 08da9d2687704725aacd11cd92c6a9f9 (table default.Table_3055bcb146aa43d883f4e7208f6c6638 [id=78236f2583524e1fa95840afb7aef43f]) successfully deleted
I20251025 07:52:45.001632 1166 ts_tablet_manager.cc:1434] T c0c6324cbae34dcebade3ef9f89a7150 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.062s user 0.002s sys 0.009s
W20251025 07:52:44.999581 451 rpcz_store.cc:269] 1025 07:52:42.223502 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:43.145942 (+922440us) service_pool.cc:225] Handling call
1025 07:52:44.931589 (+1785647us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":270783,"reported_tablets":254}
I20251025 07:52:45.004196 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet ac0ea298bc114c56a8093e12def21ce1 (table default.Table_4228cd02e10a4b65ad0134b261130561 [id=c68c2e10aae74b6ea22df9c3715eac49]) successfully deleted
I20251025 07:52:45.005038 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 4037dee385d04e85bb78c8e77cb00379 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:45.007297 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a77eb82f26ae43b48de1e85693372d34" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:45.007490 776 raft_consensus.cc:3060] T a77eb82f26ae43b48de1e85693372d34 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:45.008466 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce1ebb1ec14b4801930a72765bca1a7d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:45.008600 773 raft_consensus.cc:3060] T ce1ebb1ec14b4801930a72765bca1a7d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:45.009454 2043 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3aea2869ff814cc7b142012faa274382" instance_seqno: 1761378743627123) as {username='slave'} at 127.0.104.2:51965; Asking this server to re-register.
I20251025 07:52:45.009817 765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11bfe0c87ca941a3846462c8a4512d83" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:45.009958 765 raft_consensus.cc:3060] T 11bfe0c87ca941a3846462c8a4512d83 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:45.010493 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "91728987303b41fd840eb56f3a311c5f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:45.010648 769 raft_consensus.cc:3060] T 91728987303b41fd840eb56f3a311c5f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:45.011189 773 raft_consensus.cc:2468] T ce1ebb1ec14b4801930a72765bca1a7d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
W20251025 07:52:45.012039 454 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:35777 (request call id 194) took 2787 ms (client timeout 1000 ms). Trace:
I20251025 07:52:45.012377 447 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 84) took 2608 ms. Trace:
I20251025 07:52:45.012461 447 rpcz_store.cc:276] 1025 07:52:42.403714 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.921315 (+2517601us) service_pool.cc:225] Handling call
1025 07:52:44.922206 (+ 891us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:44.922601 (+ 395us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:44.983153 (+ 60552us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:45.011846 (+ 28693us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:45.011979 (+ 133us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:44.935916 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:44.936023 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:44.936029 (+ 6us) write_op.cc:435] Acquired schema lock
1025 07:52:44.936031 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:44.936380 (+ 349us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:44.936394 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:44.936399 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:44.936470 (+ 71us) tablet.cc:699] Row locks acquired
1025 07:52:44.936473 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:44.936513 (+ 40us) write_op.cc:270] Start()
1025 07:52:44.936555 (+ 42us) write_op.cc:276] Timestamp: P: 1761378764936497 usec, L: 0
1025 07:52:44.936558 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:44.936836 (+ 278us) log.cc:844] Serialized 433 byte log entry
1025 07:52:44.978630 (+ 41794us) op_driver.cc:464] REPLICATION: finished
1025 07:52:44.979636 (+ 1006us) write_op.cc:301] APPLY: starting
1025 07:52:44.979660 (+ 24us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:44.979711 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:44.982440 (+ 2729us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:44.982594 (+ 154us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:44.982641 (+ 47us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:44.982648 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:44.982719 (+ 71us) log.cc:844] Serialized 29 byte log entry
1025 07:52:44.982779 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.982797 (+ 18us) write_op.cc:454] Released schema lock
1025 07:52:44.982803 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:44.982871 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.982875 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":888,"apply.run_cpu_time_us":612,"apply.run_wall_time_us":3282,"num_ops":2,"prepare.queue_time_us":12835,"prepare.run_cpu_time_us":846,"prepare.run_wall_time_us":1090,"raft.queue_time_us":2235,"raft.run_cpu_time_us":580,"raft.run_wall_time_us":2907,"replication_time_us":42027}]]}
I20251025 07:52:45.013242 765 raft_consensus.cc:2468] T 11bfe0c87ca941a3846462c8a4512d83 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:45.013423 769 raft_consensus.cc:2468] T 91728987303b41fd840eb56f3a311c5f P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:45.013556 455 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 85) took 2600 ms. Trace:
I20251025 07:52:45.013722 455 rpcz_store.cc:276] 1025 07:52:42.411698 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.924356 (+2512658us) service_pool.cc:225] Handling call
1025 07:52:44.925225 (+ 869us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:44.925637 (+ 412us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:44.999008 (+ 73371us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:45.012207 (+ 13199us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:45.012287 (+ 80us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:44.937027 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:44.937126 (+ 99us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:44.937132 (+ 6us) write_op.cc:435] Acquired schema lock
1025 07:52:44.937134 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:44.937207 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:44.937216 (+ 9us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:44.937220 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:44.937283 (+ 63us) tablet.cc:699] Row locks acquired
1025 07:52:44.937286 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:44.937319 (+ 33us) write_op.cc:270] Start()
1025 07:52:44.937344 (+ 25us) write_op.cc:276] Timestamp: P: 1761378764937306 usec, L: 0
1025 07:52:44.937347 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:44.937496 (+ 149us) log.cc:844] Serialized 433 byte log entry
1025 07:52:44.978701 (+ 41205us) op_driver.cc:464] REPLICATION: finished
1025 07:52:44.982940 (+ 4239us) write_op.cc:301] APPLY: starting
1025 07:52:44.982949 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:44.998276 (+ 15327us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:44.998318 (+ 42us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:44.998485 (+ 167us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:44.998541 (+ 56us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:44.998554 (+ 13us) write_op.cc:312] APPLY: finished
1025 07:52:44.998693 (+ 139us) log.cc:844] Serialized 29 byte log entry
1025 07:52:44.998784 (+ 91us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.998819 (+ 35us) write_op.cc:454] Released schema lock
1025 07:52:44.998830 (+ 11us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":4209,"apply.run_cpu_time_us":725,"apply.run_wall_time_us":15963,"mutex_wait_us":10078,"num_ops":2,"prepare.queue_time_us":11011,"raft.queue_time_us":904,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":4,"replication_time_us":41339,"threads_started":2}]]}
I20251025 07:52:45.015054 771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bdaeedbcf35a4aad97f9ef192991978a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:45.015270 771 raft_consensus.cc:2468] T bdaeedbcf35a4aad97f9ef192991978a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:45.017117 776 raft_consensus.cc:2468] T a77eb82f26ae43b48de1e85693372d34 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:45.017871 1166 tablet_bootstrap.cc:492] T 36c414bc700341188b7d85c14173c3e0 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
W20251025 07:52:45.019234 454 rpcz_store.cc:269] 1025 07:52:42.223320 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:42.245515 (+ 22195us) service_pool.cc:225] Handling call
1025 07:52:45.010572 (+2765057us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:43.613459 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:43.613553 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:43.613559 (+ 6us) write_op.cc:435] Acquired schema lock
1025 07:52:43.613561 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:43.613661 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:43.613673 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:43.613678 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:43.613737 (+ 59us) tablet.cc:699] Row locks acquired
1025 07:52:43.613740 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:43.613775 (+ 35us) write_op.cc:270] Start()
1025 07:52:43.613808 (+ 33us) write_op.cc:276] Timestamp: P: 1761378763613758 usec, L: 0
1025 07:52:43.613811 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:43.613963 (+ 152us) log.cc:844] Serialized 356 byte log entry
1025 07:52:43.630089 (+ 16126us) op_driver.cc:464] REPLICATION: finished
1025 07:52:43.630277 (+ 188us) write_op.cc:301] APPLY: starting
1025 07:52:43.630290 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:43.630315 (+ 25us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:43.630349 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:43.630405 (+ 56us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:43.630445 (+ 40us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:43.630453 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:43.630545 (+ 92us) log.cc:844] Serialized 23 byte log entry
1025 07:52:43.630624 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:43.630643 (+ 19us) write_op.cc:454] Released schema lock
1025 07:52:43.630650 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:43.646517 (+ 15867us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:43.646526 (+ 9us) write_op.cc:454] Released schema lock
Related trace 'op':
1025 07:52:44.534692 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:44.534786 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:44.534792 (+ 6us) write_op.cc:435] Acquired schema lock
1025 07:52:44.534794 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:44.534898 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:44.534914 (+ 16us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:44.534920 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:44.534990 (+ 70us) tablet.cc:699] Row locks acquired
1025 07:52:44.534994 (+ 4us) write_op.cc:260] PREPARE: finished
1025 07:52:44.535027 (+ 33us) write_op.cc:270] Start()
1025 07:52:44.535056 (+ 29us) write_op.cc:276] Timestamp: P: 1761378764535009 usec, L: 0
1025 07:52:44.535060 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:44.535234 (+ 174us) log.cc:844] Serialized 356 byte log entry
1025 07:52:44.539476 (+ 4242us) op_driver.cc:464] REPLICATION: finished
1025 07:52:44.539570 (+ 94us) write_op.cc:301] APPLY: starting
1025 07:52:44.539584 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:44.539626 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:44.539663 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:44.539733 (+ 70us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:44.539763 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:44.539770 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:44.539841 (+ 71us) log.cc:844] Serialized 23 byte log entry
1025 07:52:44.539926 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.539948 (+ 22us) write_op.cc:454] Released schema lock
1025 07:52:44.539954 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:44.540023 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.540027 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":103044,"reported_tablets":258,"child_traces":[["op",{"apply.queue_time_us":147,"apply.run_cpu_time_us":433,"apply.run_wall_time_us":433,"num_ops":1,"prepare.queue_time_us":71552,"prepare.run_cpu_time_us":6323,"prepare.run_wall_time_us":33042,"raft.queue_time_us":827,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":16255,"thread_start_us":32409,"threads_started":1}],["op",{"apply.queue_time_us":52,"apply.run_cpu_time_us":444,"apply.run_wall_time_us":443,"num_ops":1,"prepare.queue_time_us":4766,"prepare.run_cpu_time_us":714,"prepare.run_wall_time_us":714,"raft.queue_time_us":1264,"raft.run_cpu_time_us":245,"raft.run_wall_time_us":245,"replication_time_us":4389}]]}
I20251025 07:52:45.020825 1167 tablet_bootstrap.cc:492] T c11f674385184ae6b235f1db8bf1fb91 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:45.021492 1167 ts_tablet_manager.cc:1403] T c11f674385184ae6b235f1db8bf1fb91 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.001s
I20251025 07:52:45.022333 1167 raft_consensus.cc:359] T c11f674385184ae6b235f1db8bf1fb91 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.024916 2663 raft_consensus.cc:493] T 9f039d9107ae493486d799ab6a1d731d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:45.025462 2663 raft_consensus.cc:515] T 9f039d9107ae493486d799ab6a1d731d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.026043 2663 leader_election.cc:290] T 9f039d9107ae493486d799ab6a1d731d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:45.026500 1167 raft_consensus.cc:385] T c11f674385184ae6b235f1db8bf1fb91 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:45.026705 1167 raft_consensus.cc:740] T c11f674385184ae6b235f1db8bf1fb91 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:45.027381 2169 ts_tablet_manager.cc:1916] T 551d07a9242740dcb9366bb2d707a4a8 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:45.027791 1167 consensus_queue.cc:260] T c11f674385184ae6b235f1db8bf1fb91 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.028882 1167 ts_tablet_manager.cc:1434] T c11f674385184ae6b235f1db8bf1fb91 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20251025 07:52:45.033476 905 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34c63b69904d499bb0d286941a8303f2" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:45.033962 905 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e6544202d4f489d82f22d6073ef240d" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:45.034421 905 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "48ff4ce7846b4e56a37eee9bcb17bd5e" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:45.034899 905 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b5ac7e845a24b2eb5b2429d47361232" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:45.004007 2146 ts_tablet_manager.cc:1929] T 6ec421eb820b4f40afcff5a5bf80c508 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:45.035378 2146 log.cc:1199] T 6ec421eb820b4f40afcff5a5bf80c508 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/6ec421eb820b4f40afcff5a5bf80c508
I20251025 07:52:45.035440 905 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9f039d9107ae493486d799ab6a1d731d" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:45.035920 2146 ts_tablet_manager.cc:1950] T 6ec421eb820b4f40afcff5a5bf80c508 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:45.035843 451 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_5cce578756b643d8ba30a65ed7e621fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:45.036520 451 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5cce578756b643d8ba30a65ed7e621fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:45.036969 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 127) took 6506 ms. Trace:
I20251025 07:52:45.037112 2146 rpcz_store.cc:276] 1025 07:52:38.530515 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.530760 (+ 245us) service_pool.cc:225] Handling call
1025 07:52:44.833756 (+6302996us) ts_tablet_manager.cc:1151] Deleting tablet 6ec421eb820b4f40afcff5a5bf80c508
1025 07:52:44.833772 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:44.965594 (+131822us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:45.003899 (+ 38305us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:45.036951 (+ 33052us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6302674}
I20251025 07:52:45.037757 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 6ec421eb820b4f40afcff5a5bf80c508 (table default.Table_5487882c2d0c4af4ad17df240fa90277 [id=d73352841fda408baf49087af00b4250]) successfully deleted
I20251025 07:52:45.037859 455 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_a6c869949b744bc98efb735b0e73e7aa" } new_table_name: "default.Table_fa83d0528b4b481989bfc0ca35c6de44" modify_external_catalogs: true
I20251025 07:52:45.038177 2146 tablet_replica.cc:333] T 551d07a9242740dcb9366bb2d707a4a8 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:45.038414 2146 raft_consensus.cc:2243] T 551d07a9242740dcb9366bb2d707a4a8 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:45.038633 2146 raft_consensus.cc:2272] T 551d07a9242740dcb9366bb2d707a4a8 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:45.039057 454 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_646f48a21c194af3a1611e5e12b2e11b" } new_table_name: "default.Table_c62e0dc1ae504f9bb912013506ff3031" modify_external_catalogs: true
I20251025 07:52:45.039192 1166 tablet_bootstrap.cc:654] T 36c414bc700341188b7d85c14173c3e0 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:45.039273 2146 ts_tablet_manager.cc:1916] T 551d07a9242740dcb9366bb2d707a4a8 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:45.040452 447 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_7d1445fd52a84e80b40235d993a1306a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:45.040669 847 leader_election.cc:304] T ce1ebb1ec14b4801930a72765bca1a7d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:45.041064 2710 raft_consensus.cc:2804] T ce1ebb1ec14b4801930a72765bca1a7d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
W20251025 07:52:45.041121 447 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d1445fd52a84e80b40235d993a1306a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:45.041368 2710 raft_consensus.cc:697] T ce1ebb1ec14b4801930a72765bca1a7d P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:45.041667 2710 consensus_queue.cc:237] T ce1ebb1ec14b4801930a72765bca1a7d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.045790 1167 tablet_bootstrap.cc:492] T a0ea1edeca7347fc844faa5e4be1ae94 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:45.029436 2663 raft_consensus.cc:493] T 9a8e2658a81e436298fe2190726080d4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 07:52:45.037621 710 consensus_peers.cc:597] T c88c46e022ff41a58b02538f2151e260 P 23eeffd3de254ce18db5febfde971c6b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c88c46e022ff41a58b02538f2151e260. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:45.053035 2663 raft_consensus.cc:515] T 9a8e2658a81e436298fe2190726080d4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
W20251025 07:52:45.053246 710 leader_election.cc:343] T 34c63b69904d499bb0d286941a8303f2 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:45.053409 710 leader_election.cc:343] T 7e6544202d4f489d82f22d6073ef240d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:45.053550 710 leader_election.cc:343] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:45.053642 2663 leader_election.cc:290] T 9a8e2658a81e436298fe2190726080d4 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:45.053689 710 leader_election.cc:343] T 1b5ac7e845a24b2eb5b2429d47361232 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:45.053812 710 leader_election.cc:343] T 9f039d9107ae493486d799ab6a1d731d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:45.052856 2709 consensus_queue.cc:1048] T 09898735796541b0a8f3be22cdb6cc30 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 07:52:45.029074 2166 ts_tablet_manager.cc:1929] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:45.054263 2166 log.cc:1199] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/87a93778620f4c84b3b3f7a3d7d7e87f
I20251025 07:52:45.054601 2166 ts_tablet_manager.cc:1950] T 87a93778620f4c84b3b3f7a3d7d7e87f P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:45.055449 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 114) took 6381 ms. Trace:
I20251025 07:52:45.055548 2166 rpcz_store.cc:276] 1025 07:52:38.674292 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.674338 (+ 46us) service_pool.cc:225] Handling call
1025 07:52:44.974215 (+6299877us) ts_tablet_manager.cc:1151] Deleting tablet 87a93778620f4c84b3b3f7a3d7d7e87f
1025 07:52:44.974229 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:44.984748 (+ 10519us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:45.028945 (+ 44197us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:45.055432 (+ 26487us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6297716}
I20251025 07:52:45.055919 2166 tablet_replica.cc:333] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:45.056104 2166 raft_consensus.cc:2243] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:45.056114 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 87a93778620f4c84b3b3f7a3d7d7e87f (table default.Table_0eca32ca64da44198140ae7ec9a77767 [id=2da58c150b514203abed5e20ffa98919]) successfully deleted
I20251025 07:52:45.056143 451 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 86) took 2635 ms. Trace:
I20251025 07:52:45.056234 2166 raft_consensus.cc:2272] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:45.056732 2166 ts_tablet_manager.cc:1916] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:45.056258 451 rpcz_store.cc:276] 1025 07:52:42.420994 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.002436 (+2581442us) service_pool.cc:225] Handling call
1025 07:52:45.035786 (+ 33350us) spinlock_profiling.cc:243] Waited 28.7 ms on lock 0x616000041254. stack: 00007fb90f7779f6 00007fb90f7777b0 00007fb90e7533ad 0000000000983f04 0000000000983ecd 00007fb91ea756bd 00007fb91ea757b3 00007fb91b640441 00007fb9232f69dd 00007fb923483263 00007fb919f997c6 00007fb919f994f5 00007fb91671327e 00007fb91671215e 00007fb916718395 00007fb916718cbf
1025 07:52:45.036648 (+ 862us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:45.037081 (+ 433us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:45.055872 (+ 18791us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:45.056009 (+ 137us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:45.056101 (+ 92us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:45.042338 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:45.042424 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:45.042431 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:45.042434 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:45.042526 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:45.042539 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:45.042544 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:45.042619 (+ 75us) tablet.cc:699] Row locks acquired
1025 07:52:45.042621 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:45.042665 (+ 44us) write_op.cc:270] Start()
1025 07:52:45.042699 (+ 34us) write_op.cc:276] Timestamp: P: 1761378765042648 usec, L: 0
1025 07:52:45.042703 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:45.042858 (+ 155us) log.cc:844] Serialized 433 byte log entry
1025 07:52:45.052403 (+ 9545us) op_driver.cc:464] REPLICATION: finished
1025 07:52:45.053182 (+ 779us) write_op.cc:301] APPLY: starting
1025 07:52:45.053199 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:45.053246 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:45.053303 (+ 57us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:45.053435 (+ 132us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:45.053470 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:45.053479 (+ 9us) write_op.cc:312] APPLY: finished
1025 07:52:45.053567 (+ 88us) log.cc:844] Serialized 29 byte log entry
1025 07:52:45.053641 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.053663 (+ 22us) write_op.cc:454] Released schema lock
1025 07:52:45.053670 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:45.053747 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.053751 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":65981312,"child_traces":[["op",{"apply.queue_time_us":727,"apply.run_cpu_time_us":554,"apply.run_wall_time_us":553,"num_ops":2,"prepare.queue_time_us":4771,"prepare.run_cpu_time_us":629,"prepare.run_wall_time_us":1009,"raft.queue_time_us":34,"raft.run_cpu_time_us":324,"raft.run_wall_time_us":324,"replication_time_us":9664}]]}
I20251025 07:52:45.057155 451 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_e90d7f5ff13b40878e5b625c97792c7d" } new_table_name: "default.Table_67b99749f3bf43228b40d8b9807e69b4" modify_external_catalogs: true
I20251025 07:52:45.061519 1167 tablet_bootstrap.cc:654] T a0ea1edeca7347fc844faa5e4be1ae94 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:45.072403 847 leader_election.cc:304] T 11bfe0c87ca941a3846462c8a4512d83 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:45.073268 847 leader_election.cc:304] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:45.073452 1167 tablet_bootstrap.cc:492] T a0ea1edeca7347fc844faa5e4be1ae94 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:45.073617 1167 ts_tablet_manager.cc:1403] T a0ea1edeca7347fc844faa5e4be1ae94 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.001s
I20251025 07:52:45.074564 1167 raft_consensus.cc:359] T a0ea1edeca7347fc844faa5e4be1ae94 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.074811 1167 raft_consensus.cc:385] T a0ea1edeca7347fc844faa5e4be1ae94 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:45.074944 1167 raft_consensus.cc:740] T a0ea1edeca7347fc844faa5e4be1ae94 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:45.075210 1167 consensus_queue.cc:260] T a0ea1edeca7347fc844faa5e4be1ae94 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.076491 1167 ts_tablet_manager.cc:1434] T a0ea1edeca7347fc844faa5e4be1ae94 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251025 07:52:45.076503 447 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 87) took 2622 ms. Trace:
I20251025 07:52:45.076925 1167 tablet_bootstrap.cc:492] T 682c1e22d7764dce862d970b33b88cb3 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:45.076645 447 rpcz_store.cc:276] 1025 07:52:42.453755 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.013048 (+2559293us) service_pool.cc:225] Handling call
1025 07:52:45.040368 (+ 27320us) spinlock_profiling.cc:243] Waited 27 ms on lock 0x616000041254. stack: 00007fb90f7779f6 00007fb90f7777b0 00007fb90e7533ad 0000000000983f04 0000000000983ecd 00007fb91ea756bd 00007fb91ea757b3 00007fb91b640441 00007fb9232f69dd 00007fb923483263 00007fb919f997c6 00007fb919f994f5 00007fb91671327e 00007fb91671215e 00007fb916718395 00007fb916718cbf
1025 07:52:45.041264 (+ 896us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:45.041798 (+ 534us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:45.076214 (+ 34416us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:45.076336 (+ 122us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:45.076438 (+ 102us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:45.060003 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:45.060099 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:45.060103 (+ 4us) write_op.cc:435] Acquired schema lock
1025 07:52:45.060105 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:45.060197 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:45.060207 (+ 10us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:45.060214 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:45.060278 (+ 64us) tablet.cc:699] Row locks acquired
1025 07:52:45.060280 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:45.060306 (+ 26us) write_op.cc:270] Start()
1025 07:52:45.060334 (+ 28us) write_op.cc:276] Timestamp: P: 1761378765060294 usec, L: 0
1025 07:52:45.060339 (+ 5us) op_driver.cc:348] REPLICATION: starting
1025 07:52:45.060476 (+ 137us) log.cc:844] Serialized 433 byte log entry
1025 07:52:45.067697 (+ 7221us) op_driver.cc:464] REPLICATION: finished
1025 07:52:45.075613 (+ 7916us) write_op.cc:301] APPLY: starting
1025 07:52:45.075628 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:45.075675 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:45.075718 (+ 43us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:45.075881 (+ 163us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:45.075923 (+ 42us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:45.075934 (+ 11us) write_op.cc:312] APPLY: finished
1025 07:52:45.076029 (+ 95us) log.cc:844] Serialized 29 byte log entry
1025 07:52:45.076105 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.076129 (+ 24us) write_op.cc:454] Released schema lock
1025 07:52:45.076137 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:45.076212 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.076216 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":62112000,"child_traces":[["op",{"apply.queue_time_us":7878,"apply.run_cpu_time_us":595,"apply.run_wall_time_us":595,"num_ops":2,"prepare.queue_time_us":17709,"prepare.run_cpu_time_us":548,"prepare.run_wall_time_us":3442,"replication_time_us":7338}]]}
I20251025 07:52:45.077525 447 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_fd5720d69b084df1b36485b7890ac6c2" } new_table_name: "default.Table_799dcb1cef2f4fe6b9b8f17f9f1026c5" modify_external_catalogs: true
I20251025 07:52:45.082111 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f]). Expected version 1 got 0
I20251025 07:52:45.073563 2709 raft_consensus.cc:2804] T 11bfe0c87ca941a3846462c8a4512d83 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:45.082736 2611 raft_consensus.cc:2804] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:45.082854 2611 raft_consensus.cc:697] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:45.083096 2611 consensus_queue.cc:237] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.084206 2105 consensus_queue.cc:1048] T 09898735796541b0a8f3be22cdb6cc30 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:45.085292 2709 raft_consensus.cc:697] T 11bfe0c87ca941a3846462c8a4512d83 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:45.085558 2709 consensus_queue.cc:237] T 11bfe0c87ca941a3846462c8a4512d83 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.085973 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7]). Expected version 1 got 0
I20251025 07:52:45.087445 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 04d27d54fdb7483b869408af65ff85c9 (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]). Expected version 2 got 0
I20251025 07:52:45.090396 2420 raft_consensus.cc:493] T c59f6e6eb9594a8e83f109a09b034045 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3aea2869ff814cc7b142012faa274382)
I20251025 07:52:45.090571 2420 raft_consensus.cc:515] T c59f6e6eb9594a8e83f109a09b034045 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:45.091308 2420 leader_election.cc:290] T c59f6e6eb9594a8e83f109a09b034045 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:45.082202 1166 tablet_bootstrap.cc:492] T 36c414bc700341188b7d85c14173c3e0 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:45.092712 1166 ts_tablet_manager.cc:1403] T 36c414bc700341188b7d85c14173c3e0 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.075s user 0.003s sys 0.002s
W20251025 07:52:45.093075 982 leader_election.cc:343] T 6dcab53a020847a1aa27930265ba0205 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: 6dcab53a020847a1aa27930265ba0205
I20251025 07:52:45.093428 1166 raft_consensus.cc:359] T 36c414bc700341188b7d85c14173c3e0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.093580 1166 raft_consensus.cc:385] T 36c414bc700341188b7d85c14173c3e0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:45.093648 1166 raft_consensus.cc:740] T 36c414bc700341188b7d85c14173c3e0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:45.093848 1166 consensus_queue.cc:260] T 36c414bc700341188b7d85c14173c3e0 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.095476 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet cb829649f73d4d3f8862d268c953bd86 (table default.Table_a9026f5e095c4063b86d4c544c686e22 [id=7944225d3ce7439b881bfb9289d9f040]). Expected version 2 got 0
I20251025 07:52:45.095719 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 110a8e9e4c6c434284c0ae14dd6987f7 (table default.Table_6c5a24747ba14352b0285a41ec2985a4 [id=28bbaad474164a8c8c97a76bb8081791]). Expected version 2 got 0
I20251025 07:52:45.095979 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 872cb7512d104827b1f1f186d33a4a44 (table default.Table_d5c65dce8b1b4214ba519d964012bfc0 [id=d4be274b36a845a99a4b96b8846248fd]). Expected version 1 got 0
I20251025 07:52:45.097774 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet aa9d9ab8073945d28d52fb02213f201a (table default.Table_4791672658f043f6b97b9320a531ebb5 [id=dfd20cf9c3e541c4abe23d59e0aa7f2a]). Expected version 1 got 0
I20251025 07:52:45.098950 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]). Expected version 1 got 0
I20251025 07:52:45.099285 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]). Expected version 1 got 0
I20251025 07:52:45.099542 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_a6c869949b744bc98efb735b0e73e7aa [id=bf77b5ff6f1048dd996d948437f9ff5e]). Expected version 2 got 0
I20251025 07:52:45.099730 1166 ts_tablet_manager.cc:1434] T 36c414bc700341188b7d85c14173c3e0 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251025 07:52:45.100013 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet bc2d16b33ca8426abbd7bd38339c9f1d (table default.Table_4ea985df4c5b415c9570c64506a98cc4 [id=98d2b7797e33485e8872054aa2eb8ec9]). Expected version 1 got 0
I20251025 07:52:45.100131 1166 tablet_bootstrap.cc:492] T 4a1ec8c2ed7f41abb588f592f62097d1 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:45.100257 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e]). Expected version 1 got 0
I20251025 07:52:45.100476 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet ab42909afc524f2fa3c1041495d13a59 (table default.Table_7eb8424d5192485da1a2d67bfdb2bbbf [id=40c29df99590408cbd5c583d4fc741eb]). Expected version 2 got 0
I20251025 07:52:45.100910 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 288e543bb279410c91c01dd1623e110f (table default.Table_6e242ba55b194c23a6701f4b064fdbe5 [id=4c34b64b299f4162a53866cf01c5298f]). Expected version 1 got 0
I20251025 07:52:45.101111 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 5dab439bc50d4e869548154e1d222ad6 (table default.Table_fb9def2302784c999bf0f5707c60eb4c [id=45325c188eac4d34bb016d01de091f87]). Expected version 3 got 0
I20251025 07:52:45.101382 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet f70091024c034c118fb86578db19a31c (table default.Table_b7da40b96d6c4359bed6f898e8cf6275 [id=e041946c093b4c789ce175c736cc230b]). Expected version 2 got 0
I20251025 07:52:45.094197 847 leader_election.cc:304] T bdaeedbcf35a4aad97f9ef192991978a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:45.101975 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]). Expected version 3 got 0
I20251025 07:52:45.102005 2687 raft_consensus.cc:2804] T bdaeedbcf35a4aad97f9ef192991978a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:45.102172 2687 raft_consensus.cc:493] T bdaeedbcf35a4aad97f9ef192991978a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:45.102260 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_87d38832f85e4965bfab1a077990114f [id=ad812862698d421184ebf501ce79b7be]). Expected version 2 got 0
I20251025 07:52:45.102311 2687 raft_consensus.cc:3060] T bdaeedbcf35a4aad97f9ef192991978a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:45.102475 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]). Expected version 1 got 0
I20251025 07:52:45.102746 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 22f7db660616457b83b24e947a0b0bae (table default.Table_4a86191163f2408f9bcc795cd8d1bbf6 [id=bb316a7e000748ca902a9bfb494802d7]). Expected version 1 got 0
I20251025 07:52:45.103356 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 0325d462441349ef81841f1b0cab2a20 (table default.Table_750ecb905839434fa9a1aa72f87394be [id=9ce3ba77c56a4ef09cb3b40ea0e488f6]). Expected version 1 got 0
I20251025 07:52:45.104228 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525]). Expected version 1 got 0
I20251025 07:52:45.104409 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_e90d7f5ff13b40878e5b625c97792c7d [id=fc8180c09d7b4e74b2e161f7acb8fe0e]). Expected version 1 got 0
I20251025 07:52:45.106254 2687 raft_consensus.cc:515] T bdaeedbcf35a4aad97f9ef192991978a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.107227 2687 leader_election.cc:290] T bdaeedbcf35a4aad97f9ef192991978a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:45.107683 847 leader_election.cc:304] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:45.108199 2698 raft_consensus.cc:2804] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:45.108355 2698 raft_consensus.cc:697] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:45.108736 2698 consensus_queue.cc:237] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.109743 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236]). Expected version 1 got 0
I20251025 07:52:45.110023 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3]). Expected version 1 got 0
I20251025 07:52:45.110311 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c]). Expected version 1 got 0
I20251025 07:52:45.110688 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 755f0d6c87044e20a8020f8410940123 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 4 got 0
I20251025 07:52:45.111085 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5]). Expected version 1 got 0
I20251025 07:52:45.120698 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet b56209e4934e4407a8b4518ca4af0d46 (table default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de]). Expected version 1 got 0
I20251025 07:52:45.121692 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 830fdbc6812e416690d0db42a73af503 (table default.Table_92824b3f2a774564b3aea04aaa25d204 [id=fb21a45b86ab446e8c3dae4512a9598f]). Expected version 1 got 0
I20251025 07:52:45.122283 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 5fc092c2bb2d41ac8c5b4fc07460c8ec (table default.Table_27c62e973d74406fb00db8bcf324749c [id=630c274339664b9db18ab4ef7c29c16f]). Expected version 1 got 0
I20251025 07:52:45.122694 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84]). Expected version 1 got 0
I20251025 07:52:45.124154 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3]). Expected version 2 got 0
I20251025 07:52:45.125389 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 36262ab8c6964dd684c01b4db6219a4f (table default.Table_76158f8cf82e4cdb9a90580e46aaea48 [id=f249f30497bc450c887d659721469de8]). Expected version 1 got 0
I20251025 07:52:45.130846 453 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 292fe04055954afd8c08ac2636b68191 (table default.Table_9fa69b10b5e842639fc0b0a58cd9b6a5 [id=a253ce0253bd4da4814741c482193d8f]). Expected version 1 got 0
I20251025 07:52:45.136373 1096 heartbeater.cc:461] Registering TS with master...
I20251025 07:52:45.146445 961 heartbeater.cc:461] Registering TS with master...
I20251025 07:52:45.157441 1167 tablet_bootstrap.cc:654] T 682c1e22d7764dce862d970b33b88cb3 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:45.160141 1169 raft_consensus.cc:359] T 36c8fcafd5464b4e999276a8062a812a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.160852 1169 raft_consensus.cc:385] T 36c8fcafd5464b4e999276a8062a812a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:45.160920 1169 raft_consensus.cc:740] T 36c8fcafd5464b4e999276a8062a812a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:45.161085 1169 consensus_queue.cc:260] T 36c8fcafd5464b4e999276a8062a812a P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.161515 1169 ts_tablet_manager.cc:1434] T 36c8fcafd5464b4e999276a8062a812a P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.190s user 0.001s sys 0.001s
I20251025 07:52:45.161881 1169 tablet_bootstrap.cc:492] T 0551e5d694894e8fbeb66b5e0c1cd75b P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:45.164916 1169 tablet_bootstrap.cc:654] T 0551e5d694894e8fbeb66b5e0c1cd75b P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:45.169669 875 tablet_service.cc:1505] Processing CreateTablet for tablet dfa07f780b2a4acf833a469c15971a0b (DEFAULT_TABLE table=default.Table_307630cf39c04c02a5f618b1e66a9169 [id=a75557cfc9084f07a877513a3091e273]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:45.170173 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfa07f780b2a4acf833a469c15971a0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:45.176748 2146 ts_tablet_manager.cc:1929] T 551d07a9242740dcb9366bb2d707a4a8 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:45.177146 2146 log.cc:1199] T 551d07a9242740dcb9366bb2d707a4a8 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/551d07a9242740dcb9366bb2d707a4a8
I20251025 07:52:45.177673 2146 ts_tablet_manager.cc:1950] T 551d07a9242740dcb9366bb2d707a4a8 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:45.178161 905 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a8e2658a81e436298fe2190726080d4" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:45.179056 710 leader_election.cc:343] T 9a8e2658a81e436298fe2190726080d4 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:45.179791 710 leader_election.cc:343] T c59f6e6eb9594a8e83f109a09b034045 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: c59f6e6eb9594a8e83f109a09b034045
I20251025 07:52:45.182260 748 tablet_service.cc:1505] Processing CreateTablet for tablet dfa07f780b2a4acf833a469c15971a0b (DEFAULT_TABLE table=default.Table_307630cf39c04c02a5f618b1e66a9169 [id=a75557cfc9084f07a877513a3091e273]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:45.182675 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfa07f780b2a4acf833a469c15971a0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:45.183609 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bdaeedbcf35a4aad97f9ef192991978a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:45.183791 772 raft_consensus.cc:3060] T bdaeedbcf35a4aad97f9ef192991978a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:45.186731 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 128) took 6655 ms. Trace:
I20251025 07:52:45.189136 2146 rpcz_store.cc:276] 1025 07:52:38.531088 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.536537 (+ 5449us) service_pool.cc:225] Handling call
1025 07:52:45.038101 (+6501564us) ts_tablet_manager.cc:1151] Deleting tablet 551d07a9242740dcb9366bb2d707a4a8
1025 07:52:45.038110 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:45.065690 (+ 27580us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:45.176653 (+110963us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:45.186670 (+ 10017us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8040,"tablet-delete.queue_time_us":6501381}
I20251025 07:52:45.189337 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5]). Expected version 1 got 0
I20251025 07:52:45.189647 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3]). Expected version 1 got 0
I20251025 07:52:45.189865 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_a6c869949b744bc98efb735b0e73e7aa [id=bf77b5ff6f1048dd996d948437f9ff5e]). Expected version 2 got 0
I20251025 07:52:45.189911 1166 tablet_bootstrap.cc:654] T 4a1ec8c2ed7f41abb588f592f62097d1 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:45.190078 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 551d07a9242740dcb9366bb2d707a4a8 (table default.Table_2c4afce2b14c41e29c5034dc68d31ead [id=dc6d03745bdc42d0ad8bdb9f81971b81]) successfully deleted
I20251025 07:52:45.191013 2166 ts_tablet_manager.cc:1929] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:45.191056 2146 tablet_replica.cc:333] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:45.191145 2166 log.cc:1199] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/2aa522477e934c89834026dc480fe7c0
I20251025 07:52:45.191458 2166 ts_tablet_manager.cc:1950] T 2aa522477e934c89834026dc480fe7c0 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:45.191560 2146 raft_consensus.cc:2243] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382 [term 2 LEADER]: Raft consensus shutting down.
I20251025 07:52:45.192378 2146 raft_consensus.cc:2272] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:45.193331 2146 ts_tablet_manager.cc:1916] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:45.195128 772 raft_consensus.cc:2468] T bdaeedbcf35a4aad97f9ef192991978a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:45.195695 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f]). Expected version 1 got 0
I20251025 07:52:45.196130 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7]). Expected version 1 got 0
I20251025 07:52:45.196419 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 872cb7512d104827b1f1f186d33a4a44 (table default.Table_d5c65dce8b1b4214ba519d964012bfc0 [id=d4be274b36a845a99a4b96b8846248fd]). Expected version 1 got 0
I20251025 07:52:45.196650 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 110a8e9e4c6c434284c0ae14dd6987f7 (table default.Table_6c5a24747ba14352b0285a41ec2985a4 [id=28bbaad474164a8c8c97a76bb8081791]). Expected version 2 got 0
I20251025 07:52:45.196460 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 115) took 6521 ms. Trace:
I20251025 07:52:45.196933 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 04d27d54fdb7483b869408af65ff85c9 (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]). Expected version 2 got 0
I20251025 07:52:45.198166 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet aa9d9ab8073945d28d52fb02213f201a (table default.Table_4791672658f043f6b97b9320a531ebb5 [id=dfd20cf9c3e541c4abe23d59e0aa7f2a]). Expected version 1 got 0
I20251025 07:52:45.197170 2166 rpcz_store.cc:276] 1025 07:52:38.674899 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.674942 (+ 43us) service_pool.cc:225] Handling call
1025 07:52:45.055870 (+6380928us) ts_tablet_manager.cc:1151] Deleting tablet 2aa522477e934c89834026dc480fe7c0
1025 07:52:45.055881 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:45.059974 (+ 4093us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:45.190864 (+130890us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:45.196437 (+ 5573us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6379317}
I20251025 07:52:45.198738 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 292fe04055954afd8c08ac2636b68191 (table default.Table_9fa69b10b5e842639fc0b0a58cd9b6a5 [id=a253ce0253bd4da4814741c482193d8f]). Expected version 1 got 0
I20251025 07:52:45.198987 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 5dab439bc50d4e869548154e1d222ad6 (table default.Table_fb9def2302784c999bf0f5707c60eb4c [id=45325c188eac4d34bb016d01de091f87]). Expected version 3 got 0
I20251025 07:52:45.199043 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 2aa522477e934c89834026dc480fe7c0 (table default.Table_7ce70fea0fe04c52a62033afb8eef977 [id=a8b2a16fd037435893c8f60e92fd1627]) successfully deleted
I20251025 07:52:45.199299 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet f70091024c034c118fb86578db19a31c (table default.Table_b7da40b96d6c4359bed6f898e8cf6275 [id=e041946c093b4c789ce175c736cc230b]). Expected version 2 got 0
I20251025 07:52:45.199515 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 116) took 6524 ms. Trace:
I20251025 07:52:45.199952 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 5fc092c2bb2d41ac8c5b4fc07460c8ec (table default.Table_27c62e973d74406fb00db8bcf324749c [id=630c274339664b9db18ab4ef7c29c16f]). Expected version 1 got 0
I20251025 07:52:45.200129 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]). Expected version 3 got 0
I20251025 07:52:45.200079 2166 rpcz_store.cc:276] 1025 07:52:38.675484 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.675525 (+ 41us) service_pool.cc:225] Handling call
1025 07:52:45.199340 (+6523815us) ts_tablet_manager.cc:1151] Deleting tablet 3775a2e08d4e40bd9f57e4cda759d535
1025 07:52:45.199351 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:45.199498 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6522746}
I20251025 07:52:45.200333 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 22f7db660616457b83b24e947a0b0bae (table default.Table_4a86191163f2408f9bcc795cd8d1bbf6 [id=bb316a7e000748ca902a9bfb494802d7]). Expected version 1 got 0
I20251025 07:52:45.200524 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]). Expected version 1 got 0
I20251025 07:52:45.201191 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 0325d462441349ef81841f1b0cab2a20 (table default.Table_750ecb905839434fa9a1aa72f87394be [id=9ce3ba77c56a4ef09cb3b40ea0e488f6]). Expected version 1 got 0
I20251025 07:52:45.201581 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525]). Expected version 1 got 0
I20251025 07:52:45.201749 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_e90d7f5ff13b40878e5b625c97792c7d [id=fc8180c09d7b4e74b2e161f7acb8fe0e]). Expected version 1 got 0
I20251025 07:52:45.201951 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236]). Expected version 1 got 0
I20251025 07:52:45.202219 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 755f0d6c87044e20a8020f8410940123 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 4 got 0
I20251025 07:52:45.203003 847 leader_election.cc:304] T bdaeedbcf35a4aad97f9ef192991978a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:45.203251 2687 raft_consensus.cc:2804] T bdaeedbcf35a4aad97f9ef192991978a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:45.203305 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet b56209e4934e4407a8b4518ca4af0d46 (table default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de]). Expected version 1 got 0
I20251025 07:52:45.203392 2687 raft_consensus.cc:697] T bdaeedbcf35a4aad97f9ef192991978a P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:45.203449 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 830fdbc6812e416690d0db42a73af503 (table default.Table_92824b3f2a774564b3aea04aaa25d204 [id=fb21a45b86ab446e8c3dae4512a9598f]). Expected version 1 got 0
I20251025 07:52:45.203668 2687 consensus_queue.cc:237] T bdaeedbcf35a4aad97f9ef192991978a P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.203902 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c]). Expected version 1 got 0
I20251025 07:52:45.204249 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84]). Expected version 1 got 0
I20251025 07:52:45.204329 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34c63b69904d499bb0d286941a8303f2" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:45.204465 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_87d38832f85e4965bfab1a077990114f [id=ad812862698d421184ebf501ce79b7be]). Expected version 2 got 0
I20251025 07:52:45.205030 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 36262ab8c6964dd684c01b4db6219a4f (table default.Table_76158f8cf82e4cdb9a90580e46aaea48 [id=f249f30497bc450c887d659721469de8]). Expected version 1 got 0
W20251025 07:52:45.205539 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 3775a2e08d4e40bd9f57e4cda759d535 because the tablet was not found. No further retry: Not found: Tablet not found: 3775a2e08d4e40bd9f57e4cda759d535
I20251025 07:52:45.205997 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3]). Expected version 2 got 0
I20251025 07:52:45.206276 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 288e543bb279410c91c01dd1623e110f (table default.Table_6e242ba55b194c23a6701f4b064fdbe5 [id=4c34b64b299f4162a53866cf01c5298f]). Expected version 1 got 0
I20251025 07:52:45.206753 2166 tablet_replica.cc:333] T 35ba9d23262248cda6fdf03c51be11a4 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:45.206583 452 catalog_manager.cc:5649] T 66f0d398e59a4175aa9fe90715124d1a P db42be7ad3c741a0aa427f27e245993b reported cstate change: term changed from 0 to 2, leader changed from <none> to db42be7ad3c741a0aa427f27e245993b (127.0.104.3). New cstate: current_term: 2 leader_uuid: "db42be7ad3c741a0aa427f27e245993b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } }
I20251025 07:52:45.207381 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet ab42909afc524f2fa3c1041495d13a59 (table default.Table_7eb8424d5192485da1a2d67bfdb2bbbf [id=40c29df99590408cbd5c583d4fc741eb]). Expected version 2 got 0
I20251025 07:52:45.207808 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet bc2d16b33ca8426abbd7bd38339c9f1d (table default.Table_4ea985df4c5b415c9570c64506a98cc4 [id=98d2b7797e33485e8872054aa2eb8ec9]). Expected version 1 got 0
I20251025 07:52:45.208030 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e]). Expected version 1 got 0
I20251025 07:52:45.208281 2166 raft_consensus.cc:2243] T 35ba9d23262248cda6fdf03c51be11a4 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:45.208428 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]). Expected version 1 got 0
I20251025 07:52:45.208700 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]). Expected version 1 got 0
I20251025 07:52:45.208716 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e6544202d4f489d82f22d6073ef240d" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:45.209162 2166 raft_consensus.cc:2272] T 35ba9d23262248cda6fdf03c51be11a4 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:45.209980 2166 ts_tablet_manager.cc:1916] T 35ba9d23262248cda6fdf03c51be11a4 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:45.217495 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "48ff4ce7846b4e56a37eee9bcb17bd5e" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:45.218151 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b5ac7e845a24b2eb5b2429d47361232" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:45.219789 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9f039d9107ae493486d799ab6a1d731d" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:45.220314 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a8e2658a81e436298fe2190726080d4" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:45.229012 2663 raft_consensus.cc:493] T 5fca357d7e24494fb6af4bd73d98833c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db42be7ad3c741a0aa427f27e245993b)
I20251025 07:52:45.229168 2663 raft_consensus.cc:515] T 5fca357d7e24494fb6af4bd73d98833c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:45.229770 2663 leader_election.cc:290] T 5fca357d7e24494fb6af4bd73d98833c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:45.233160 2169 ts_tablet_manager.cc:1929] T 551d07a9242740dcb9366bb2d707a4a8 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:45.233336 2169 log.cc:1199] T 551d07a9242740dcb9366bb2d707a4a8 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/551d07a9242740dcb9366bb2d707a4a8
I20251025 07:52:45.233772 2169 ts_tablet_manager.cc:1950] T 551d07a9242740dcb9366bb2d707a4a8 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:45.233891 1166 tablet_bootstrap.cc:492] T 4a1ec8c2ed7f41abb588f592f62097d1 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:45.234066 1166 ts_tablet_manager.cc:1403] T 4a1ec8c2ed7f41abb588f592f62097d1 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.134s user 0.001s sys 0.002s
I20251025 07:52:45.236053 983 leader_election.cc:304] T 6dcab53a020847a1aa27930265ba0205 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b
I20251025 07:52:45.237576 2040 ts_manager.cc:194] Registered new tserver with Master: 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:45.246570 983 leader_election.cc:304] T b2044ca206634eb285197af4ba2073e7 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:45.247090 1166 raft_consensus.cc:359] T 4a1ec8c2ed7f41abb588f592f62097d1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
W20251025 07:52:45.248677 712 leader_election.cc:343] T 34c63b69904d499bb0d286941a8303f2 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:45.248821 712 leader_election.cc:304] T 34c63b69904d499bb0d286941a8303f2 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:45.249281 2663 raft_consensus.cc:2749] T 34c63b69904d499bb0d286941a8303f2 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:45.249421 1166 raft_consensus.cc:385] T 4a1ec8c2ed7f41abb588f592f62097d1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:45.249536 1166 raft_consensus.cc:740] T 4a1ec8c2ed7f41abb588f592f62097d1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:45.249679 2467 raft_consensus.cc:2804] T 6dcab53a020847a1aa27930265ba0205 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:45.249799 2467 raft_consensus.cc:493] T 6dcab53a020847a1aa27930265ba0205 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:45.249776 1166 consensus_queue.cc:260] T 4a1ec8c2ed7f41abb588f592f62097d1 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.249900 2671 raft_consensus.cc:3060] T b2044ca206634eb285197af4ba2073e7 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:45.249924 2467 raft_consensus.cc:3060] T 6dcab53a020847a1aa27930265ba0205 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:45.250394 1166 ts_tablet_manager.cc:1434] T 4a1ec8c2ed7f41abb588f592f62097d1 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251025 07:52:45.237747 1169 tablet_bootstrap.cc:492] T 0551e5d694894e8fbeb66b5e0c1cd75b P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:45.241894 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bdaeedbcf35a4aad97f9ef192991978a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:45.241964 1019 tablet_service.cc:1505] Processing CreateTablet for tablet dfa07f780b2a4acf833a469c15971a0b (DEFAULT_TABLE table=default.Table_307630cf39c04c02a5f618b1e66a9169 [id=a75557cfc9084f07a877513a3091e273]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:45.252990 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfa07f780b2a4acf833a469c15971a0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:45.252996 1166 tablet_bootstrap.cc:492] T 2e2e6893a48449969b71ee1c08d1960e P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:45.253758 1167 tablet_bootstrap.cc:492] T 682c1e22d7764dce862d970b33b88cb3 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:45.253881 1167 ts_tablet_manager.cc:1403] T 682c1e22d7764dce862d970b33b88cb3 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.177s user 0.006s sys 0.000s
I20251025 07:52:45.254711 1166 tablet_bootstrap.cc:654] T 2e2e6893a48449969b71ee1c08d1960e P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:45.254655 1167 raft_consensus.cc:359] T 682c1e22d7764dce862d970b33b88cb3 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.254801 1167 raft_consensus.cc:385] T 682c1e22d7764dce862d970b33b88cb3 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:45.254844 1167 raft_consensus.cc:740] T 682c1e22d7764dce862d970b33b88cb3 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:45.255046 1167 consensus_queue.cc:260] T 682c1e22d7764dce862d970b33b88cb3 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.255466 1167 ts_tablet_manager.cc:1434] T 682c1e22d7764dce862d970b33b88cb3 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251025 07:52:45.256590 1169 ts_tablet_manager.cc:1403] T 0551e5d694894e8fbeb66b5e0c1cd75b P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.095s user 0.004s sys 0.008s
I20251025 07:52:45.257258 2709 consensus_queue.cc:1048] T b2044ca206634eb285197af4ba2073e7 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:45.257501 1169 raft_consensus.cc:359] T 0551e5d694894e8fbeb66b5e0c1cd75b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.257776 1169 raft_consensus.cc:385] T 0551e5d694894e8fbeb66b5e0c1cd75b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:45.257874 1169 raft_consensus.cc:740] T 0551e5d694894e8fbeb66b5e0c1cd75b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:45.258127 1169 consensus_queue.cc:260] T 0551e5d694894e8fbeb66b5e0c1cd75b P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.258678 1169 ts_tablet_manager.cc:1434] T 0551e5d694894e8fbeb66b5e0c1cd75b P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251025 07:52:45.258970 1169 tablet_bootstrap.cc:492] T ab42909afc524f2fa3c1041495d13a59 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
W20251025 07:52:45.260545 710 leader_election.cc:343] T 5fca357d7e24494fb6af4bd73d98833c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 5fca357d7e24494fb6af4bd73d98833c
I20251025 07:52:45.261729 2117 raft_consensus.cc:493] T 7c7a9d7a4fb3449bb57eafa83230a839 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3aea2869ff814cc7b142012faa274382)
W20251025 07:52:45.261776 850 leader_election.cc:343] T bdaeedbcf35a4aad97f9ef192991978a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:45.261901 2117 raft_consensus.cc:515] T 7c7a9d7a4fb3449bb57eafa83230a839 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
W20251025 07:52:45.262715 712 leader_election.cc:343] T 7e6544202d4f489d82f22d6073ef240d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:45.262729 2117 leader_election.cc:290] T 7c7a9d7a4fb3449bb57eafa83230a839 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:45.262835 712 leader_election.cc:304] T 7e6544202d4f489d82f22d6073ef240d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:45.263808 712 leader_election.cc:343] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:45.263927 712 leader_election.cc:304] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:45.264309 2663 raft_consensus.cc:2749] T 7e6544202d4f489d82f22d6073ef240d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:45.264444 2663 raft_consensus.cc:2749] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:45.265012 712 leader_election.cc:343] T 1b5ac7e845a24b2eb5b2429d47361232 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:45.265103 712 leader_election.cc:304] T 1b5ac7e845a24b2eb5b2429d47361232 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:45.265472 2663 raft_consensus.cc:2749] T 1b5ac7e845a24b2eb5b2429d47361232 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:45.265856 712 consensus_peers.cc:597] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d6a9937d10a4cc2813671dd1bfa1664. This is attempt 1: this message will repeat every 5th retry.
W20251025 07:52:45.266206 712 leader_election.cc:343] T 9f039d9107ae493486d799ab6a1d731d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:45.266299 712 leader_election.cc:304] T 9f039d9107ae493486d799ab6a1d731d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:45.266492 2117 raft_consensus.cc:2749] T 9f039d9107ae493486d799ab6a1d731d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:45.266819 712 leader_election.cc:343] T 9a8e2658a81e436298fe2190726080d4 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:45.266912 712 leader_election.cc:304] T 9a8e2658a81e436298fe2190726080d4 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:45.267087 2117 raft_consensus.cc:2749] T 9a8e2658a81e436298fe2190726080d4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:45.268178 712 leader_election.cc:343] T c59f6e6eb9594a8e83f109a09b034045 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: c59f6e6eb9594a8e83f109a09b034045
I20251025 07:52:45.268281 712 leader_election.cc:304] T c59f6e6eb9594a8e83f109a09b034045 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:45.268455 2663 raft_consensus.cc:2749] T c59f6e6eb9594a8e83f109a09b034045 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251025 07:52:45.269354 712 consensus_peers.cc:597] T c88c46e022ff41a58b02538f2151e260 P 23eeffd3de254ce18db5febfde971c6b -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c88c46e022ff41a58b02538f2151e260. This is attempt 1: this message will repeat every 5th retry.
W20251025 07:52:45.272769 710 leader_election.cc:343] T 7c7a9d7a4fb3449bb57eafa83230a839 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 7c7a9d7a4fb3449bb57eafa83230a839
I20251025 07:52:45.273523 1167 tablet_bootstrap.cc:492] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:45.273756 776 raft_consensus.cc:1275] T 66f0d398e59a4175aa9fe90715124d1a P 23eeffd3de254ce18db5febfde971c6b [term 2 FOLLOWER]: Refusing update from remote peer db42be7ad3c741a0aa427f27e245993b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 07:52:45.275036 2484 consensus_queue.cc:1048] T 66f0d398e59a4175aa9fe90715124d1a P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:45.277853 1167 tablet_bootstrap.cc:654] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:45.279561 712 leader_election.cc:343] T 5fca357d7e24494fb6af4bd73d98833c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 5fca357d7e24494fb6af4bd73d98833c
I20251025 07:52:45.279690 712 leader_election.cc:304] T 5fca357d7e24494fb6af4bd73d98833c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:45.279930 2663 raft_consensus.cc:2749] T 5fca357d7e24494fb6af4bd73d98833c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 07:52:45.285600 2484 raft_consensus.cc:493] T 050a5d3b5f934dd19bf2e221f2fe0780 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:45.285779 2484 raft_consensus.cc:515] T 050a5d3b5f934dd19bf2e221f2fe0780 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.286379 1169 tablet_bootstrap.cc:654] T ab42909afc524f2fa3c1041495d13a59 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:45.287250 2663 raft_consensus.cc:493] T 74e2d78c024849b0af5c0bde1f48c36f P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db42be7ad3c741a0aa427f27e245993b)
I20251025 07:52:45.287371 2663 raft_consensus.cc:515] T 74e2d78c024849b0af5c0bde1f48c36f P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:45.287911 2663 leader_election.cc:290] T 74e2d78c024849b0af5c0bde1f48c36f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:45.288554 2484 leader_election.cc:290] T 050a5d3b5f934dd19bf2e221f2fe0780 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:45.289402 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "050a5d3b5f934dd19bf2e221f2fe0780" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
W20251025 07:52:45.291391 710 leader_election.cc:343] T 74e2d78c024849b0af5c0bde1f48c36f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 74e2d78c024849b0af5c0bde1f48c36f
I20251025 07:52:45.291781 1167 tablet_bootstrap.cc:492] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:45.291929 1167 ts_tablet_manager.cc:1403] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20251025 07:52:45.292586 2709 raft_consensus.cc:493] T 36c414bc700341188b7d85c14173c3e0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:45.292840 2709 raft_consensus.cc:515] T 36c414bc700341188b7d85c14173c3e0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.293848 2709 leader_election.cc:290] T 36c414bc700341188b7d85c14173c3e0 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:45.294237 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36c414bc700341188b7d85c14173c3e0" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:45.294699 850 leader_election.cc:343] T 36c414bc700341188b7d85c14173c3e0 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:45.295595 712 leader_election.cc:343] T 7c7a9d7a4fb3449bb57eafa83230a839 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 7c7a9d7a4fb3449bb57eafa83230a839
I20251025 07:52:45.295713 712 leader_election.cc:304] T 7c7a9d7a4fb3449bb57eafa83230a839 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:45.296113 2663 raft_consensus.cc:2749] T 7c7a9d7a4fb3449bb57eafa83230a839 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 07:52:45.296294 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "050a5d3b5f934dd19bf2e221f2fe0780" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:45.295714 1167 raft_consensus.cc:359] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.297071 1167 raft_consensus.cc:385] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:45.297354 1167 raft_consensus.cc:740] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:45.298223 2663 raft_consensus.cc:493] T c11f674385184ae6b235f1db8bf1fb91 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:45.298329 2663 raft_consensus.cc:515] T c11f674385184ae6b235f1db8bf1fb91 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.298832 2663 leader_election.cc:290] T c11f674385184ae6b235f1db8bf1fb91 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:45.299331 1167 consensus_queue.cc:260] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
W20251025 07:52:45.300561 712 leader_election.cc:343] T 74e2d78c024849b0af5c0bde1f48c36f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 74e2d78c024849b0af5c0bde1f48c36f
I20251025 07:52:45.300671 712 leader_election.cc:304] T 74e2d78c024849b0af5c0bde1f48c36f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:45.300952 1166 tablet_bootstrap.cc:492] T 2e2e6893a48449969b71ee1c08d1960e P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:45.301209 2663 raft_consensus.cc:2749] T 74e2d78c024849b0af5c0bde1f48c36f P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 07:52:45.301467 1166 ts_tablet_manager.cc:1403] T 2e2e6893a48449969b71ee1c08d1960e P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.049s user 0.013s sys 0.006s
I20251025 07:52:45.301716 1167 ts_tablet_manager.cc:1434] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251025 07:52:45.302008 1167 tablet_bootstrap.cc:492] T c187a40f8ff14c6a8c184fbeb030ad13 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:45.302888 1166 raft_consensus.cc:359] T 2e2e6893a48449969b71ee1c08d1960e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.303673 983 leader_election.cc:304] T 050a5d3b5f934dd19bf2e221f2fe0780 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:45.303761 1167 tablet_bootstrap.cc:654] T c187a40f8ff14c6a8c184fbeb030ad13 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:45.303920 1166 raft_consensus.cc:385] T 2e2e6893a48449969b71ee1c08d1960e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:45.303944 2712 raft_consensus.cc:2749] T 050a5d3b5f934dd19bf2e221f2fe0780 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:45.304219 1166 raft_consensus.cc:740] T 2e2e6893a48449969b71ee1c08d1960e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:45.304452 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c11f674385184ae6b235f1db8bf1fb91" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:45.304607 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36c414bc700341188b7d85c14173c3e0" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:45.304759 772 raft_consensus.cc:2468] T 36c414bc700341188b7d85c14173c3e0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:45.305450 1166 consensus_queue.cc:260] T 2e2e6893a48449969b71ee1c08d1960e P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.305089 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c11f674385184ae6b235f1db8bf1fb91" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:45.306185 712 leader_election.cc:343] T c11f674385184ae6b235f1db8bf1fb91 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:45.306497 1166 ts_tablet_manager.cc:1434] T 2e2e6893a48449969b71ee1c08d1960e P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251025 07:52:45.307065 2040 ts_manager.cc:194] Registered new tserver with Master: db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:45.307297 1166 tablet_bootstrap.cc:492] T 627a9cfd398346f2aa34c8932a11b5a8 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:45.308955 847 leader_election.cc:304] T 36c414bc700341188b7d85c14173c3e0 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:45.308987 710 leader_election.cc:343] T c11f674385184ae6b235f1db8bf1fb91 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:45.309098 710 leader_election.cc:304] T c11f674385184ae6b235f1db8bf1fb91 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:45.309453 2709 raft_consensus.cc:2804] T 36c414bc700341188b7d85c14173c3e0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:45.309545 2663 raft_consensus.cc:2749] T c11f674385184ae6b235f1db8bf1fb91 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:45.309571 2709 raft_consensus.cc:493] T 36c414bc700341188b7d85c14173c3e0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:45.309677 2709 raft_consensus.cc:3060] T 36c414bc700341188b7d85c14173c3e0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:45.310487 1166 tablet_bootstrap.cc:654] T 627a9cfd398346f2aa34c8932a11b5a8 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:45.313409 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 131) took 6491 ms. Trace:
I20251025 07:52:45.313602 2169 rpcz_store.cc:276] 1025 07:52:38.822563 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.822729 (+ 166us) service_pool.cc:225] Handling call
1025 07:52:44.983494 (+6160765us) ts_tablet_manager.cc:1151] Deleting tablet 551d07a9242740dcb9366bb2d707a4a8
1025 07:52:44.983504 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:45.131927 (+148423us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:45.233013 (+101086us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:45.313385 (+ 80372us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6160609}
I20251025 07:52:45.314301 2169 tablet_replica.cc:333] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:45.314242 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 551d07a9242740dcb9366bb2d707a4a8 (table default.Table_2c4afce2b14c41e29c5034dc68d31ead [id=dc6d03745bdc42d0ad8bdb9f81971b81]) successfully deleted
I20251025 07:52:45.314639 2169 raft_consensus.cc:2243] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P db42be7ad3c741a0aa427f27e245993b [term 2 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:45.315032 2169 raft_consensus.cc:2272] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P db42be7ad3c741a0aa427f27e245993b [term 2 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:45.316176 2169 ts_tablet_manager.cc:1916] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:45.317775 983 consensus_peers.cc:597] T 6ec421eb820b4f40afcff5a5bf80c508 P db42be7ad3c741a0aa427f27e245993b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6ec421eb820b4f40afcff5a5bf80c508. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:45.329175 1167 tablet_bootstrap.cc:492] T c187a40f8ff14c6a8c184fbeb030ad13 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:45.329358 1167 ts_tablet_manager.cc:1403] T c187a40f8ff14c6a8c184fbeb030ad13 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20251025 07:52:45.331929 2671 raft_consensus.cc:2749] T b2044ca206634eb285197af4ba2073e7 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:45.332641 1167 raft_consensus.cc:359] T c187a40f8ff14c6a8c184fbeb030ad13 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.332809 1167 raft_consensus.cc:385] T c187a40f8ff14c6a8c184fbeb030ad13 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:45.332855 1167 raft_consensus.cc:740] T c187a40f8ff14c6a8c184fbeb030ad13 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:45.333527 2687 consensus_queue.cc:1048] T b2044ca206634eb285197af4ba2073e7 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:45.333719 1167 consensus_queue.cc:260] T c187a40f8ff14c6a8c184fbeb030ad13 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.334640 1167 ts_tablet_manager.cc:1434] T c187a40f8ff14c6a8c184fbeb030ad13 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251025 07:52:45.334934 1167 tablet_bootstrap.cc:492] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:45.335450 1169 tablet_bootstrap.cc:492] T ab42909afc524f2fa3c1041495d13a59 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:45.335680 1169 ts_tablet_manager.cc:1403] T ab42909afc524f2fa3c1041495d13a59 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.077s user 0.002s sys 0.008s
I20251025 07:52:45.336578 1167 tablet_bootstrap.cc:654] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:45.336488 1169 raft_consensus.cc:359] T ab42909afc524f2fa3c1041495d13a59 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.336786 1169 raft_consensus.cc:385] T ab42909afc524f2fa3c1041495d13a59 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:45.336894 1169 raft_consensus.cc:740] T ab42909afc524f2fa3c1041495d13a59 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:45.337169 1169 consensus_queue.cc:260] T ab42909afc524f2fa3c1041495d13a59 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.338027 1169 ts_tablet_manager.cc:1434] T ab42909afc524f2fa3c1041495d13a59 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:45.338433 1169 tablet_bootstrap.cc:492] T bdaeedbcf35a4aad97f9ef192991978a P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:45.339155 2117 raft_consensus.cc:493] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3aea2869ff814cc7b142012faa274382)
I20251025 07:52:45.339298 2117 raft_consensus.cc:515] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:45.339972 2117 leader_election.cc:290] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:45.340734 712 leader_election.cc:343] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: db99a1e68daf4e948a3d66ad276cc5f9
I20251025 07:52:45.342999 1169 tablet_bootstrap.cc:654] T bdaeedbcf35a4aad97f9ef192991978a P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:45.348479 1046 raft_consensus.cc:3060] T 050a5d3b5f934dd19bf2e221f2fe0780 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:45.351291 2146 ts_tablet_manager.cc:1929] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20251025 07:52:45.351470 2146 log.cc:1199] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/7fbae8be73bf4e18b5a5c4ec4cdebc12
W20251025 07:52:45.352020 710 leader_election.cc:343] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: db99a1e68daf4e948a3d66ad276cc5f9
I20251025 07:52:45.352133 710 leader_election.cc:304] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:45.352349 1166 tablet_bootstrap.cc:492] T 627a9cfd398346f2aa34c8932a11b5a8 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:45.352487 1166 ts_tablet_manager.cc:1403] T 627a9cfd398346f2aa34c8932a11b5a8 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.045s user 0.010s sys 0.027s
I20251025 07:52:45.352484 2117 raft_consensus.cc:2749] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 07:52:45.353281 1166 raft_consensus.cc:359] T 627a9cfd398346f2aa34c8932a11b5a8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.353458 2687 consensus_queue.cc:1048] T ebc824ec4c60411dab1a2ea4a385148e P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:45.354378 2709 raft_consensus.cc:515] T 36c414bc700341188b7d85c14173c3e0 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.355533 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36c414bc700341188b7d85c14173c3e0" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:45.355644 2709 leader_election.cc:290] T 36c414bc700341188b7d85c14173c3e0 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
W20251025 07:52:45.356043 850 leader_election.cc:343] T 36c414bc700341188b7d85c14173c3e0 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:45.356190 2146 ts_tablet_manager.cc:1950] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:45.356191 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36c414bc700341188b7d85c14173c3e0" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:45.356455 772 raft_consensus.cc:3060] T 36c414bc700341188b7d85c14173c3e0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:45.361124 2746 consensus_queue.cc:1048] T 4b28215f61574cf48c447dd01955ac77 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:45.364902 1167 tablet_bootstrap.cc:492] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:45.365742 1167 ts_tablet_manager.cc:1403] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.002s
I20251025 07:52:45.353466 1166 raft_consensus.cc:385] T 627a9cfd398346f2aa34c8932a11b5a8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:45.366320 1166 raft_consensus.cc:740] T 627a9cfd398346f2aa34c8932a11b5a8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:45.366544 1166 consensus_queue.cc:260] T 627a9cfd398346f2aa34c8932a11b5a8 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.367089 1166 ts_tablet_manager.cc:1434] T 627a9cfd398346f2aa34c8932a11b5a8 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.014s user 0.000s sys 0.003s
I20251025 07:52:45.367379 1166 tablet_bootstrap.cc:492] T 570201e7bf05406a97c81dde071d894b P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:45.368131 1167 raft_consensus.cc:359] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.369122 1166 tablet_bootstrap.cc:654] T 570201e7bf05406a97c81dde071d894b P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:45.374234 1167 raft_consensus.cc:385] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:45.374416 1167 raft_consensus.cc:740] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:45.374713 1167 consensus_queue.cc:260] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.383502 1167 ts_tablet_manager.cc:1434] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20251025 07:52:45.385955 1166 tablet_bootstrap.cc:492] T 570201e7bf05406a97c81dde071d894b P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:45.386127 1166 ts_tablet_manager.cc:1403] T 570201e7bf05406a97c81dde071d894b P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.002s
I20251025 07:52:45.386946 1166 raft_consensus.cc:359] T 570201e7bf05406a97c81dde071d894b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.387148 1166 raft_consensus.cc:385] T 570201e7bf05406a97c81dde071d894b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:45.387216 1166 raft_consensus.cc:740] T 570201e7bf05406a97c81dde071d894b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:45.387420 1166 consensus_queue.cc:260] T 570201e7bf05406a97c81dde071d894b P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.388046 1166 ts_tablet_manager.cc:1434] T 570201e7bf05406a97c81dde071d894b P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:45.388338 1166 tablet_bootstrap.cc:492] T c7691c7bdd89437c8db060254c0db25d P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:45.390379 1166 tablet_bootstrap.cc:654] T c7691c7bdd89437c8db060254c0db25d P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:45.390724 850 consensus_peers.cc:597] T ebc824ec4c60411dab1a2ea4a385148e P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:45.393283 1167 tablet_bootstrap.cc:492] T e9e8fffb8e984463b5fa2c7dd9e93227 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:45.394353 907 raft_consensus.cc:1275] T 66f0d398e59a4175aa9fe90715124d1a P 3aea2869ff814cc7b142012faa274382 [term 2 FOLLOWER]: Refusing update from remote peer db42be7ad3c741a0aa427f27e245993b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251025 07:52:45.394994 2712 consensus_queue.cc:1048] T 66f0d398e59a4175aa9fe90715124d1a P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251025 07:52:45.398010 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_a6c869949b744bc98efb735b0e73e7aa [id=bf77b5ff6f1048dd996d948437f9ff5e]). Expected version 2 got 0
I20251025 07:52:45.398283 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3]). Expected version 1 got 0
I20251025 07:52:45.398439 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5]). Expected version 1 got 0
I20251025 07:52:45.399020 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f]). Expected version 1 got 0
I20251025 07:52:45.399233 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7]). Expected version 1 got 0
I20251025 07:52:45.399646 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 872cb7512d104827b1f1f186d33a4a44 (table default.Table_d5c65dce8b1b4214ba519d964012bfc0 [id=d4be274b36a845a99a4b96b8846248fd]). Expected version 1 got 0
I20251025 07:52:45.401176 1167 tablet_bootstrap.cc:654] T e9e8fffb8e984463b5fa2c7dd9e93227 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:45.404340 772 raft_consensus.cc:2468] T 36c414bc700341188b7d85c14173c3e0 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:45.404747 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 129) took 6867 ms. Trace:
I20251025 07:52:45.404879 847 leader_election.cc:304] T 36c414bc700341188b7d85c14173c3e0 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:45.405153 2166 ts_tablet_manager.cc:1929] T 35ba9d23262248cda6fdf03c51be11a4 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:45.405246 2166 log.cc:1199] T 35ba9d23262248cda6fdf03c51be11a4 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/35ba9d23262248cda6fdf03c51be11a4
I20251025 07:52:45.404888 2146 rpcz_store.cc:276] 1025 07:52:38.536909 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.536952 (+ 43us) service_pool.cc:225] Handling call
1025 07:52:45.190982 (+6654030us) ts_tablet_manager.cc:1151] Deleting tablet 7fbae8be73bf4e18b5a5c4ec4cdebc12
1025 07:52:45.190993 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:45.243608 (+ 52615us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:45.351120 (+107512us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:45.404702 (+ 53582us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6653890}
I20251025 07:52:45.405536 2166 ts_tablet_manager.cc:1950] T 35ba9d23262248cda6fdf03c51be11a4 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:45.405541 2146 tablet_replica.cc:333] T 50ae9bca6bed4f749d54191ac1543410 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:45.405776 2146 raft_consensus.cc:2243] T 50ae9bca6bed4f749d54191ac1543410 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:45.406152 2146 raft_consensus.cc:2272] T 50ae9bca6bed4f749d54191ac1543410 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:45.406731 2146 ts_tablet_manager.cc:1916] T 50ae9bca6bed4f749d54191ac1543410 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:45.408757 2698 raft_consensus.cc:2804] T 36c414bc700341188b7d85c14173c3e0 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:45.408962 2698 raft_consensus.cc:697] T 36c414bc700341188b7d85c14173c3e0 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:45.409260 2698 consensus_queue.cc:237] T 36c414bc700341188b7d85c14173c3e0 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.410837 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 7fbae8be73bf4e18b5a5c4ec4cdebc12 (table default.Table_6c5a24747ba14352b0285a41ec2985a4 [id=28bbaad474164a8c8c97a76bb8081791]) successfully deleted
I20251025 07:52:45.414340 1169 tablet_bootstrap.cc:492] T bdaeedbcf35a4aad97f9ef192991978a P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:45.414479 1169 ts_tablet_manager.cc:1403] T bdaeedbcf35a4aad97f9ef192991978a P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.076s user 0.001s sys 0.006s
I20251025 07:52:45.415346 1169 raft_consensus.cc:359] T bdaeedbcf35a4aad97f9ef192991978a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.415602 1169 raft_consensus.cc:385] T bdaeedbcf35a4aad97f9ef192991978a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:45.415978 1166 tablet_bootstrap.cc:492] T c7691c7bdd89437c8db060254c0db25d P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:45.416115 1166 ts_tablet_manager.cc:1403] T c7691c7bdd89437c8db060254c0db25d P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.003s
I20251025 07:52:45.416857 1166 raft_consensus.cc:359] T c7691c7bdd89437c8db060254c0db25d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.417025 1166 raft_consensus.cc:385] T c7691c7bdd89437c8db060254c0db25d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:45.417479 1166 raft_consensus.cc:740] T c7691c7bdd89437c8db060254c0db25d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:45.417734 1166 consensus_queue.cc:260] T c7691c7bdd89437c8db060254c0db25d P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.418365 1166 ts_tablet_manager.cc:1434] T c7691c7bdd89437c8db060254c0db25d P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:45.418668 1166 tablet_bootstrap.cc:492] T 37c7938e8c5e41f08506aff6747801c9 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:45.419817 1169 raft_consensus.cc:740] T bdaeedbcf35a4aad97f9ef192991978a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:45.414364 2467 raft_consensus.cc:515] T 6dcab53a020847a1aa27930265ba0205 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.417691 2597 raft_consensus.cc:493] T 0551e5d694894e8fbeb66b5e0c1cd75b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:45.420130 1169 consensus_queue.cc:260] T bdaeedbcf35a4aad97f9ef192991978a P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
W20251025 07:52:45.416371 982 consensus_peers.cc:597] T 4b28215f61574cf48c447dd01955ac77 P db42be7ad3c741a0aa427f27e245993b -> Peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Couldn't send request to peer 23eeffd3de254ce18db5febfde971c6b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b28215f61574cf48c447dd01955ac77. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:45.420354 1166 tablet_bootstrap.cc:654] T 37c7938e8c5e41f08506aff6747801c9 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:45.420405 2597 raft_consensus.cc:515] T 0551e5d694894e8fbeb66b5e0c1cd75b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.420511 2467 leader_election.cc:290] T 6dcab53a020847a1aa27930265ba0205 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:45.421303 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dcab53a020847a1aa27930265ba0205" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:45.421404 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0551e5d694894e8fbeb66b5e0c1cd75b" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:45.421491 902 raft_consensus.cc:3060] T 6dcab53a020847a1aa27930265ba0205 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:45.422158 2597 leader_election.cc:290] T 0551e5d694894e8fbeb66b5e0c1cd75b P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:45.422180 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 04d27d54fdb7483b869408af65ff85c9 (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]). Expected version 2 got 0
I20251025 07:52:45.422595 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet cb829649f73d4d3f8862d268c953bd86 (table default.Table_a9026f5e095c4063b86d4c544c686e22 [id=7944225d3ce7439b881bfb9289d9f040]). Expected version 2 got 0
I20251025 07:52:45.422801 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 110a8e9e4c6c434284c0ae14dd6987f7 (table default.Table_6c5a24747ba14352b0285a41ec2985a4 [id=28bbaad474164a8c8c97a76bb8081791]). Expected version 2 got 0
I20251025 07:52:45.423734 446 catalog_manager.cc:5649] T b2044ca206634eb285197af4ba2073e7 P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } }
I20251025 07:52:45.424381 446 catalog_manager.cc:5649] T 09898735796541b0a8f3be22cdb6cc30 P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } }
I20251025 07:52:45.424959 1169 ts_tablet_manager.cc:1434] T bdaeedbcf35a4aad97f9ef192991978a P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20251025 07:52:45.425078 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 292fe04055954afd8c08ac2636b68191 (table default.Table_9fa69b10b5e842639fc0b0a58cd9b6a5 [id=a253ce0253bd4da4814741c482193d8f]). Expected version 1 got 0
I20251025 07:52:45.425444 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 5dab439bc50d4e869548154e1d222ad6 (table default.Table_fb9def2302784c999bf0f5707c60eb4c [id=45325c188eac4d34bb016d01de091f87]). Expected version 3 got 0
I20251025 07:52:45.425675 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet aa9d9ab8073945d28d52fb02213f201a (table default.Table_4791672658f043f6b97b9320a531ebb5 [id=dfd20cf9c3e541c4abe23d59e0aa7f2a]). Expected version 1 got 0
I20251025 07:52:45.426209 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet f70091024c034c118fb86578db19a31c (table default.Table_b7da40b96d6c4359bed6f898e8cf6275 [id=e041946c093b4c789ce175c736cc230b]). Expected version 2 got 0
I20251025 07:52:45.426678 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]). Expected version 3 got 0
I20251025 07:52:45.426983 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 5fc092c2bb2d41ac8c5b4fc07460c8ec (table default.Table_27c62e973d74406fb00db8bcf324749c [id=630c274339664b9db18ab4ef7c29c16f]). Expected version 1 got 0
I20251025 07:52:45.427132 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 22f7db660616457b83b24e947a0b0bae (table default.Table_4a86191163f2408f9bcc795cd8d1bbf6 [id=bb316a7e000748ca902a9bfb494802d7]). Expected version 1 got 0
I20251025 07:52:45.427938 2169 ts_tablet_manager.cc:1929] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20251025 07:52:45.427968 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 0325d462441349ef81841f1b0cab2a20 (table default.Table_750ecb905839434fa9a1aa72f87394be [id=9ce3ba77c56a4ef09cb3b40ea0e488f6]). Expected version 1 got 0
I20251025 07:52:45.428058 2169 log.cc:1199] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/7fbae8be73bf4e18b5a5c4ec4cdebc12
I20251025 07:52:45.428442 2169 ts_tablet_manager.cc:1950] T 7fbae8be73bf4e18b5a5c4ec4cdebc12 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:45.428406 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525]). Expected version 1 got 0
I20251025 07:52:45.428575 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_e90d7f5ff13b40878e5b625c97792c7d [id=fc8180c09d7b4e74b2e161f7acb8fe0e]). Expected version 1 got 0
I20251025 07:52:45.428819 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]). Expected version 1 got 0
I20251025 07:52:45.429098 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c]). Expected version 1 got 0
I20251025 07:52:45.429352 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236]). Expected version 1 got 0
I20251025 07:52:45.429874 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 755f0d6c87044e20a8020f8410940123 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 4 got 0
I20251025 07:52:45.430155 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet b56209e4934e4407a8b4518ca4af0d46 (table default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de]). Expected version 1 got 0
I20251025 07:52:45.430294 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 830fdbc6812e416690d0db42a73af503 (table default.Table_92824b3f2a774564b3aea04aaa25d204 [id=fb21a45b86ab446e8c3dae4512a9598f]). Expected version 1 got 0
I20251025 07:52:45.430780 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_87d38832f85e4965bfab1a077990114f [id=ad812862698d421184ebf501ce79b7be]). Expected version 2 got 0
I20251025 07:52:45.430781 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 117) took 6754 ms. Trace:
I20251025 07:52:45.430984 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84]). Expected version 1 got 0
I20251025 07:52:45.430881 2166 rpcz_store.cc:276] 1025 07:52:38.676185 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.676228 (+ 43us) service_pool.cc:225] Handling call
1025 07:52:45.206700 (+6530472us) ts_tablet_manager.cc:1151] Deleting tablet 35ba9d23262248cda6fdf03c51be11a4
1025 07:52:45.206711 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:45.323122 (+116411us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:45.405039 (+ 81917us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:45.430750 (+ 25711us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6530075}
I20251025 07:52:45.431358 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 118) took 6753 ms. Trace:
I20251025 07:52:45.431455 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3]). Expected version 2 got 0
I20251025 07:52:45.431449 2166 rpcz_store.cc:276] 1025 07:52:38.677800 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.677843 (+ 43us) service_pool.cc:225] Handling call
1025 07:52:45.431251 (+6753408us) ts_tablet_manager.cc:1151] Deleting tablet 2209089667dd4937abd96615f6929449
1025 07:52:45.431262 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:45.431341 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6753248}
I20251025 07:52:45.431699 2166 tablet_replica.cc:333] T 74e2d78c024849b0af5c0bde1f48c36f P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:45.431914 2166 raft_consensus.cc:2243] T 74e2d78c024849b0af5c0bde1f48c36f P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:45.431975 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 36262ab8c6964dd684c01b4db6219a4f (table default.Table_76158f8cf82e4cdb9a90580e46aaea48 [id=f249f30497bc450c887d659721469de8]). Expected version 1 got 0
I20251025 07:52:45.432089 2166 raft_consensus.cc:2272] T 74e2d78c024849b0af5c0bde1f48c36f P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:45.432365 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 288e543bb279410c91c01dd1623e110f (table default.Table_6e242ba55b194c23a6701f4b064fdbe5 [id=4c34b64b299f4162a53866cf01c5298f]). Expected version 1 got 0
I20251025 07:52:45.432675 2166 ts_tablet_manager.cc:1916] T 74e2d78c024849b0af5c0bde1f48c36f P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:45.433501 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet ab42909afc524f2fa3c1041495d13a59 (table default.Table_7eb8424d5192485da1a2d67bfdb2bbbf [id=40c29df99590408cbd5c583d4fc741eb]). Expected version 2 got 0
I20251025 07:52:45.433717 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e]). Expected version 1 got 0
I20251025 07:52:45.433908 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet bc2d16b33ca8426abbd7bd38339c9f1d (table default.Table_4ea985df4c5b415c9570c64506a98cc4 [id=98d2b7797e33485e8872054aa2eb8ec9]). Expected version 1 got 0
I20251025 07:52:45.434646 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 35ba9d23262248cda6fdf03c51be11a4 (table default.Table_072e0092adb64ef0ac6834dc18688ccf [id=1d4cbe4a4f38445297ceeebb8ffdaa8e]) successfully deleted
W20251025 07:52:45.435014 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 2209089667dd4937abd96615f6929449 because the tablet was not found. No further retry: Not found: Tablet not found: 2209089667dd4937abd96615f6929449
I20251025 07:52:45.435770 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0551e5d694894e8fbeb66b5e0c1cd75b" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
W20251025 07:52:45.436286 982 leader_election.cc:343] T 6dcab53a020847a1aa27930265ba0205 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: 6dcab53a020847a1aa27930265ba0205
I20251025 07:52:45.436825 982 leader_election.cc:304] T 0551e5d694894e8fbeb66b5e0c1cd75b P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:45.438357 1169 tablet_bootstrap.cc:492] T f656945454fe4921ab819c772a089eb3 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:45.440135 1169 tablet_bootstrap.cc:654] T f656945454fe4921ab819c772a089eb3 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:45.441257 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]). Expected version 1 got 0
I20251025 07:52:45.441488 446 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]). Expected version 1 got 0
I20251025 07:52:45.443333 2728 raft_consensus.cc:2749] T 0551e5d694894e8fbeb66b5e0c1cd75b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:45.443663 449 catalog_manager.cc:6194] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: default.Table_2b9b772d4c234ac286bef3658482e490 [id=409fab60f9d744798e6f92b868f0f7c2] alter complete (version 1)
I20251025 07:52:45.451740 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ec421eb820b4f40afcff5a5bf80c508 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:45.451901 875 tablet_service.cc:1552] Processing DeleteTablet for tablet 5361f08f50b745eb98bf02e979d7c71e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:45.451946 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 7fbae8be73bf4e18b5a5c4ec4cdebc12 with delete_type TABLET_DATA_DELETED (replaced by tablet 4c155aa7fa5b4d68bbab1918b8432372) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:45.452044 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 53198283718d4b45b30df65b5bde96a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:45.452198 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 551d07a9242740dcb9366bb2d707a4a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:59616
W20251025 07:52:45.454928 449 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:38361 (request call id 213) took 1974 ms (client timeout 1000 ms). Trace:
I20251025 07:52:45.455643 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet f3c87a83082b421cb4be546ac4ce7ad8 with delete_type TABLET_DATA_DELETED (replaced by tablet b17d2a9ad6be49d29f2413666fba7e69) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:45.455765 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 08da9d2687704725aacd11cd92c6a9f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:45.455978 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 551d07a9242740dcb9366bb2d707a4a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:45.455993 2687 raft_consensus.cc:493] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:45.456172 2687 raft_consensus.cc:515] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.456806 2687 leader_election.cc:290] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:45.457165 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b6383c99e9e4c90a374da9ccd830a6f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:45.456908 449 rpcz_store.cc:269] 1025 07:52:43.480014 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:43.489041 (+ 9027us) service_pool.cc:225] Handling call
1025 07:52:44.398606 (+909565us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.398611 (+ 5us) write_op.cc:454] Released schema lock
1025 07:52:44.869858 (+471247us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.869879 (+ 21us) write_op.cc:454] Released schema lock
1025 07:52:45.454857 (+584978us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:44.391789 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:44.391937 (+ 148us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:44.391944 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:44.391946 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:44.392068 (+ 122us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:44.392082 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:44.392088 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:44.392152 (+ 64us) tablet.cc:699] Row locks acquired
1025 07:52:44.392154 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:44.392192 (+ 38us) write_op.cc:270] Start()
1025 07:52:44.392225 (+ 33us) write_op.cc:276] Timestamp: P: 1761378764392173 usec, L: 0
1025 07:52:44.392229 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:44.392422 (+ 193us) log.cc:844] Serialized 527 byte log entry
1025 07:52:44.396213 (+ 3791us) op_driver.cc:464] REPLICATION: finished
1025 07:52:44.396327 (+ 114us) write_op.cc:301] APPLY: starting
1025 07:52:44.396343 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:44.396380 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:44.396413 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:44.396462 (+ 49us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:44.396498 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:44.396506 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:44.396573 (+ 67us) log.cc:844] Serialized 23 byte log entry
1025 07:52:44.396628 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.396645 (+ 17us) write_op.cc:454] Released schema lock
1025 07:52:44.396652 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
1025 07:52:44.838885 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:44.839044 (+ 159us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:44.839051 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:44.839054 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:44.839156 (+ 102us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:44.839171 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:44.839176 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:44.839240 (+ 64us) tablet.cc:699] Row locks acquired
1025 07:52:44.839243 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:44.839280 (+ 37us) write_op.cc:270] Start()
1025 07:52:44.839314 (+ 34us) write_op.cc:276] Timestamp: P: 1761378764839261 usec, L: 0
1025 07:52:44.839318 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:44.839522 (+ 204us) log.cc:844] Serialized 356 byte log entry
1025 07:52:44.843265 (+ 3743us) op_driver.cc:464] REPLICATION: finished
1025 07:52:44.843683 (+ 418us) write_op.cc:301] APPLY: starting
1025 07:52:44.843703 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:44.843759 (+ 56us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:44.843793 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:44.843853 (+ 60us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:44.843888 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:44.843906 (+ 18us) write_op.cc:312] APPLY: finished
1025 07:52:44.843992 (+ 86us) log.cc:844] Serialized 23 byte log entry
1025 07:52:44.844055 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.844078 (+ 23us) write_op.cc:454] Released schema lock
1025 07:52:44.844088 (+ 10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":244,"child_traces":[["op",{"apply.queue_time_us":66,"apply.run_cpu_time_us":375,"apply.run_wall_time_us":374,"num_ops":1,"prepare.queue_time_us":35426,"prepare.run_cpu_time_us":751,"prepare.run_wall_time_us":751,"raft.queue_time_us":400,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":248,"replication_time_us":3956}],["op",{"apply.queue_time_us":327,"apply.run_cpu_time_us":472,"apply.run_wall_time_us":473,"num_ops":1,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":748,"prepare.run_wall_time_us":748,"raft.queue_time_us":391,"raft.run_cpu_time_us":375,"raft.run_wall_time_us":375,"replication_time_us":3918}]]}
W20251025 07:52:45.457672 850 leader_election.cc:343] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:45.458465 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b6383c99e9e4c90a374da9ccd830a6f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:45.458652 772 raft_consensus.cc:2468] T 5b6383c99e9e4c90a374da9ccd830a6f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
W20251025 07:52:45.459322 450 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:42387 (request call id 215) took 2169 ms (client timeout 1000 ms). Trace:
W20251025 07:52:45.459511 450 rpcz_store.cc:269] 1025 07:52:43.289883 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:43.428743 (+138860us) service_pool.cc:225] Handling call
1025 07:52:44.139092 (+710349us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.139099 (+ 7us) write_op.cc:454] Released schema lock
1025 07:52:45.459285 (+1320186us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:44.129801 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:44.129992 (+ 191us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:44.130002 (+ 10us) write_op.cc:435] Acquired schema lock
1025 07:52:44.130007 (+ 5us) tablet.cc:660] Decoding operations
1025 07:52:44.130342 (+ 335us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:44.130360 (+ 18us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:44.130367 (+ 7us) tablet.cc:683] Acquiring locks for 10 operations
1025 07:52:44.130564 (+ 197us) tablet.cc:699] Row locks acquired
1025 07:52:44.130566 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:44.130603 (+ 37us) write_op.cc:270] Start()
1025 07:52:44.130636 (+ 33us) write_op.cc:276] Timestamp: P: 1761378764130580 usec, L: 0
1025 07:52:44.130640 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:44.130854 (+ 214us) log.cc:844] Serialized 3894 byte log entry
1025 07:52:44.135939 (+ 5085us) op_driver.cc:464] REPLICATION: finished
1025 07:52:44.136073 (+ 134us) write_op.cc:301] APPLY: starting
1025 07:52:44.136090 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:44.136163 (+ 73us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:44.136190 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:44.136360 (+ 170us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:44.136407 (+ 47us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=10
1025 07:52:44.136415 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:44.136517 (+ 102us) log.cc:844] Serialized 77 byte log entry
1025 07:52:44.136582 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.136617 (+ 35us) write_op.cc:454] Released schema lock
1025 07:52:44.136625 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":93836,"reported_tablets":235,"child_traces":[["op",{"apply.queue_time_us":83,"apply.run_cpu_time_us":606,"apply.run_wall_time_us":606,"num_ops":10,"prepare.queue_time_us":241,"prepare.run_cpu_time_us":1165,"prepare.run_wall_time_us":1166,"raft.queue_time_us":401,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":280,"replication_time_us":5267}]]}
I20251025 07:52:45.460757 902 raft_consensus.cc:2468] T 6dcab53a020847a1aa27930265ba0205 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:45.461231 983 leader_election.cc:304] T 6dcab53a020847a1aa27930265ba0205 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b
I20251025 07:52:45.461719 2597 raft_consensus.cc:2804] T 6dcab53a020847a1aa27930265ba0205 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:45.461941 2597 raft_consensus.cc:697] T 6dcab53a020847a1aa27930265ba0205 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:45.461915 450 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_acb609759cce423fa0dd70c8d6be1373"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:45.462237 2597 consensus_queue.cc:237] T 6dcab53a020847a1aa27930265ba0205 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.460508 449 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_27c62e973d74406fb00db8bcf324749c" } new_table_name: "default.Table_cd20e202397645f8821dcde13b5c0ce3" modify_external_catalogs: true
W20251025 07:52:45.463492 450 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_acb609759cce423fa0dd70c8d6be1373 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:45.464306 2146 ts_tablet_manager.cc:1929] T 50ae9bca6bed4f749d54191ac1543410 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:45.464514 2146 log.cc:1199] T 50ae9bca6bed4f749d54191ac1543410 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/50ae9bca6bed4f749d54191ac1543410
I20251025 07:52:45.464990 2146 ts_tablet_manager.cc:1950] T 50ae9bca6bed4f749d54191ac1543410 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:45.466063 1167 tablet_bootstrap.cc:492] T e9e8fffb8e984463b5fa2c7dd9e93227 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:45.466233 1167 ts_tablet_manager.cc:1403] T e9e8fffb8e984463b5fa2c7dd9e93227 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.073s user 0.003s sys 0.001s
I20251025 07:52:45.467042 1167 raft_consensus.cc:359] T e9e8fffb8e984463b5fa2c7dd9e93227 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.467211 1167 raft_consensus.cc:385] T e9e8fffb8e984463b5fa2c7dd9e93227 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:45.467260 1167 raft_consensus.cc:740] T e9e8fffb8e984463b5fa2c7dd9e93227 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:45.467458 1167 consensus_queue.cc:260] T e9e8fffb8e984463b5fa2c7dd9e93227 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.467926 1167 ts_tablet_manager.cc:1434] T e9e8fffb8e984463b5fa2c7dd9e93227 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 07:52:45.468200 1167 tablet_bootstrap.cc:492] T dff7f86ae8924685b26011e504e2247b P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:45.469942 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 132) took 6646 ms. Trace:
I20251025 07:52:45.470041 2169 rpcz_store.cc:276] 1025 07:52:38.824014 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.824441 (+ 427us) service_pool.cc:225] Handling call
1025 07:52:45.314252 (+6489811us) ts_tablet_manager.cc:1151] Deleting tablet 7fbae8be73bf4e18b5a5c4ec4cdebc12
1025 07:52:45.314263 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:45.405831 (+ 91568us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:45.427790 (+ 21959us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:45.469919 (+ 42129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6489654}
I20251025 07:52:45.470463 2169 tablet_replica.cc:333] T 6ec421eb820b4f40afcff5a5bf80c508 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:45.470646 2169 raft_consensus.cc:2243] T 6ec421eb820b4f40afcff5a5bf80c508 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:45.470829 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 7fbae8be73bf4e18b5a5c4ec4cdebc12 (table default.Table_6c5a24747ba14352b0285a41ec2985a4 [id=28bbaad474164a8c8c97a76bb8081791]) successfully deleted
I20251025 07:52:45.471019 2169 raft_consensus.cc:2272] T 6ec421eb820b4f40afcff5a5bf80c508 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:45.471549 2169 ts_tablet_manager.cc:1916] T 6ec421eb820b4f40afcff5a5bf80c508 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:45.472440 1169 tablet_bootstrap.cc:492] T f656945454fe4921ab819c772a089eb3 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:45.472579 1169 ts_tablet_manager.cc:1403] T f656945454fe4921ab819c772a089eb3 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.006s
I20251025 07:52:45.473371 1169 raft_consensus.cc:359] T f656945454fe4921ab819c772a089eb3 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.473510 1169 raft_consensus.cc:385] T f656945454fe4921ab819c772a089eb3 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:45.473551 1169 raft_consensus.cc:740] T f656945454fe4921ab819c772a089eb3 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:45.473711 1169 consensus_queue.cc:260] T f656945454fe4921ab819c772a089eb3 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.474272 1169 ts_tablet_manager.cc:1434] T f656945454fe4921ab819c772a089eb3 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:45.474512 1169 tablet_bootstrap.cc:492] T 4678f7eab5e8497db308e8a38e387618 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:45.475687 847 leader_election.cc:304] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:45.476598 1169 tablet_bootstrap.cc:654] T 4678f7eab5e8497db308e8a38e387618 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:45.477267 2686 raft_consensus.cc:2804] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:45.477362 2686 raft_consensus.cc:493] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:45.477451 2686 raft_consensus.cc:3060] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:45.478988 1048 raft_consensus.cc:3060] T ab42909afc524f2fa3c1041495d13a59 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:45.486379 450 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 93) took 1818 ms. Trace:
I20251025 07:52:45.486583 450 rpcz_store.cc:276] 1025 07:52:43.667628 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.461724 (+1794096us) service_pool.cc:225] Handling call
1025 07:52:45.464028 (+ 2304us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:45.464455 (+ 427us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:45.486039 (+ 21584us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.486045 (+ 6us) write_op.cc:454] Released schema lock
1025 07:52:45.486112 (+ 67us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:45.486242 (+ 130us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:45.486341 (+ 99us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:45.475657 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:45.475774 (+ 117us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:45.475779 (+ 5us) write_op.cc:435] Acquired schema lock
1025 07:52:45.475783 (+ 4us) tablet.cc:660] Decoding operations
1025 07:52:45.475889 (+ 106us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:45.475904 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:45.475908 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:45.475983 (+ 75us) tablet.cc:699] Row locks acquired
1025 07:52:45.475985 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:45.476017 (+ 32us) write_op.cc:270] Start()
1025 07:52:45.476047 (+ 30us) write_op.cc:276] Timestamp: P: 1761378765473493 usec, L: 0
1025 07:52:45.476051 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:45.476217 (+ 166us) log.cc:844] Serialized 433 byte log entry
1025 07:52:45.478905 (+ 2688us) op_driver.cc:464] REPLICATION: finished
1025 07:52:45.479004 (+ 99us) write_op.cc:301] APPLY: starting
1025 07:52:45.479019 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:45.479061 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:45.479096 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:45.479200 (+ 104us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:45.479232 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:45.479239 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:45.479311 (+ 72us) log.cc:844] Serialized 29 byte log entry
1025 07:52:45.479368 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.479387 (+ 19us) write_op.cc:454] Released schema lock
1025 07:52:45.479393 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":53,"apply.run_cpu_time_us":440,"apply.run_wall_time_us":439,"mutex_wait_us":10759,"num_ops":2,"prepare.queue_time_us":10807,"prepare.run_cpu_time_us":661,"prepare.run_wall_time_us":660,"raft.queue_time_us":345,"raft.run_cpu_time_us":381,"raft.run_wall_time_us":380,"replication_time_us":2826}]]}
I20251025 07:52:45.496708 450 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_7dcc5d1b56a9449e8cd9f533a5bef482"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:45.497488 450 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7dcc5d1b56a9449e8cd9f533a5bef482 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:45.502010 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 130) took 6964 ms. Trace:
I20251025 07:52:45.502207 2146 rpcz_store.cc:276] 1025 07:52:38.537241 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.537276 (+ 35us) service_pool.cc:225] Handling call
1025 07:52:45.405479 (+6868203us) ts_tablet_manager.cc:1151] Deleting tablet 50ae9bca6bed4f749d54191ac1543410
1025 07:52:45.405489 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:45.460693 (+ 55204us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:45.464184 (+ 3491us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:45.501946 (+ 37762us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6868082}
I20251025 07:52:45.502822 2146 tablet_replica.cc:333] T 4b28215f61574cf48c447dd01955ac77 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:45.504949 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 50ae9bca6bed4f749d54191ac1543410 (table default.Table_b42187d20a7e4f8783a8d25acb249350 [id=a5c73cb1e5a349ca8675f4ba4cb8a82d]) successfully deleted
I20251025 07:52:45.510260 450 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 94) took 1816 ms. Trace:
W20251025 07:52:45.511129 447 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8707b93d491e4e4386d148087a2f0870 consensus state has no leader
I20251025 07:52:45.511144 1167 tablet_bootstrap.cc:654] T dff7f86ae8924685b26011e504e2247b P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:45.511305 447 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:47610 (request call id 91) took 1871 ms. Trace:
I20251025 07:52:45.511387 447 rpcz_store.cc:276] 1025 07:52:43.639685 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.077307 (+1437622us) service_pool.cc:225] Handling call
1025 07:52:45.077750 (+ 443us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:45.077987 (+ 237us) catalog_manager.cc:3708] Apply alter schema
1025 07:52:45.078199 (+ 212us) catalog_manager.cc:3740] Acquired catalog manager lock
1025 07:52:45.078290 (+ 91us) catalog_manager.cc:3898] Updating metadata on disk
1025 07:52:45.113803 (+ 35513us) catalog_manager.cc:3929] Committing alterations to in-memory state
1025 07:52:45.511253 (+397450us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:45.078800 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:45.078909 (+ 109us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:45.078914 (+ 5us) write_op.cc:435] Acquired schema lock
1025 07:52:45.078916 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:45.079014 (+ 98us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:45.079025 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:45.079029 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:45.079087 (+ 58us) tablet.cc:699] Row locks acquired
1025 07:52:45.079090 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:45.079122 (+ 32us) write_op.cc:270] Start()
1025 07:52:45.079155 (+ 33us) write_op.cc:276] Timestamp: P: 1761378765079109 usec, L: 0
1025 07:52:45.079160 (+ 5us) op_driver.cc:348] REPLICATION: starting
1025 07:52:45.079321 (+ 161us) log.cc:844] Serialized 405 byte log entry
1025 07:52:45.098759 (+ 19438us) op_driver.cc:464] REPLICATION: finished
1025 07:52:45.109034 (+ 10275us) write_op.cc:301] APPLY: starting
1025 07:52:45.109053 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:45.109092 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:45.109147 (+ 55us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:45.109209 (+ 62us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:45.109253 (+ 44us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:45.109259 (+ 6us) write_op.cc:312] APPLY: finished
1025 07:52:45.109386 (+ 127us) log.cc:844] Serialized 23 byte log entry
1025 07:52:45.112702 (+ 3316us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.112722 (+ 20us) write_op.cc:454] Released schema lock
1025 07:52:45.112728 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:45.112798 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.112802 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":10169,"apply.run_cpu_time_us":531,"apply.run_wall_time_us":3769,"num_ops":1,"prepare.queue_time_us":43,"prepare.run_cpu_time_us":585,"prepare.run_wall_time_us":585,"replication_time_us":19567}]]}
I20251025 07:52:45.512212 447 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_5fd6059c44b34bc5a067c100d84f4c3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:45.512822 447 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5fd6059c44b34bc5a067c100d84f4c3d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:45.529498 2663 raft_consensus.cc:493] T 8956ad28b35d45f28c683ac055bba441 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db42be7ad3c741a0aa427f27e245993b)
I20251025 07:52:45.529671 2663 raft_consensus.cc:515] T 8956ad28b35d45f28c683ac055bba441 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:45.530303 2663 leader_election.cc:290] T 8956ad28b35d45f28c683ac055bba441 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:45.530540 447 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 95) took 1587 ms. Trace:
I20251025 07:52:45.530652 447 rpcz_store.cc:276] 1025 07:52:43.943429 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.512007 (+1568578us) service_pool.cc:225] Handling call
1025 07:52:45.512911 (+ 904us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:45.513348 (+ 437us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:45.530260 (+ 16912us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:45.530379 (+ 119us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:45.530479 (+ 100us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:45.514175 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:45.514278 (+ 103us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:45.514284 (+ 6us) write_op.cc:435] Acquired schema lock
1025 07:52:45.514286 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:45.514396 (+ 110us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:45.514413 (+ 17us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:45.514421 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:45.514517 (+ 96us) tablet.cc:699] Row locks acquired
1025 07:52:45.514519 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:45.514555 (+ 36us) write_op.cc:270] Start()
1025 07:52:45.514589 (+ 34us) write_op.cc:276] Timestamp: P: 1761378765512030 usec, L: 0
1025 07:52:45.514594 (+ 5us) op_driver.cc:348] REPLICATION: starting
1025 07:52:45.514802 (+ 208us) log.cc:844] Serialized 433 byte log entry
1025 07:52:45.521537 (+ 6735us) op_driver.cc:464] REPLICATION: finished
1025 07:52:45.521678 (+ 141us) write_op.cc:301] APPLY: starting
1025 07:52:45.521694 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:45.521737 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:45.521762 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:45.521881 (+ 119us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:45.521915 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:45.521922 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:45.521991 (+ 69us) log.cc:844] Serialized 29 byte log entry
1025 07:52:45.522048 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.522067 (+ 19us) write_op.cc:454] Released schema lock
1025 07:52:45.522072 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:45.522155 (+ 83us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.522159 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":90,"apply.run_cpu_time_us":451,"apply.run_wall_time_us":451,"num_ops":2,"prepare.queue_time_us":314,"prepare.run_cpu_time_us":736,"prepare.run_wall_time_us":737,"raft.queue_time_us":1882,"raft.run_cpu_time_us":460,"raft.run_wall_time_us":459,"replication_time_us":6914}]]}
I20251025 07:52:45.531373 447 master_service.cc:954] ReplaceTablet: received request to replace tablet 0551e5d694894e8fbeb66b5e0c1cd75b from {username='slave'} at 127.0.0.1:47610
W20251025 07:52:45.531512 712 leader_election.cc:343] T 8956ad28b35d45f28c683ac055bba441 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 8956ad28b35d45f28c683ac055bba441
I20251025 07:52:45.534165 1166 tablet_bootstrap.cc:492] T 37c7938e8c5e41f08506aff6747801c9 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:45.534317 1166 ts_tablet_manager.cc:1403] T 37c7938e8c5e41f08506aff6747801c9 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.116s user 0.003s sys 0.009s
I20251025 07:52:45.535107 1166 raft_consensus.cc:359] T 37c7938e8c5e41f08506aff6747801c9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.535266 1166 raft_consensus.cc:385] T 37c7938e8c5e41f08506aff6747801c9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:45.535320 1166 raft_consensus.cc:740] T 37c7938e8c5e41f08506aff6747801c9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:45.535509 1166 consensus_queue.cc:260] T 37c7938e8c5e41f08506aff6747801c9 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.535985 1166 ts_tablet_manager.cc:1434] T 37c7938e8c5e41f08506aff6747801c9 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20251025 07:52:45.536319 449 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5fc092c2bb2d41ac8c5b4fc07460c8ec consensus state has no leader
W20251025 07:52:45.536448 710 leader_election.cc:343] T 8956ad28b35d45f28c683ac055bba441 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 8956ad28b35d45f28c683ac055bba441
I20251025 07:52:45.536517 449 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:47610 (request call id 92) took 1893 ms. Trace:
I20251025 07:52:45.536561 710 leader_election.cc:304] T 8956ad28b35d45f28c683ac055bba441 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:45.537022 2663 raft_consensus.cc:2749] T 8956ad28b35d45f28c683ac055bba441 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 07:52:45.536592 449 rpcz_store.cc:276] 1025 07:52:43.643297 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.460317 (+1817020us) service_pool.cc:225] Handling call
1025 07:52:45.462522 (+ 2205us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:45.462810 (+ 288us) catalog_manager.cc:3708] Apply alter schema
1025 07:52:45.462983 (+ 173us) catalog_manager.cc:3740] Acquired catalog manager lock
1025 07:52:45.463038 (+ 55us) catalog_manager.cc:3898] Updating metadata on disk
1025 07:52:45.475416 (+ 12378us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.475422 (+ 6us) write_op.cc:454] Released schema lock
1025 07:52:45.475496 (+ 74us) catalog_manager.cc:3929] Committing alterations to in-memory state
1025 07:52:45.536461 (+ 60965us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:45.464874 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:45.464983 (+ 109us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:45.464990 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:45.464992 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:45.465084 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:45.465099 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:45.465103 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:45.465161 (+ 58us) tablet.cc:699] Row locks acquired
1025 07:52:45.465164 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:45.465194 (+ 30us) write_op.cc:270] Start()
1025 07:52:45.465225 (+ 31us) write_op.cc:276] Timestamp: P: 1761378765462670 usec, L: 0
1025 07:52:45.465228 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:45.465385 (+ 157us) log.cc:844] Serialized 405 byte log entry
1025 07:52:45.468403 (+ 3018us) op_driver.cc:464] REPLICATION: finished
1025 07:52:45.468508 (+ 105us) write_op.cc:301] APPLY: starting
1025 07:52:45.468524 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:45.468568 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:45.468592 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:45.468657 (+ 65us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:45.468688 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:45.468695 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:45.468762 (+ 67us) log.cc:844] Serialized 23 byte log entry
1025 07:52:45.468821 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.468844 (+ 23us) write_op.cc:454] Released schema lock
1025 07:52:45.468853 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":55,"apply.run_cpu_time_us":404,"apply.run_wall_time_us":403,"num_ops":1,"prepare.queue_time_us":878,"prepare.run_cpu_time_us":612,"prepare.run_wall_time_us":613,"raft.queue_time_us":443,"raft.run_cpu_time_us":464,"raft.run_wall_time_us":463,"replication_time_us":3144}]]}
I20251025 07:52:45.537643 449 master_service.cc:954] ReplaceTablet: received request to replace tablet 623dec4eab1d4666baf2334a01eeb36d from {username='slave'} at 127.0.0.1:47610
W20251025 07:52:45.538053 454 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3a0f50c8000144a4acb5219484b01800 consensus state has no leader
I20251025 07:52:45.538216 454 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:47610 (request call id 89) took 1909 ms. Trace:
I20251025 07:52:45.538311 454 rpcz_store.cc:276] 1025 07:52:43.629055 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.021431 (+1392376us) service_pool.cc:225] Handling call
1025 07:52:45.039011 (+ 17580us) spinlock_profiling.cc:243] Waited 17.3 ms on lock 0x616000041254. stack: 00007fb90f7779f6 00007fb90f7777b0 00007fb90e7533ad 0000000000983f04 0000000000983ecd 00007fb91ea756bd 00007fb91ea757b3 00007fb91b640441 00007fb9232f69dd 00007fb9234854f6 00007fb919f9d146 00007fb919f9ce75 00007fb91671327e 00007fb91671215e 00007fb916718395 00007fb916718cbf
1025 07:52:45.039219 (+ 208us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:45.039388 (+ 169us) catalog_manager.cc:3708] Apply alter schema
1025 07:52:45.039573 (+ 185us) catalog_manager.cc:3740] Acquired catalog manager lock
1025 07:52:45.039677 (+ 104us) catalog_manager.cc:3898] Updating metadata on disk
1025 07:52:45.075583 (+ 35906us) catalog_manager.cc:3929] Committing alterations to in-memory state
1025 07:52:45.538187 (+462604us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:45.059078 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:45.059171 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:45.059175 (+ 4us) write_op.cc:435] Acquired schema lock
1025 07:52:45.059177 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:45.059258 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:45.059268 (+ 10us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:45.059272 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:45.059323 (+ 51us) tablet.cc:699] Row locks acquired
1025 07:52:45.059325 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:45.059349 (+ 24us) write_op.cc:270] Start()
1025 07:52:45.059373 (+ 24us) write_op.cc:276] Timestamp: P: 1761378765059337 usec, L: 0
1025 07:52:45.059376 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:45.059524 (+ 148us) log.cc:844] Serialized 405 byte log entry
1025 07:52:45.067623 (+ 8099us) op_driver.cc:464] REPLICATION: finished
1025 07:52:45.067827 (+ 204us) write_op.cc:301] APPLY: starting
1025 07:52:45.067844 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:45.067898 (+ 54us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:45.067931 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:45.067984 (+ 53us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:45.068038 (+ 54us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:45.068049 (+ 11us) write_op.cc:312] APPLY: finished
1025 07:52:45.068126 (+ 77us) log.cc:844] Serialized 23 byte log entry
1025 07:52:45.068189 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.068208 (+ 19us) write_op.cc:454] Released schema lock
1025 07:52:45.068216 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:45.075518 (+ 7302us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.075525 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":39853440,"child_traces":[["op",{"apply.queue_time_us":153,"apply.run_cpu_time_us":485,"apply.run_wall_time_us":7673,"num_ops":1,"prepare.queue_time_us":18942,"prepare.run_cpu_time_us":554,"prepare.run_wall_time_us":895,"raft.queue_time_us":47,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":245,"replication_time_us":8213}]]}
I20251025 07:52:45.538872 1169 tablet_bootstrap.cc:492] T 4678f7eab5e8497db308e8a38e387618 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:45.538978 454 master_service.cc:954] ReplaceTablet: received request to replace tablet cb829649f73d4d3f8862d268c953bd86 from {username='slave'} at 127.0.0.1:47610
I20251025 07:52:45.539003 1169 ts_tablet_manager.cc:1403] T 4678f7eab5e8497db308e8a38e387618 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.065s user 0.004s sys 0.000s
I20251025 07:52:45.539716 1169 raft_consensus.cc:359] T 4678f7eab5e8497db308e8a38e387618 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
W20251025 07:52:45.539810 455 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d46673a283fe42b78ddeb99fbd2678eb consensus state has no leader
I20251025 07:52:45.539883 1169 raft_consensus.cc:385] T 4678f7eab5e8497db308e8a38e387618 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:45.539947 1169 raft_consensus.cc:740] T 4678f7eab5e8497db308e8a38e387618 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:45.539950 455 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:47610 (request call id 88) took 2190 ms. Trace:
I20251025 07:52:45.540134 1169 consensus_queue.cc:260] T 4678f7eab5e8497db308e8a38e387618 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.540042 455 rpcz_store.cc:276] 1025 07:52:43.349659 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.014443 (+1664784us) service_pool.cc:225] Handling call
1025 07:52:45.037805 (+ 23362us) spinlock_profiling.cc:243] Waited 23 ms on lock 0x616000041254. stack: 00007fb90f7779f6 00007fb90f7777b0 00007fb90e7533ad 0000000000983f04 0000000000983ecd 00007fb91ea756bd 00007fb91ea757b3 00007fb91b640441 00007fb9232f69dd 00007fb9234854f6 00007fb919f9d146 00007fb919f9ce75 00007fb91671327e 00007fb91671215e 00007fb916718395 00007fb916718cbf
1025 07:52:45.038018 (+ 213us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:45.038238 (+ 220us) catalog_manager.cc:3708] Apply alter schema
1025 07:52:45.038406 (+ 168us) catalog_manager.cc:3740] Acquired catalog manager lock
1025 07:52:45.038463 (+ 57us) catalog_manager.cc:3898] Updating metadata on disk
1025 07:52:45.058300 (+ 19837us) catalog_manager.cc:3929] Committing alterations to in-memory state
1025 07:52:45.539925 (+481625us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:45.043386 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:45.043490 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:45.043494 (+ 4us) write_op.cc:435] Acquired schema lock
1025 07:52:45.043497 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:45.043574 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:45.043583 (+ 9us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:45.043588 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:45.043666 (+ 78us) tablet.cc:699] Row locks acquired
1025 07:52:45.043669 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:45.043696 (+ 27us) write_op.cc:270] Start()
1025 07:52:45.043722 (+ 26us) write_op.cc:276] Timestamp: P: 1761378765043682 usec, L: 0
1025 07:52:45.043725 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:45.043876 (+ 151us) log.cc:844] Serialized 405 byte log entry
1025 07:52:45.053075 (+ 9199us) op_driver.cc:464] REPLICATION: finished
1025 07:52:45.053822 (+ 747us) write_op.cc:301] APPLY: starting
1025 07:52:45.053833 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:45.053864 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:45.053894 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:45.053940 (+ 46us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:45.053965 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:45.053971 (+ 6us) write_op.cc:312] APPLY: finished
1025 07:52:45.054029 (+ 58us) log.cc:844] Serialized 23 byte log entry
1025 07:52:45.054075 (+ 46us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.054094 (+ 19us) write_op.cc:454] Released schema lock
1025 07:52:45.054102 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:45.059008 (+ 4906us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.059014 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":52989696,"child_traces":[["op",{"apply.queue_time_us":707,"apply.run_cpu_time_us":370,"apply.run_wall_time_us":369,"num_ops":1,"prepare.queue_time_us":4531,"prepare.run_cpu_time_us":609,"prepare.run_wall_time_us":15629,"raft.queue_time_us":25,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":78,"replication_time_us":9322}]]}
I20251025 07:52:45.540653 455 master_service.cc:954] ReplaceTablet: received request to replace tablet ef1854d35a3047e3a9fbe0166ab1295a from {username='slave'} at 127.0.0.1:47610
I20251025 07:52:45.543668 1169 ts_tablet_manager.cc:1434] T 4678f7eab5e8497db308e8a38e387618 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251025 07:52:45.543965 1169 tablet_bootstrap.cc:492] T 91728987303b41fd840eb56f3a311c5f P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:45.546298 1169 tablet_bootstrap.cc:654] T 91728987303b41fd840eb56f3a311c5f P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:45.560377 2686 raft_consensus.cc:515] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.561034 2686 leader_election.cc:290] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:45.565379 452 catalog_manager.cc:6194] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: default.Table_a9026f5e095c4063b86d4c544c686e22 [id=7944225d3ce7439b881bfb9289d9f040] alter complete (version 2)
I20251025 07:52:45.566215 1167 tablet_bootstrap.cc:492] T dff7f86ae8924685b26011e504e2247b P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:45.566337 1167 ts_tablet_manager.cc:1403] T dff7f86ae8924685b26011e504e2247b P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.098s user 0.012s sys 0.011s
I20251025 07:52:45.567754 1169 tablet_bootstrap.cc:492] T 91728987303b41fd840eb56f3a311c5f P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:45.567886 1169 ts_tablet_manager.cc:1403] T 91728987303b41fd840eb56f3a311c5f P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.004s
I20251025 07:52:45.568612 1169 raft_consensus.cc:359] T 91728987303b41fd840eb56f3a311c5f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.568763 1169 raft_consensus.cc:385] T 91728987303b41fd840eb56f3a311c5f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:45.568809 1169 raft_consensus.cc:740] T 91728987303b41fd840eb56f3a311c5f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:45.569018 1169 consensus_queue.cc:260] T 91728987303b41fd840eb56f3a311c5f P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.570036 2687 consensus_queue.cc:1048] T ab42909afc524f2fa3c1041495d13a59 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:45.570925 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b6383c99e9e4c90a374da9ccd830a6f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:45.571938 850 leader_election.cc:343] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:45.572546 1169 ts_tablet_manager.cc:1434] T 91728987303b41fd840eb56f3a311c5f P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251025 07:52:45.572826 1169 tablet_bootstrap.cc:492] T 11bfe0c87ca941a3846462c8a4512d83 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:45.576216 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b6383c99e9e4c90a374da9ccd830a6f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:45.576756 772 raft_consensus.cc:3060] T 5b6383c99e9e4c90a374da9ccd830a6f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:45.577589 2687 consensus_queue.cc:1048] T 30418858bf95438686bdfe64ac733144 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:45.579869 1167 raft_consensus.cc:359] T dff7f86ae8924685b26011e504e2247b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.580016 1167 raft_consensus.cc:385] T dff7f86ae8924685b26011e504e2247b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:45.580062 1167 raft_consensus.cc:740] T dff7f86ae8924685b26011e504e2247b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:45.580291 1167 consensus_queue.cc:260] T dff7f86ae8924685b26011e504e2247b P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.580770 1167 ts_tablet_manager.cc:1434] T dff7f86ae8924685b26011e504e2247b P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20251025 07:52:45.581046 1167 tablet_bootstrap.cc:492] T 1068f566477e4c6db7dde566fff4c2fb P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:45.584146 1167 tablet_bootstrap.cc:654] T 1068f566477e4c6db7dde566fff4c2fb P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:45.510468 450 rpcz_store.cc:276] 1025 07:52:43.693682 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.496455 (+1802773us) service_pool.cc:225] Handling call
1025 07:52:45.497659 (+ 1204us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:45.498227 (+ 568us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:45.509450 (+ 11223us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.509458 (+ 8us) write_op.cc:454] Released schema lock
1025 07:52:45.509531 (+ 73us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:45.509653 (+ 122us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:45.510218 (+ 565us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:45.498798 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:45.498930 (+ 132us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:45.498938 (+ 8us) write_op.cc:435] Acquired schema lock
1025 07:52:45.498940 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:45.499032 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:45.499046 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:45.499051 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:45.499146 (+ 95us) tablet.cc:699] Row locks acquired
1025 07:52:45.499148 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:45.499184 (+ 36us) write_op.cc:270] Start()
1025 07:52:45.499222 (+ 38us) write_op.cc:276] Timestamp: P: 1761378765496659 usec, L: 0
1025 07:52:45.499225 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:45.499400 (+ 175us) log.cc:844] Serialized 433 byte log entry
1025 07:52:45.502778 (+ 3378us) op_driver.cc:464] REPLICATION: finished
1025 07:52:45.502879 (+ 101us) write_op.cc:301] APPLY: starting
1025 07:52:45.502894 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:45.502936 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:45.502967 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:45.503069 (+ 102us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:45.503102 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:45.503109 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:45.503183 (+ 74us) log.cc:844] Serialized 29 byte log entry
1025 07:52:45.503251 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.503273 (+ 22us) write_op.cc:454] Released schema lock
1025 07:52:45.503279 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":55,"apply.run_cpu_time_us":450,"apply.run_wall_time_us":449,"num_ops":2,"prepare.queue_time_us":45,"prepare.run_cpu_time_us":713,"prepare.run_wall_time_us":712,"raft.queue_time_us":414,"raft.run_cpu_time_us":481,"raft.run_wall_time_us":478,"replication_time_us":3512}]]}
I20251025 07:52:45.592378 2675 catalog_manager.cc:1269] Loaded cluster ID: aee99068aba246df9094aaec3350d6ff
I20251025 07:52:45.593467 2597 raft_consensus.cc:493] T f661d5909657455581dcb5e6eadd3c0b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:45.593636 2597 raft_consensus.cc:515] T f661d5909657455581dcb5e6eadd3c0b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.594401 2597 leader_election.cc:290] T f661d5909657455581dcb5e6eadd3c0b P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:45.595089 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f661d5909657455581dcb5e6eadd3c0b" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:45.595641 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f661d5909657455581dcb5e6eadd3c0b" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:45.591089 451 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet df0d86ad301245c1a6bcb44805936d25 consensus state has no leader
I20251025 07:52:45.595922 1169 tablet_bootstrap.cc:654] T 11bfe0c87ca941a3846462c8a4512d83 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:45.595945 451 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:47610 (request call id 90) took 1959 ms. Trace:
I20251025 07:52:45.596074 451 rpcz_store.cc:276] 1025 07:52:43.636355 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.056979 (+1420624us) service_pool.cc:225] Handling call
1025 07:52:45.057315 (+ 336us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:45.057562 (+ 247us) catalog_manager.cc:3708] Apply alter schema
1025 07:52:45.057774 (+ 212us) catalog_manager.cc:3740] Acquired catalog manager lock
1025 07:52:45.057828 (+ 54us) catalog_manager.cc:3898] Updating metadata on disk
1025 07:52:45.086126 (+ 28298us) catalog_manager.cc:3929] Committing alterations to in-memory state
1025 07:52:45.595902 (+509776us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:45.063485 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:45.063571 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:45.063575 (+ 4us) write_op.cc:435] Acquired schema lock
1025 07:52:45.063577 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:45.063681 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:45.063689 (+ 8us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:45.063694 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:45.063755 (+ 61us) tablet.cc:699] Row locks acquired
1025 07:52:45.063757 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:45.063787 (+ 30us) write_op.cc:270] Start()
1025 07:52:45.063810 (+ 23us) write_op.cc:276] Timestamp: P: 1761378765063775 usec, L: 0
1025 07:52:45.063813 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:45.063936 (+ 123us) log.cc:844] Serialized 405 byte log entry
1025 07:52:45.085273 (+ 21337us) op_driver.cc:464] REPLICATION: finished
1025 07:52:45.085695 (+ 422us) write_op.cc:301] APPLY: starting
1025 07:52:45.085712 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:45.085754 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:45.085777 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:45.085831 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:45.085878 (+ 47us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:45.085886 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:45.085969 (+ 83us) log.cc:844] Serialized 23 byte log entry
1025 07:52:45.086028 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.086044 (+ 16us) write_op.cc:454] Released schema lock
1025 07:52:45.086050 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:45.086121 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.086125 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":361,"apply.run_cpu_time_us":423,"apply.run_wall_time_us":423,"num_ops":1,"prepare.queue_time_us":4682,"prepare.run_cpu_time_us":523,"prepare.run_wall_time_us":1965,"replication_time_us":21433}]]}
I20251025 07:52:45.596860 451 master_service.cc:954] ReplaceTablet: received request to replace tablet ebf3a2a926754305b5bd95fcebb25d29 from {username='slave'} at 127.0.0.1:47610
W20251025 07:52:45.597170 442 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 5fc092c2bb2d41ac8c5b4fc07460c8ec (table default.Table_cd20e202397645f8821dcde13b5c0ce3 [id=630c274339664b9db18ab4ef7c29c16f], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5fc092c2bb2d41ac8c5b4fc07460c8ec consensus state has no leader
I20251025 07:52:45.598388 450 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_7eb8424d5192485da1a2d67bfdb2bbbf" } modify_external_catalogs: true
I20251025 07:52:45.598569 450 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_7eb8424d5192485da1a2d67bfdb2bbbf" } modify_external_catalogs: true
I20251025 07:52:45.596403 983 leader_election.cc:304] T f661d5909657455581dcb5e6eadd3c0b P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:45.602090 2597 raft_consensus.cc:2749] T f661d5909657455581dcb5e6eadd3c0b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:45.610595 2146 raft_consensus.cc:2243] T 4b28215f61574cf48c447dd01955ac77 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:45.611608 2675 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: loading cluster ID for follower catalog manager: success
I20251025 07:52:45.611963 2146 raft_consensus.cc:2272] T 4b28215f61574cf48c447dd01955ac77 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:45.616477 1166 tablet_bootstrap.cc:492] T 143aa3fce9ba4850b000a94054e5dc3a P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:45.618005 1166 tablet_bootstrap.cc:654] T 143aa3fce9ba4850b000a94054e5dc3a P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:45.627364 447 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet 0551e5d694894e8fbeb66b5e0c1cd75b
I20251025 07:52:45.627911 447 catalog_manager.cc:6719] ReplaceTablet: tablet 0551e5d694894e8fbeb66b5e0c1cd75b deleted and replaced by tablet ae835c2078624c42a57ca63b7951dc79
I20251025 07:52:45.628031 447 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:47610 (request call id 96) took 1468 ms. Trace:
I20251025 07:52:45.628134 447 rpcz_store.cc:276] 1025 07:52:44.159313 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.531305 (+1371992us) service_pool.cc:225] Handling call
1025 07:52:45.628003 (+ 96698us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:45.619964 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:45.620088 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:45.620095 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:45.620097 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:45.620214 (+ 117us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:45.620228 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:45.620233 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:45.620322 (+ 89us) tablet.cc:699] Row locks acquired
1025 07:52:45.620324 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:45.620358 (+ 34us) write_op.cc:270] Start()
1025 07:52:45.620392 (+ 34us) write_op.cc:276] Timestamp: P: 1761378765617834 usec, L: 0
1025 07:52:45.620396 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:45.620582 (+ 186us) log.cc:844] Serialized 656 byte log entry
1025 07:52:45.625933 (+ 5351us) op_driver.cc:464] REPLICATION: finished
1025 07:52:45.626040 (+ 107us) write_op.cc:301] APPLY: starting
1025 07:52:45.626056 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:45.626103 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:45.626151 (+ 48us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:45.626278 (+ 127us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:45.626336 (+ 58us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:45.626343 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:45.626414 (+ 71us) log.cc:844] Serialized 29 byte log entry
1025 07:52:45.626472 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.626492 (+ 20us) write_op.cc:454] Released schema lock
1025 07:52:45.626498 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:45.626567 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.626571 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":58,"apply.run_cpu_time_us":517,"apply.run_wall_time_us":517,"num_ops":2,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":759,"prepare.run_wall_time_us":759,"raft.queue_time_us":485,"raft.run_cpu_time_us":394,"raft.run_wall_time_us":393,"replication_time_us":5507}]]}
I20251025 07:52:45.628907 447 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_6c5a24747ba14352b0285a41ec2985a4" } modify_external_catalogs: true
I20251025 07:52:45.629081 447 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_6c5a24747ba14352b0285a41ec2985a4" } modify_external_catalogs: true
I20251025 07:52:45.629776 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 0551e5d694894e8fbeb66b5e0c1cd75b with delete_type TABLET_DATA_DELETED (replaced by tablet ae835c2078624c42a57ca63b7951dc79) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:45.629884 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 0551e5d694894e8fbeb66b5e0c1cd75b with delete_type TABLET_DATA_DELETED (replaced by tablet ae835c2078624c42a57ca63b7951dc79) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:45.630893 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 0551e5d694894e8fbeb66b5e0c1cd75b with delete_type TABLET_DATA_DELETED (replaced by tablet ae835c2078624c42a57ca63b7951dc79) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:45.631572 2686 consensus_queue.cc:1048] T ab42909afc524f2fa3c1041495d13a59 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:45.632979 2710 consensus_queue.cc:1048] T 30418858bf95438686bdfe64ac733144 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:45.642091 2663 raft_consensus.cc:493] T 682c1e22d7764dce862d970b33b88cb3 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:45.642272 2663 raft_consensus.cc:515] T 682c1e22d7764dce862d970b33b88cb3 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.642422 2117 raft_consensus.cc:493] T dda0b26e4f1a44928bc4b84f333901c0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:45.642589 2117 raft_consensus.cc:515] T dda0b26e4f1a44928bc4b84f333901c0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.642784 822 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
I20251025 07:52:45.643014 2663 leader_election.cc:290] T 682c1e22d7764dce862d970b33b88cb3 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:45.643054 2117 leader_election.cc:290] T dda0b26e4f1a44928bc4b84f333901c0 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:45.644452 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dda0b26e4f1a44928bc4b84f333901c0" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:45.645112 712 leader_election.cc:343] T dda0b26e4f1a44928bc4b84f333901c0 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:45.645430 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "682c1e22d7764dce862d970b33b88cb3" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:45.646301 907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dda0b26e4f1a44928bc4b84f333901c0" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:45.646561 712 leader_election.cc:343] T 682c1e22d7764dce862d970b33b88cb3 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:45.646304 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "682c1e22d7764dce862d970b33b88cb3" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:45.647143 710 leader_election.cc:343] T dda0b26e4f1a44928bc4b84f333901c0 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:45.647264 710 leader_election.cc:304] T dda0b26e4f1a44928bc4b84f333901c0 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:45.647653 710 leader_election.cc:343] T 682c1e22d7764dce862d970b33b88cb3 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:45.647742 710 leader_election.cc:304] T 682c1e22d7764dce862d970b33b88cb3 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:45.647794 2117 raft_consensus.cc:2749] T dda0b26e4f1a44928bc4b84f333901c0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:45.647938 2117 raft_consensus.cc:2749] T 682c1e22d7764dce862d970b33b88cb3 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:45.648911 447 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet 110a8e9e4c6c434284c0ae14dd6987f7
I20251025 07:52:45.649621 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 110a8e9e4c6c434284c0ae14dd6987f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:45 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:45.649904 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 110a8e9e4c6c434284c0ae14dd6987f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:45 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:45.650283 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 110a8e9e4c6c434284c0ae14dd6987f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:45 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:45.651274 447 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_73279bdee234454e9def34441da9579a" } modify_external_catalogs: true
I20251025 07:52:45.651458 447 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_73279bdee234454e9def34441da9579a" } modify_external_catalogs: true
I20251025 07:52:45.652938 2687 consensus_queue.cc:1048] T f656945454fe4921ab819c772a089eb3 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:45.655330 1169 tablet_bootstrap.cc:492] T 11bfe0c87ca941a3846462c8a4512d83 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:45.655474 1169 ts_tablet_manager.cc:1403] T 11bfe0c87ca941a3846462c8a4512d83 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.083s user 0.002s sys 0.003s
I20251025 07:52:45.664661 1048 raft_consensus.cc:3060] T f661d5909657455581dcb5e6eadd3c0b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:45.665063 449 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet 623dec4eab1d4666baf2334a01eeb36d
I20251025 07:52:45.665104 2169 ts_tablet_manager.cc:1929] T 6ec421eb820b4f40afcff5a5bf80c508 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:45.665199 2169 log.cc:1199] T 6ec421eb820b4f40afcff5a5bf80c508 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/6ec421eb820b4f40afcff5a5bf80c508
I20251025 07:52:45.665584 2169 ts_tablet_manager.cc:1950] T 6ec421eb820b4f40afcff5a5bf80c508 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:45.665832 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 623dec4eab1d4666baf2334a01eeb36d with delete_type TABLET_DATA_DELETED (replaced by tablet b2ac640e69794a09811bd93aa02a22ea) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:45.665901 449 catalog_manager.cc:6719] ReplaceTablet: tablet 623dec4eab1d4666baf2334a01eeb36d deleted and replaced by tablet b2ac640e69794a09811bd93aa02a22ea
I20251025 07:52:45.665956 772 raft_consensus.cc:2468] T 5b6383c99e9e4c90a374da9ccd830a6f P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:45.666018 449 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:47610 (request call id 97) took 1371 ms. Trace:
I20251025 07:52:45.666445 2166 ts_tablet_manager.cc:1929] T 74e2d78c024849b0af5c0bde1f48c36f P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:45.666543 2166 log.cc:1199] T 74e2d78c024849b0af5c0bde1f48c36f P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/74e2d78c024849b0af5c0bde1f48c36f
I20251025 07:52:45.666896 2166 ts_tablet_manager.cc:1950] T 74e2d78c024849b0af5c0bde1f48c36f P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:45.666110 449 rpcz_store.cc:276] 1025 07:52:44.294372 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.537553 (+1243181us) service_pool.cc:225] Handling call
1025 07:52:45.665987 (+128434us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:45.657952 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:45.658102 (+ 150us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:45.658116 (+ 14us) write_op.cc:435] Acquired schema lock
1025 07:52:45.658121 (+ 5us) tablet.cc:660] Decoding operations
1025 07:52:45.658281 (+ 160us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:45.658299 (+ 18us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:45.658307 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:45.658403 (+ 96us) tablet.cc:699] Row locks acquired
1025 07:52:45.658408 (+ 5us) write_op.cc:260] PREPARE: finished
1025 07:52:45.658443 (+ 35us) write_op.cc:270] Start()
1025 07:52:45.658487 (+ 44us) write_op.cc:276] Timestamp: P: 1761378765655918 usec, L: 0
1025 07:52:45.658492 (+ 5us) op_driver.cc:348] REPLICATION: starting
1025 07:52:45.658694 (+ 202us) log.cc:844] Serialized 622 byte log entry
1025 07:52:45.663396 (+ 4702us) op_driver.cc:464] REPLICATION: finished
1025 07:52:45.663510 (+ 114us) write_op.cc:301] APPLY: starting
1025 07:52:45.663528 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:45.663610 (+ 82us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:45.663654 (+ 44us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:45.663774 (+ 120us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:45.663851 (+ 77us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:45.663860 (+ 9us) write_op.cc:312] APPLY: finished
1025 07:52:45.663945 (+ 85us) log.cc:844] Serialized 29 byte log entry
1025 07:52:45.664033 (+ 88us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.664057 (+ 24us) write_op.cc:454] Released schema lock
1025 07:52:45.664067 (+ 10us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:45.664150 (+ 83us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.664157 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":61,"apply.run_cpu_time_us":626,"apply.run_wall_time_us":625,"num_ops":2,"prepare.queue_time_us":50,"prepare.run_cpu_time_us":875,"prepare.run_wall_time_us":876,"raft.queue_time_us":1184,"raft.run_cpu_time_us":549,"raft.run_wall_time_us":548,"replication_time_us":4869}]]}
I20251025 07:52:45.667457 847 leader_election.cc:304] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:45.667629 449 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_d5f74e259f754aa49c24a74e75cdd23c" } modify_external_catalogs: true
I20251025 07:52:45.667805 449 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_d5f74e259f754aa49c24a74e75cdd23c" } modify_external_catalogs: true
I20251025 07:52:45.667894 2710 raft_consensus.cc:2804] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:45.668017 2710 raft_consensus.cc:697] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:45.668246 2710 consensus_queue.cc:237] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.668586 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 623dec4eab1d4666baf2334a01eeb36d with delete_type TABLET_DATA_DELETED (replaced by tablet b2ac640e69794a09811bd93aa02a22ea) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:45.669603 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 623dec4eab1d4666baf2334a01eeb36d with delete_type TABLET_DATA_DELETED (replaced by tablet b2ac640e69794a09811bd93aa02a22ea) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:45.677433 1169 raft_consensus.cc:359] T 11bfe0c87ca941a3846462c8a4512d83 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.677655 1169 raft_consensus.cc:385] T 11bfe0c87ca941a3846462c8a4512d83 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:45.677718 1169 raft_consensus.cc:740] T 11bfe0c87ca941a3846462c8a4512d83 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:45.677901 1169 consensus_queue.cc:260] T 11bfe0c87ca941a3846462c8a4512d83 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.681326 1095 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
I20251025 07:52:45.685973 2597 raft_consensus.cc:493] T 36c8fcafd5464b4e999276a8062a812a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:45.686144 2597 raft_consensus.cc:515] T 36c8fcafd5464b4e999276a8062a812a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.686813 2597 leader_election.cc:290] T 36c8fcafd5464b4e999276a8062a812a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:45.687533 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36c8fcafd5464b4e999276a8062a812a" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:45.688776 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36c8fcafd5464b4e999276a8062a812a" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:45.689344 982 leader_election.cc:304] T 36c8fcafd5464b4e999276a8062a812a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:45.690044 2671 raft_consensus.cc:2749] T 36c8fcafd5464b4e999276a8062a812a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:45.691977 1169 ts_tablet_manager.cc:1434] T 11bfe0c87ca941a3846462c8a4512d83 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.036s user 0.001s sys 0.001s
I20251025 07:52:45.693064 1169 tablet_bootstrap.cc:492] T ce1ebb1ec14b4801930a72765bca1a7d P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:45.694689 2675 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: acquiring CA information for follower catalog manager: success
I20251025 07:52:45.695155 1169 tablet_bootstrap.cc:654] T ce1ebb1ec14b4801930a72765bca1a7d P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:45.696749 2146 ts_tablet_manager.cc:1916] T 4b28215f61574cf48c447dd01955ac77 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:45.700074 1167 tablet_bootstrap.cc:492] T 1068f566477e4c6db7dde566fff4c2fb P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:45.700215 1167 ts_tablet_manager.cc:1403] T 1068f566477e4c6db7dde566fff4c2fb P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.119s user 0.003s sys 0.005s
I20251025 07:52:45.700959 1167 raft_consensus.cc:359] T 1068f566477e4c6db7dde566fff4c2fb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:45.701118 1167 raft_consensus.cc:385] T 1068f566477e4c6db7dde566fff4c2fb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:45.701171 1167 raft_consensus.cc:740] T 1068f566477e4c6db7dde566fff4c2fb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:45.701375 1167 consensus_queue.cc:260] T 1068f566477e4c6db7dde566fff4c2fb P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:45.701939 1167 ts_tablet_manager.cc:1434] T 1068f566477e4c6db7dde566fff4c2fb P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:45.702422 1046 raft_consensus.cc:3060] T f656945454fe4921ab819c772a089eb3 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:45.702620 1167 tablet_bootstrap.cc:492] T 2b594d4718344aafbdaec34040695589 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:45.704167 454 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet cb829649f73d4d3f8862d268c953bd86
I20251025 07:52:45.704761 454 catalog_manager.cc:6719] ReplaceTablet: tablet cb829649f73d4d3f8862d268c953bd86 deleted and replaced by tablet e330e1b53bed4e7cace1c4fb7fa6b565
I20251025 07:52:45.704914 454 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:47610 (request call id 98) took 1262 ms. Trace:
I20251025 07:52:45.705026 454 rpcz_store.cc:276] 1025 07:52:44.442588 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.538910 (+1096322us) service_pool.cc:225] Handling call
1025 07:52:45.704858 (+165948us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:45.690651 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:45.690771 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:45.690778 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:45.690780 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:45.690900 (+ 120us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:45.690917 (+ 17us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:45.690925 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:45.691017 (+ 92us) tablet.cc:699] Row locks acquired
1025 07:52:45.691020 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:45.691057 (+ 37us) write_op.cc:270] Start()
1025 07:52:45.691094 (+ 37us) write_op.cc:276] Timestamp: P: 1761378765688532 usec, L: 0
1025 07:52:45.691097 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:45.691274 (+ 177us) log.cc:844] Serialized 656 byte log entry
1025 07:52:45.703110 (+ 11836us) op_driver.cc:464] REPLICATION: finished
1025 07:52:45.703207 (+ 97us) write_op.cc:301] APPLY: starting
1025 07:52:45.703222 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:45.703285 (+ 63us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:45.703308 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:45.703413 (+ 105us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:45.703476 (+ 63us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:45.703484 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:45.703554 (+ 70us) log.cc:844] Serialized 29 byte log entry
1025 07:52:45.703645 (+ 91us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.703662 (+ 17us) write_op.cc:454] Released schema lock
1025 07:52:45.703668 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:45.703744 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.703747 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":54,"apply.run_cpu_time_us":523,"apply.run_wall_time_us":521,"num_ops":2,"prepare.queue_time_us":113,"prepare.run_cpu_time_us":727,"prepare.run_wall_time_us":727,"raft.queue_time_us":1087,"raft.run_cpu_time_us":568,"raft.run_wall_time_us":567,"replication_time_us":11988}]]}
I20251025 07:52:45.707442 960 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
I20251025 07:52:45.710780 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet cb829649f73d4d3f8862d268c953bd86 with delete_type TABLET_DATA_DELETED (replaced by tablet e330e1b53bed4e7cace1c4fb7fa6b565) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:45.710963 2687 raft_consensus.cc:493] T 627a9cfd398346f2aa34c8932a11b5a8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:45.711009 878 tablet_service.cc:1552] Processing DeleteTablet for tablet cb829649f73d4d3f8862d268c953bd86 with delete_type TABLET_DATA_DELETED (replaced by tablet e330e1b53bed4e7cace1c4fb7fa6b565) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:45.711083 2687 raft_consensus.cc:515] T 627a9cfd398346f2aa34c8932a11b5a8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.711668 2687 leader_election.cc:290] T 627a9cfd398346f2aa34c8932a11b5a8 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:45.712008 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627a9cfd398346f2aa34c8932a11b5a8" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:45.712423 748 tablet_service.cc:1552] Processing DeleteTablet for tablet cb829649f73d4d3f8862d268c953bd86 with delete_type TABLET_DATA_DELETED (replaced by tablet e330e1b53bed4e7cace1c4fb7fa6b565) from {username='slave'} at 127.0.0.1:58412
W20251025 07:52:45.712479 850 leader_election.cc:343] T 627a9cfd398346f2aa34c8932a11b5a8 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:45.706091 454 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_91898f2a551144d18df0309c827855bf" } modify_external_catalogs: true
I20251025 07:52:45.712785 454 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_91898f2a551144d18df0309c827855bf" } modify_external_catalogs: true
I20251025 07:52:45.712863 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627a9cfd398346f2aa34c8932a11b5a8" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:45.713145 776 raft_consensus.cc:2468] T 627a9cfd398346f2aa34c8932a11b5a8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:45.713539 847 leader_election.cc:304] T 627a9cfd398346f2aa34c8932a11b5a8 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:45.713912 2687 raft_consensus.cc:2804] T 627a9cfd398346f2aa34c8932a11b5a8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:45.713995 2687 raft_consensus.cc:493] T 627a9cfd398346f2aa34c8932a11b5a8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:45.714071 2687 raft_consensus.cc:3060] T 627a9cfd398346f2aa34c8932a11b5a8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:45.717738 1167 tablet_bootstrap.cc:654] T 2b594d4718344aafbdaec34040695589 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:45.721340 451 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet ebf3a2a926754305b5bd95fcebb25d29
I20251025 07:52:45.721798 451 catalog_manager.cc:6719] ReplaceTablet: tablet ebf3a2a926754305b5bd95fcebb25d29 deleted and replaced by tablet 8f05894c45154cd0b65f5b9bcef8cd4c
I20251025 07:52:45.721928 451 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:47610 (request call id 100) took 1067 ms. Trace:
I20251025 07:52:45.722021 451 rpcz_store.cc:276] 1025 07:52:44.654958 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.596806 (+941848us) service_pool.cc:225] Handling call
1025 07:52:45.721882 (+125076us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:45.694370 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:45.694486 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:45.694494 (+ 8us) write_op.cc:435] Acquired schema lock
1025 07:52:45.694499 (+ 5us) tablet.cc:660] Decoding operations
1025 07:52:45.694614 (+ 115us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:45.694625 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:45.694629 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:45.694697 (+ 68us) tablet.cc:699] Row locks acquired
1025 07:52:45.694700 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:45.694729 (+ 29us) write_op.cc:270] Start()
1025 07:52:45.694759 (+ 30us) write_op.cc:276] Timestamp: P: 1761378765692206 usec, L: 0
1025 07:52:45.694762 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:45.694920 (+ 158us) log.cc:844] Serialized 656 byte log entry
1025 07:52:45.720025 (+ 25105us) op_driver.cc:464] REPLICATION: finished
1025 07:52:45.720271 (+ 246us) write_op.cc:301] APPLY: starting
1025 07:52:45.720287 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:45.720328 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:45.720359 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:45.720477 (+ 118us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:45.720524 (+ 47us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:45.720532 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:45.720597 (+ 65us) log.cc:844] Serialized 29 byte log entry
1025 07:52:45.720662 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.720680 (+ 18us) write_op.cc:454] Released schema lock
1025 07:52:45.720686 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:45.720745 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.720748 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":198,"apply.run_cpu_time_us":467,"apply.run_wall_time_us":467,"num_ops":2,"prepare.queue_time_us":49,"prepare.run_cpu_time_us":621,"prepare.run_wall_time_us":621,"replication_time_us":25230}]]}
I20251025 07:52:45.723232 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet ebf3a2a926754305b5bd95fcebb25d29 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f05894c45154cd0b65f5b9bcef8cd4c) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:45.723523 878 tablet_service.cc:1552] Processing DeleteTablet for tablet ebf3a2a926754305b5bd95fcebb25d29 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f05894c45154cd0b65f5b9bcef8cd4c) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:45.724383 748 tablet_service.cc:1552] Processing DeleteTablet for tablet ebf3a2a926754305b5bd95fcebb25d29 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f05894c45154cd0b65f5b9bcef8cd4c) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:45.725152 455 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet ef1854d35a3047e3a9fbe0166ab1295a
I20251025 07:52:45.725697 455 catalog_manager.cc:6719] ReplaceTablet: tablet ef1854d35a3047e3a9fbe0166ab1295a deleted and replaced by tablet 3760348dbe284c41a01a4fe479a2103f
I20251025 07:52:45.725829 455 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:47610 (request call id 99) took 1252 ms. Trace:
I20251025 07:52:45.725914 455 rpcz_store.cc:276] 1025 07:52:44.473650 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.540595 (+1066945us) service_pool.cc:225] Handling call
1025 07:52:45.725775 (+185180us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:45.696232 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:45.696360 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:45.696368 (+ 8us) write_op.cc:435] Acquired schema lock
1025 07:52:45.696372 (+ 4us) tablet.cc:660] Decoding operations
1025 07:52:45.696491 (+ 119us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:45.696504 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:45.696511 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:45.696628 (+ 117us) tablet.cc:699] Row locks acquired
1025 07:52:45.696633 (+ 5us) write_op.cc:260] PREPARE: finished
1025 07:52:45.696667 (+ 34us) write_op.cc:270] Start()
1025 07:52:45.696701 (+ 34us) write_op.cc:276] Timestamp: P: 1761378765694142 usec, L: 0
1025 07:52:45.696706 (+ 5us) op_driver.cc:348] REPLICATION: starting
1025 07:52:45.696854 (+ 148us) log.cc:844] Serialized 656 byte log entry
1025 07:52:45.720093 (+ 23239us) op_driver.cc:464] REPLICATION: finished
1025 07:52:45.720809 (+ 716us) write_op.cc:301] APPLY: starting
1025 07:52:45.720817 (+ 8us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:45.720846 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:45.720872 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:45.720946 (+ 74us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:45.720985 (+ 39us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:45.720992 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:45.721039 (+ 47us) log.cc:844] Serialized 29 byte log entry
1025 07:52:45.721073 (+ 34us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.721086 (+ 13us) write_op.cc:454] Released schema lock
1025 07:52:45.721091 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":684,"num_ops":2,"prepare.queue_time_us":37,"prepare.run_cpu_time_us":682,"prepare.run_wall_time_us":681,"replication_time_us":23366,"threads_started":1}]]}
I20251025 07:52:45.727473 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet ef1854d35a3047e3a9fbe0166ab1295a with delete_type TABLET_DATA_DELETED (replaced by tablet 3760348dbe284c41a01a4fe479a2103f) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:45.727710 878 tablet_service.cc:1552] Processing DeleteTablet for tablet ef1854d35a3047e3a9fbe0166ab1295a with delete_type TABLET_DATA_DELETED (replaced by tablet 3760348dbe284c41a01a4fe479a2103f) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:45.728188 748 tablet_service.cc:1552] Processing DeleteTablet for tablet ef1854d35a3047e3a9fbe0166ab1295a with delete_type TABLET_DATA_DELETED (replaced by tablet 3760348dbe284c41a01a4fe479a2103f) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:45.736176 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 133) took 6910 ms. Trace:
I20251025 07:52:45.736284 2169 rpcz_store.cc:276] 1025 07:52:38.825645 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.834438 (+ 8793us) service_pool.cc:225] Handling call
1025 07:52:45.470413 (+6635975us) ts_tablet_manager.cc:1151] Deleting tablet 6ec421eb820b4f40afcff5a5bf80c508
1025 07:52:45.470425 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:45.564012 (+ 93587us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:45.664967 (+100955us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:45.736154 (+ 71187us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6633953}
I20251025 07:52:45.736583 2169 tablet_replica.cc:333] T 3abfd52327a04ba88625f391a8658fbb P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:45.736752 2169 raft_consensus.cc:2243] T 3abfd52327a04ba88625f391a8658fbb P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:45.737118 2169 raft_consensus.cc:2272] T 3abfd52327a04ba88625f391a8658fbb P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:45.737654 2169 ts_tablet_manager.cc:1916] T 3abfd52327a04ba88625f391a8658fbb P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:45.738341 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 119) took 7059 ms. Trace:
I20251025 07:52:45.738435 2166 rpcz_store.cc:276] 1025 07:52:38.679209 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.679253 (+ 44us) service_pool.cc:225] Handling call
1025 07:52:45.431656 (+6752403us) ts_tablet_manager.cc:1151] Deleting tablet 74e2d78c024849b0af5c0bde1f48c36f
1025 07:52:45.431663 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:45.566359 (+134696us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:45.666341 (+ 99982us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:45.738320 (+ 71979us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6752182}
I20251025 07:52:45.738812 2166 tablet_replica.cc:333] T 5fca357d7e24494fb6af4bd73d98833c P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:45.739035 2166 raft_consensus.cc:2243] T 5fca357d7e24494fb6af4bd73d98833c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:45.739205 2166 raft_consensus.cc:2272] T 5fca357d7e24494fb6af4bd73d98833c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:45.739864 2166 ts_tablet_manager.cc:1916] T 5fca357d7e24494fb6af4bd73d98833c P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:45.743109 1167 tablet_bootstrap.cc:492] T 2b594d4718344aafbdaec34040695589 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:45.743139 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 6ec421eb820b4f40afcff5a5bf80c508 (table default.Table_5487882c2d0c4af4ad17df240fa90277 [id=d73352841fda408baf49087af00b4250]) successfully deleted
I20251025 07:52:45.743238 1167 ts_tablet_manager.cc:1403] T 2b594d4718344aafbdaec34040695589 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.041s user 0.001s sys 0.002s
I20251025 07:52:45.747999 1167 raft_consensus.cc:359] T 2b594d4718344aafbdaec34040695589 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.748215 1167 raft_consensus.cc:385] T 2b594d4718344aafbdaec34040695589 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:45.748277 1167 raft_consensus.cc:740] T 2b594d4718344aafbdaec34040695589 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:45.748575 1167 consensus_queue.cc:260] T 2b594d4718344aafbdaec34040695589 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.749136 1167 ts_tablet_manager.cc:1434] T 2b594d4718344aafbdaec34040695589 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251025 07:52:45.749434 1167 tablet_bootstrap.cc:492] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:45.749976 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 74e2d78c024849b0af5c0bde1f48c36f (table default.Table_434ef1ade0a14af987864563993afae8 [id=3a3af35e11944a6f97fc0144cb9d08d4]) successfully deleted
I20251025 07:52:45.751135 1167 tablet_bootstrap.cc:654] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:45.756752 2687 raft_consensus.cc:515] T 627a9cfd398346f2aa34c8932a11b5a8 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.758163 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627a9cfd398346f2aa34c8932a11b5a8" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:45.758594 771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627a9cfd398346f2aa34c8932a11b5a8" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:45.758759 771 raft_consensus.cc:3060] T 627a9cfd398346f2aa34c8932a11b5a8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:45.763396 850 leader_election.cc:343] T 627a9cfd398346f2aa34c8932a11b5a8 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:45.766522 2687 leader_election.cc:290] T 627a9cfd398346f2aa34c8932a11b5a8 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:45.775262 1166 tablet_bootstrap.cc:492] T 143aa3fce9ba4850b000a94054e5dc3a P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:45.775424 1166 ts_tablet_manager.cc:1403] T 143aa3fce9ba4850b000a94054e5dc3a P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.159s user 0.000s sys 0.002s
I20251025 07:52:45.776188 1166 raft_consensus.cc:359] T 143aa3fce9ba4850b000a94054e5dc3a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.776365 1166 raft_consensus.cc:385] T 143aa3fce9ba4850b000a94054e5dc3a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:45.776427 1166 raft_consensus.cc:740] T 143aa3fce9ba4850b000a94054e5dc3a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:45.776687 1166 consensus_queue.cc:260] T 143aa3fce9ba4850b000a94054e5dc3a P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.777172 1166 ts_tablet_manager.cc:1434] T 143aa3fce9ba4850b000a94054e5dc3a P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251025 07:52:45.777448 1166 tablet_bootstrap.cc:492] T dda0b26e4f1a44928bc4b84f333901c0 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:45.779088 1166 tablet_bootstrap.cc:654] T dda0b26e4f1a44928bc4b84f333901c0 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:45.785611 1166 tablet_bootstrap.cc:492] T dda0b26e4f1a44928bc4b84f333901c0 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:45.785763 1166 ts_tablet_manager.cc:1403] T dda0b26e4f1a44928bc4b84f333901c0 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.002s
I20251025 07:52:45.791556 1166 raft_consensus.cc:359] T dda0b26e4f1a44928bc4b84f333901c0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.791769 1166 raft_consensus.cc:385] T dda0b26e4f1a44928bc4b84f333901c0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:45.791822 1166 raft_consensus.cc:740] T dda0b26e4f1a44928bc4b84f333901c0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:45.792050 1166 consensus_queue.cc:260] T dda0b26e4f1a44928bc4b84f333901c0 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.793958 1166 ts_tablet_manager.cc:1434] T dda0b26e4f1a44928bc4b84f333901c0 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251025 07:52:45.797792 1166 tablet_bootstrap.cc:492] T 7e6544202d4f489d82f22d6073ef240d P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:45.799456 1166 tablet_bootstrap.cc:654] T 7e6544202d4f489d82f22d6073ef240d P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:45.803509 1167 tablet_bootstrap.cc:492] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:45.803704 1167 ts_tablet_manager.cc:1403] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.054s user 0.005s sys 0.005s
I20251025 07:52:45.804622 1167 raft_consensus.cc:359] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.804744 1166 tablet_bootstrap.cc:492] T 7e6544202d4f489d82f22d6073ef240d P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:45.804809 1167 raft_consensus.cc:385] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:45.804873 1166 ts_tablet_manager.cc:1403] T 7e6544202d4f489d82f22d6073ef240d P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20251025 07:52:45.804877 1167 raft_consensus.cc:740] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:45.805106 1167 consensus_queue.cc:260] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.805617 1167 ts_tablet_manager.cc:1434] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 07:52:45.805563 1166 raft_consensus.cc:359] T 7e6544202d4f489d82f22d6073ef240d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.805752 1166 raft_consensus.cc:385] T 7e6544202d4f489d82f22d6073ef240d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:45.805812 1166 raft_consensus.cc:740] T 7e6544202d4f489d82f22d6073ef240d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:45.805915 1167 tablet_bootstrap.cc:492] T c501ef5d70e742598c1a902efe47feb1 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:45.807672 1167 tablet_bootstrap.cc:654] T c501ef5d70e742598c1a902efe47feb1 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:45.812237 2671 raft_consensus.cc:493] T 4678f7eab5e8497db308e8a38e387618 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:45.812397 2671 raft_consensus.cc:515] T 4678f7eab5e8497db308e8a38e387618 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.813033 2671 leader_election.cc:290] T 4678f7eab5e8497db308e8a38e387618 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:45.814074 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4678f7eab5e8497db308e8a38e387618" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:45.814074 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4678f7eab5e8497db308e8a38e387618" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:45.814342 776 raft_consensus.cc:2393] T 4678f7eab5e8497db308e8a38e387618 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:45.816468 983 leader_election.cc:304] T 4678f7eab5e8497db308e8a38e387618 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:45.817718 2671 raft_consensus.cc:3060] T 4678f7eab5e8497db308e8a38e387618 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:45.818261 1166 consensus_queue.cc:260] T 7e6544202d4f489d82f22d6073ef240d P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.818928 1166 ts_tablet_manager.cc:1434] T 7e6544202d4f489d82f22d6073ef240d P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20251025 07:52:45.820827 1166 tablet_bootstrap.cc:492] T 6d3c9954693d4e8782ee0128ef066aa9 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:45.825575 1166 tablet_bootstrap.cc:654] T 6d3c9954693d4e8782ee0128ef066aa9 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:45.828320 2675 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 8603bddd8a084ff7a73a25641768dfa9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251025 07:52:45.832674 771 raft_consensus.cc:2468] T 627a9cfd398346f2aa34c8932a11b5a8 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:45.833262 2146 ts_tablet_manager.cc:1929] T 4b28215f61574cf48c447dd01955ac77 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:45.833375 2146 log.cc:1199] T 4b28215f61574cf48c447dd01955ac77 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/4b28215f61574cf48c447dd01955ac77
I20251025 07:52:45.833699 2146 ts_tablet_manager.cc:1950] T 4b28215f61574cf48c447dd01955ac77 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:45.834308 2687 consensus_queue.cc:1048] T f656945454fe4921ab819c772a089eb3 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:45.835232 1167 tablet_bootstrap.cc:492] T c501ef5d70e742598c1a902efe47feb1 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:45.835371 1167 ts_tablet_manager.cc:1403] T c501ef5d70e742598c1a902efe47feb1 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20251025 07:52:45.835460 2615 raft_consensus.cc:493] T c0c6324cbae34dcebade3ef9f89a7150 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:45.835589 2615 raft_consensus.cc:515] T c0c6324cbae34dcebade3ef9f89a7150 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.836166 2615 leader_election.cc:290] T c0c6324cbae34dcebade3ef9f89a7150 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:45.836284 2615 raft_consensus.cc:493] T 143aa3fce9ba4850b000a94054e5dc3a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:45.836231 1167 raft_consensus.cc:359] T c501ef5d70e742598c1a902efe47feb1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.836377 1167 raft_consensus.cc:385] T c501ef5d70e742598c1a902efe47feb1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:45.836424 1167 raft_consensus.cc:740] T c501ef5d70e742598c1a902efe47feb1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:45.836374 847 leader_election.cc:304] T 627a9cfd398346f2aa34c8932a11b5a8 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:45.836652 1167 consensus_queue.cc:260] T c501ef5d70e742598c1a902efe47feb1 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.837167 2709 raft_consensus.cc:2804] T 627a9cfd398346f2aa34c8932a11b5a8 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:45.837271 2709 raft_consensus.cc:697] T 627a9cfd398346f2aa34c8932a11b5a8 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:45.837481 2709 consensus_queue.cc:237] T 627a9cfd398346f2aa34c8932a11b5a8 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.837219 1167 ts_tablet_manager.cc:1434] T c501ef5d70e742598c1a902efe47feb1 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 07:52:45.840204 1167 tablet_bootstrap.cc:492] T fa826b85f0c94150a17d4e417dd3d62f P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:45.840240 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0c6324cbae34dcebade3ef9f89a7150" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:45.841313 850 leader_election.cc:343] T c0c6324cbae34dcebade3ef9f89a7150 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:45.841935 1167 tablet_bootstrap.cc:654] T fa826b85f0c94150a17d4e417dd3d62f P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:45.842514 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0c6324cbae34dcebade3ef9f89a7150" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:45.842696 768 raft_consensus.cc:2468] T c0c6324cbae34dcebade3ef9f89a7150 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:45.843106 847 leader_election.cc:304] T c0c6324cbae34dcebade3ef9f89a7150 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:45.843523 2687 raft_consensus.cc:2804] T c0c6324cbae34dcebade3ef9f89a7150 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:45.843658 2687 raft_consensus.cc:493] T c0c6324cbae34dcebade3ef9f89a7150 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:45.843726 2687 raft_consensus.cc:3060] T c0c6324cbae34dcebade3ef9f89a7150 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:45.836375 2615 raft_consensus.cc:515] T 143aa3fce9ba4850b000a94054e5dc3a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.844424 2615 leader_election.cc:290] T 143aa3fce9ba4850b000a94054e5dc3a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:45.845433 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143aa3fce9ba4850b000a94054e5dc3a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:45.845578 768 raft_consensus.cc:2468] T 143aa3fce9ba4850b000a94054e5dc3a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:45.845901 847 leader_election.cc:304] T 143aa3fce9ba4850b000a94054e5dc3a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:45.846282 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143aa3fce9ba4850b000a94054e5dc3a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:45.850368 850 leader_election.cc:343] T 143aa3fce9ba4850b000a94054e5dc3a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:45.850661 2615 raft_consensus.cc:2804] T 143aa3fce9ba4850b000a94054e5dc3a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:45.850721 2615 raft_consensus.cc:493] T 143aa3fce9ba4850b000a94054e5dc3a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:45.850772 2615 raft_consensus.cc:3060] T 143aa3fce9ba4850b000a94054e5dc3a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:45.875207 449 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet f69cdaa7ee094283890036c0189e2c81
I20251025 07:52:45.876366 748 tablet_service.cc:1552] Processing DeleteTablet for tablet f69cdaa7ee094283890036c0189e2c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:45 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:45.877665 2420 raft_consensus.cc:493] T 1068f566477e4c6db7dde566fff4c2fb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:45.877816 2420 raft_consensus.cc:515] T 1068f566477e4c6db7dde566fff4c2fb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:45.878464 2420 leader_election.cc:290] T 1068f566477e4c6db7dde566fff4c2fb P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:45.879702 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1068f566477e4c6db7dde566fff4c2fb" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:45.879673 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1068f566477e4c6db7dde566fff4c2fb" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:45.879834 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet f69cdaa7ee094283890036c0189e2c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:45 UTC) from {username='slave'} at 127.0.0.1:44572
W20251025 07:52:45.881045 712 leader_election.cc:343] T 1068f566477e4c6db7dde566fff4c2fb P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:45.881192 710 leader_election.cc:343] T 1068f566477e4c6db7dde566fff4c2fb P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:45.881296 712 leader_election.cc:304] T 1068f566477e4c6db7dde566fff4c2fb P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:45.882292 2698 consensus_queue.cc:1048] T 4678f7eab5e8497db308e8a38e387618 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:45.882606 878 tablet_service.cc:1552] Processing DeleteTablet for tablet f69cdaa7ee094283890036c0189e2c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:45 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:45.883229 2420 raft_consensus.cc:2749] T 1068f566477e4c6db7dde566fff4c2fb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:45.898983 1166 tablet_bootstrap.cc:492] T 6d3c9954693d4e8782ee0128ef066aa9 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:45.899305 1166 ts_tablet_manager.cc:1403] T 6d3c9954693d4e8782ee0128ef066aa9 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.079s user 0.002s sys 0.005s
I20251025 07:52:45.900703 1166 raft_consensus.cc:359] T 6d3c9954693d4e8782ee0128ef066aa9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.901032 1166 raft_consensus.cc:385] T 6d3c9954693d4e8782ee0128ef066aa9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:45.901151 1166 raft_consensus.cc:740] T 6d3c9954693d4e8782ee0128ef066aa9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:45.901500 1166 consensus_queue.cc:260] T 6d3c9954693d4e8782ee0128ef066aa9 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.902400 1166 ts_tablet_manager.cc:1434] T 6d3c9954693d4e8782ee0128ef066aa9 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251025 07:52:45.902391 455 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_e84098955d80499cafb72da95cf65cb1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:45.902760 1166 tablet_bootstrap.cc:492] T 9a8e2658a81e436298fe2190726080d4 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
W20251025 07:52:45.903043 455 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e84098955d80499cafb72da95cf65cb1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:45.905112 451 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_59197c6875cd424f89a3e4a6cba7629f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:45.905577 451 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59197c6875cd424f89a3e4a6cba7629f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:45.911341 1167 tablet_bootstrap.cc:492] T fa826b85f0c94150a17d4e417dd3d62f P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:45.911468 1167 ts_tablet_manager.cc:1403] T fa826b85f0c94150a17d4e417dd3d62f P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.071s user 0.019s sys 0.003s
I20251025 07:52:45.912742 1167 raft_consensus.cc:359] T fa826b85f0c94150a17d4e417dd3d62f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.912923 1167 raft_consensus.cc:385] T fa826b85f0c94150a17d4e417dd3d62f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:45.912982 1167 raft_consensus.cc:740] T fa826b85f0c94150a17d4e417dd3d62f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:45.913193 1167 consensus_queue.cc:260] T fa826b85f0c94150a17d4e417dd3d62f P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.913688 1167 ts_tablet_manager.cc:1434] T fa826b85f0c94150a17d4e417dd3d62f P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:45.914335 1167 tablet_bootstrap.cc:492] T e8402fd33ba842e981c6c68787c7c548 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:45.915940 1167 tablet_bootstrap.cc:654] T e8402fd33ba842e981c6c68787c7c548 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:45.917685 447 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet df1248f6dbd94af68c587da7d149e279
I20251025 07:52:45.918620 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet df1248f6dbd94af68c587da7d149e279 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:45 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:45.923542 878 tablet_service.cc:1552] Processing DeleteTablet for tablet df1248f6dbd94af68c587da7d149e279 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:45 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:45.926469 2169 ts_tablet_manager.cc:1929] T 3abfd52327a04ba88625f391a8658fbb P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:45.926594 2169 log.cc:1199] T 3abfd52327a04ba88625f391a8658fbb P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/3abfd52327a04ba88625f391a8658fbb
I20251025 07:52:45.927140 2169 ts_tablet_manager.cc:1950] T 3abfd52327a04ba88625f391a8658fbb P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:45.927460 2166 ts_tablet_manager.cc:1929] T 5fca357d7e24494fb6af4bd73d98833c P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:45.927551 2166 log.cc:1199] T 5fca357d7e24494fb6af4bd73d98833c P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/5fca357d7e24494fb6af4bd73d98833c
I20251025 07:52:45.927834 2166 ts_tablet_manager.cc:1950] T 5fca357d7e24494fb6af4bd73d98833c P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:45.928162 2671 raft_consensus.cc:2749] T 4678f7eab5e8497db308e8a38e387618 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:45.926103 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 131) took 7388 ms. Trace:
I20251025 07:52:45.933810 2146 rpcz_store.cc:276] 1025 07:52:38.537537 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.570198 (+ 32661us) service_pool.cc:225] Handling call
1025 07:52:45.502737 (+6932539us) ts_tablet_manager.cc:1151] Deleting tablet 4b28215f61574cf48c447dd01955ac77
1025 07:52:45.502747 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:45.766966 (+264219us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:45.833141 (+ 66175us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:45.926085 (+ 92944us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6736013}
I20251025 07:52:45.934088 2709 raft_consensus.cc:493] T 2e2e6893a48449969b71ee1c08d1960e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:45.928493 2687 raft_consensus.cc:515] T c0c6324cbae34dcebade3ef9f89a7150 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.934891 2687 leader_election.cc:290] T c0c6324cbae34dcebade3ef9f89a7150 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:45.934232 2709 raft_consensus.cc:515] T 2e2e6893a48449969b71ee1c08d1960e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.935928 2709 leader_election.cc:290] T 2e2e6893a48449969b71ee1c08d1960e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:45.936954 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e2e6893a48449969b71ee1c08d1960e" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:45.937433 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 134) took 7366 ms. Trace:
I20251025 07:52:45.937661 2146 rpcz_store.cc:276] 1025 07:52:38.570663 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.570717 (+ 54us) service_pool.cc:225] Handling call
1025 07:52:45.937272 (+7366555us) ts_tablet_manager.cc:1151] Deleting tablet af743ee344e543a0aa6da3cd7a30e467
1025 07:52:45.937288 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:45.937411 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7170397}
I20251025 07:52:45.938036 2146 tablet_replica.cc:333] T 34315f64b89148efbf1f20a43d5c5efb P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:45.938294 2146 raft_consensus.cc:2243] T 34315f64b89148efbf1f20a43d5c5efb P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:45.938674 2146 raft_consensus.cc:2272] T 34315f64b89148efbf1f20a43d5c5efb P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:45.939368 2146 ts_tablet_manager.cc:1916] T 34315f64b89148efbf1f20a43d5c5efb P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:45.933535 2615 raft_consensus.cc:515] T 143aa3fce9ba4850b000a94054e5dc3a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.943274 2615 leader_election.cc:290] T 143aa3fce9ba4850b000a94054e5dc3a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:45.947202 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143aa3fce9ba4850b000a94054e5dc3a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:45.947973 850 leader_election.cc:343] T 143aa3fce9ba4850b000a94054e5dc3a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:45.951089 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0c6324cbae34dcebade3ef9f89a7150" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:45.952454 850 leader_election.cc:343] T c0c6324cbae34dcebade3ef9f89a7150 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:45.953541 2671 raft_consensus.cc:493] T bdaeedbcf35a4aad97f9ef192991978a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:45.953714 2671 raft_consensus.cc:515] T bdaeedbcf35a4aad97f9ef192991978a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.954347 2671 leader_election.cc:290] T bdaeedbcf35a4aad97f9ef192991978a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:45.955183 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bdaeedbcf35a4aad97f9ef192991978a" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:45.956275 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bdaeedbcf35a4aad97f9ef192991978a" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:45.956514 776 raft_consensus.cc:2393] T bdaeedbcf35a4aad97f9ef192991978a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:45.957015 982 leader_election.cc:304] T bdaeedbcf35a4aad97f9ef192991978a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:45.957581 2671 raft_consensus.cc:3060] T bdaeedbcf35a4aad97f9ef192991978a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:45.960233 2671 raft_consensus.cc:2749] T bdaeedbcf35a4aad97f9ef192991978a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:45.961103 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 120) took 7280 ms. Trace:
I20251025 07:52:45.961210 2166 rpcz_store.cc:276] 1025 07:52:38.680773 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.680909 (+ 136us) service_pool.cc:225] Handling call
1025 07:52:45.738757 (+7057848us) ts_tablet_manager.cc:1151] Deleting tablet 5fca357d7e24494fb6af4bd73d98833c
1025 07:52:45.738770 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:45.829505 (+ 90735us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:45.927349 (+ 97844us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:45.961083 (+ 33734us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7057666}
I20251025 07:52:45.961587 2166 tablet_replica.cc:333] T c4610b0c62e741eca28c21f63cc3de81 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:45.961795 2166 raft_consensus.cc:2243] T c4610b0c62e741eca28c21f63cc3de81 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:45.962100 2166 raft_consensus.cc:2272] T c4610b0c62e741eca28c21f63cc3de81 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:45.963737 2166 ts_tablet_manager.cc:1916] T c4610b0c62e741eca28c21f63cc3de81 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:45.966840 1167 tablet_bootstrap.cc:492] T e8402fd33ba842e981c6c68787c7c548 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:45.966984 1167 ts_tablet_manager.cc:1403] T e8402fd33ba842e981c6c68787c7c548 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.053s user 0.006s sys 0.000s
I20251025 07:52:45.967700 1167 raft_consensus.cc:359] T e8402fd33ba842e981c6c68787c7c548 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:45.967871 1167 raft_consensus.cc:385] T e8402fd33ba842e981c6c68787c7c548 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:45.967924 1167 raft_consensus.cc:740] T e8402fd33ba842e981c6c68787c7c548 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:45.968122 1167 consensus_queue.cc:260] T e8402fd33ba842e981c6c68787c7c548 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:45.968607 1167 ts_tablet_manager.cc:1434] T e8402fd33ba842e981c6c68787c7c548 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 07:52:45.969020 1167 tablet_bootstrap.cc:492] T 6a33cfb2f2de48d4a5e9e39730d4900e P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:45.970700 1167 tablet_bootstrap.cc:654] T 6a33cfb2f2de48d4a5e9e39730d4900e P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:45.974970 2146 ts_tablet_manager.cc:1929] T 34315f64b89148efbf1f20a43d5c5efb P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:45.988193 1169 tablet_bootstrap.cc:492] T ce1ebb1ec14b4801930a72765bca1a7d P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:45.988406 1169 ts_tablet_manager.cc:1403] T ce1ebb1ec14b4801930a72765bca1a7d P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.296s user 0.005s sys 0.017s
I20251025 07:52:45.989198 1169 raft_consensus.cc:359] T ce1ebb1ec14b4801930a72765bca1a7d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.989418 1169 raft_consensus.cc:385] T ce1ebb1ec14b4801930a72765bca1a7d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:45.989504 1169 raft_consensus.cc:740] T ce1ebb1ec14b4801930a72765bca1a7d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:45.989781 1169 consensus_queue.cc:260] T ce1ebb1ec14b4801930a72765bca1a7d P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:45.998699 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 134) took 7163 ms. Trace:
I20251025 07:52:45.998827 2169 rpcz_store.cc:276] 1025 07:52:38.834887 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.834954 (+ 67us) service_pool.cc:225] Handling call
1025 07:52:45.736539 (+6901585us) ts_tablet_manager.cc:1151] Deleting tablet 3abfd52327a04ba88625f391a8658fbb
1025 07:52:45.736548 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:45.826231 (+ 89683us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:45.926337 (+100106us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:45.998675 (+ 72338us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6900025}
I20251025 07:52:45.999123 2169 tablet_replica.cc:333] T d66ad049d5484228a268b0facb00ecf1 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:45.999306 2169 raft_consensus.cc:2243] T d66ad049d5484228a268b0facb00ecf1 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:45.999617 2169 raft_consensus.cc:2272] T d66ad049d5484228a268b0facb00ecf1 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:46.000177 2169 ts_tablet_manager.cc:1916] T d66ad049d5484228a268b0facb00ecf1 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:46.005336 2146 log.cc:1199] T 34315f64b89148efbf1f20a43d5c5efb P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/34315f64b89148efbf1f20a43d5c5efb
I20251025 07:52:46.005736 2146 ts_tablet_manager.cc:1950] T 34315f64b89148efbf1f20a43d5c5efb P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:46.006023 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 135) took 7435 ms. Trace:
I20251025 07:52:46.006096 2146 rpcz_store.cc:276] 1025 07:52:38.570879 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.570951 (+ 72us) service_pool.cc:225] Handling call
1025 07:52:45.937977 (+7367026us) ts_tablet_manager.cc:1151] Deleting tablet 34315f64b89148efbf1f20a43d5c5efb
1025 07:52:45.937988 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:45.951588 (+ 13600us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:45.974859 (+ 23271us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.006008 (+ 31149us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7170997}
I20251025 07:52:46.010470 1169 ts_tablet_manager.cc:1434] T ce1ebb1ec14b4801930a72765bca1a7d P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20251025 07:52:46.012403 2169 ts_tablet_manager.cc:1929] T d66ad049d5484228a268b0facb00ecf1 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:46.012641 2169 log.cc:1199] T d66ad049d5484228a268b0facb00ecf1 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/d66ad049d5484228a268b0facb00ecf1
I20251025 07:52:46.013283 2169 ts_tablet_manager.cc:1950] T d66ad049d5484228a268b0facb00ecf1 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
W20251025 07:52:45.986254 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:46231, user_credentials={real_user=slave}} blocked reactor thread for 104674us
W20251025 07:52:45.992022 850 leader_election.cc:343] T 2e2e6893a48449969b71ee1c08d1960e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:46.016736 1169 tablet_bootstrap.cc:492] T a77eb82f26ae43b48de1e85693372d34 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:46.017444 2146 tablet_replica.cc:333] T a4e4b99be0434baaa75d988281797846 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:46.017799 771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c0c6324cbae34dcebade3ef9f89a7150" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:46.018009 771 raft_consensus.cc:3060] T c0c6324cbae34dcebade3ef9f89a7150 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:46.018502 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e2e6893a48449969b71ee1c08d1960e" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:46.018713 776 raft_consensus.cc:2468] T 2e2e6893a48449969b71ee1c08d1960e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:46.019335 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143aa3fce9ba4850b000a94054e5dc3a" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:46.019454 776 raft_consensus.cc:3060] T 143aa3fce9ba4850b000a94054e5dc3a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:46.020830 771 raft_consensus.cc:2468] T c0c6324cbae34dcebade3ef9f89a7150 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:46.022634 776 raft_consensus.cc:2468] T 143aa3fce9ba4850b000a94054e5dc3a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
W20251025 07:52:46.027731 440 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:37611, user_credentials={real_user=slave}} blocked reactor thread for 92349.3us
I20251025 07:52:46.028331 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 4b28215f61574cf48c447dd01955ac77 (table default.Table_a437a2623d704e3ea3c5b8c65c88b936 [id=51b12903e6024e1a9adc2f9fa7400937]) successfully deleted
I20251025 07:52:46.028425 2146 raft_consensus.cc:2243] T a4e4b99be0434baaa75d988281797846 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:46.029649 2420 raft_consensus.cc:493] T a0ea1edeca7347fc844faa5e4be1ae94 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:46.029804 2420 raft_consensus.cc:515] T a0ea1edeca7347fc844faa5e4be1ae94 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
W20251025 07:52:46.029968 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet af743ee344e543a0aa6da3cd7a30e467 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af743ee344e543a0aa6da3cd7a30e467 already in progress: creating tablet
I20251025 07:52:46.030545 2420 leader_election.cc:290] T a0ea1edeca7347fc844faa5e4be1ae94 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:46.031564 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 34315f64b89148efbf1f20a43d5c5efb (table default.Table_4b3d7b3a731942189cdc214938a45820 [id=88f1e8b1c5fa4913a5f7b5b2c6cbf6ec]) successfully deleted
I20251025 07:52:46.033586 1044 raft_consensus.cc:3060] T ce1ebb1ec14b4801930a72765bca1a7d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:46.033603 1048 raft_consensus.cc:3060] T 36c8fcafd5464b4e999276a8062a812a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:46.036069 2146 raft_consensus.cc:2272] T a4e4b99be0434baaa75d988281797846 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:46.037694 1169 tablet_bootstrap.cc:654] T a77eb82f26ae43b48de1e85693372d34 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:46.033525 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0ea1edeca7347fc844faa5e4be1ae94" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:46.038718 712 leader_election.cc:343] T a0ea1edeca7347fc844faa5e4be1ae94 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:46.039902 2698 consensus_queue.cc:1048] T ce1ebb1ec14b4801930a72765bca1a7d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:46.046245 2671 raft_consensus.cc:493] T 11bfe0c87ca941a3846462c8a4512d83 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:46.046417 2671 raft_consensus.cc:515] T 11bfe0c87ca941a3846462c8a4512d83 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.047055 2671 leader_election.cc:290] T 11bfe0c87ca941a3846462c8a4512d83 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:46.047597 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11bfe0c87ca941a3846462c8a4512d83" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:46.047657 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11bfe0c87ca941a3846462c8a4512d83" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:46.047830 768 raft_consensus.cc:2393] T 11bfe0c87ca941a3846462c8a4512d83 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:46.048202 982 leader_election.cc:304] T 11bfe0c87ca941a3846462c8a4512d83 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:46.048784 2728 raft_consensus.cc:3060] T 11bfe0c87ca941a3846462c8a4512d83 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:46.002290 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 3abfd52327a04ba88625f391a8658fbb (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]) successfully deleted
W20251025 07:52:46.049470 442 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:45417, user_credentials={real_user=slave}} blocked reactor thread for 47183.4us
I20251025 07:52:46.050419 2686 raft_consensus.cc:493] T 4a1ec8c2ed7f41abb588f592f62097d1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:46.050544 2686 raft_consensus.cc:515] T 4a1ec8c2ed7f41abb588f592f62097d1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.051041 2686 leader_election.cc:290] T 4a1ec8c2ed7f41abb588f592f62097d1 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:46.051414 2728 raft_consensus.cc:2749] T 11bfe0c87ca941a3846462c8a4512d83 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:46.051774 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a1ec8c2ed7f41abb588f592f62097d1" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:46.052464 850 leader_election.cc:343] T 4a1ec8c2ed7f41abb588f592f62097d1 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:46.067001 2420 raft_consensus.cc:493] T dff7f86ae8924685b26011e504e2247b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:46.067195 2420 raft_consensus.cc:515] T dff7f86ae8924685b26011e504e2247b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.067828 2420 leader_election.cc:290] T dff7f86ae8924685b26011e504e2247b P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:46.068607 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dff7f86ae8924685b26011e504e2247b" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:46.068918 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 135) took 7230 ms. Trace:
W20251025 07:52:46.069132 712 leader_election.cc:343] T dff7f86ae8924685b26011e504e2247b P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:46.069005 2169 rpcz_store.cc:276] 1025 07:52:38.839242 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.839533 (+ 291us) service_pool.cc:225] Handling call
1025 07:52:45.999074 (+7159541us) ts_tablet_manager.cc:1151] Deleting tablet d66ad049d5484228a268b0facb00ecf1
1025 07:52:45.999085 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:46.008036 (+ 8951us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.012218 (+ 4182us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.068902 (+ 56684us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7159351}
I20251025 07:52:46.071256 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 136) took 7230 ms. Trace:
I20251025 07:52:46.071259 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet d66ad049d5484228a268b0facb00ecf1 (table default.Table_9f52d94037424befbdea5c711068396a [id=55440328803042a28df1b6869d279f59]) successfully deleted
I20251025 07:52:46.071372 2169 rpcz_store.cc:276] 1025 07:52:38.841694 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.841744 (+ 50us) service_pool.cc:225] Handling call
1025 07:52:46.071130 (+7229386us) ts_tablet_manager.cc:1151] Deleting tablet af743ee344e543a0aa6da3cd7a30e467
1025 07:52:46.071139 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:46.071222 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7229233}
I20251025 07:52:46.071786 2169 tablet_replica.cc:333] T 34315f64b89148efbf1f20a43d5c5efb P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
W20251025 07:52:46.071779 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet af743ee344e543a0aa6da3cd7a30e467 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af743ee344e543a0aa6da3cd7a30e467 already in progress: creating tablet
I20251025 07:52:46.072008 2169 raft_consensus.cc:2243] T 34315f64b89148efbf1f20a43d5c5efb P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:46.072876 2169 raft_consensus.cc:2272] T 34315f64b89148efbf1f20a43d5c5efb P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:46.073500 2169 ts_tablet_manager.cc:1916] T 34315f64b89148efbf1f20a43d5c5efb P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:46.074518 2146 ts_tablet_manager.cc:1916] T a4e4b99be0434baaa75d988281797846 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:46.096402 2420 raft_consensus.cc:493] T c187a40f8ff14c6a8c184fbeb030ad13 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:46.096596 2420 raft_consensus.cc:515] T c187a40f8ff14c6a8c184fbeb030ad13 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.097224 2420 leader_election.cc:290] T c187a40f8ff14c6a8c184fbeb030ad13 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:46.097829 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c187a40f8ff14c6a8c184fbeb030ad13" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:46.106545 712 leader_election.cc:343] T c187a40f8ff14c6a8c184fbeb030ad13 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:46.103765 1169 tablet_bootstrap.cc:492] T a77eb82f26ae43b48de1e85693372d34 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:46.109869 1169 ts_tablet_manager.cc:1403] T a77eb82f26ae43b48de1e85693372d34 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.093s user 0.003s sys 0.000s
W20251025 07:52:46.111516 439 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:42203, user_credentials={real_user=slave}} blocked reactor thread for 199824us
I20251025 07:52:46.112085 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 5fca357d7e24494fb6af4bd73d98833c (table default.Table_400f06d166054eeb97890ad56453afb2 [id=5f20de141a6f452a82ed736f41c29d9b]) successfully deleted
I20251025 07:52:46.112397 748 tablet_service.cc:1552] Processing DeleteTablet for tablet df1248f6dbd94af68c587da7d149e279 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:45 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:46.117796 2166 ts_tablet_manager.cc:1929] T c4610b0c62e741eca28c21f63cc3de81 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:46.117904 2166 log.cc:1199] T c4610b0c62e741eca28c21f63cc3de81 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/c4610b0c62e741eca28c21f63cc3de81
I20251025 07:52:46.118589 2420 raft_consensus.cc:493] T 70fe9aa2f58d4abea3b3b71066a29653 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db42be7ad3c741a0aa427f27e245993b)
I20251025 07:52:46.118724 2420 raft_consensus.cc:515] T 70fe9aa2f58d4abea3b3b71066a29653 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:46.119328 2420 leader_election.cc:290] T 70fe9aa2f58d4abea3b3b71066a29653 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:46.116243 1169 raft_consensus.cc:359] T a77eb82f26ae43b48de1e85693372d34 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.119799 1169 raft_consensus.cc:385] T a77eb82f26ae43b48de1e85693372d34 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:46.119867 1169 raft_consensus.cc:740] T a77eb82f26ae43b48de1e85693372d34 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:46.120095 1169 consensus_queue.cc:260] T a77eb82f26ae43b48de1e85693372d34 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
W20251025 07:52:46.120361 712 leader_election.cc:343] T 70fe9aa2f58d4abea3b3b71066a29653 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 70fe9aa2f58d4abea3b3b71066a29653
I20251025 07:52:46.120796 1169 ts_tablet_manager.cc:1434] T a77eb82f26ae43b48de1e85693372d34 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20251025 07:52:46.126044 1166 tablet_bootstrap.cc:654] T 9a8e2658a81e436298fe2190726080d4 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:46.128206 1169 tablet_bootstrap.cc:492] T ebc824ec4c60411dab1a2ea4a385148e P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:46.130088 1169 tablet_bootstrap.cc:654] T ebc824ec4c60411dab1a2ea4a385148e P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:46.135927 1046 raft_consensus.cc:3060] T 0551e5d694894e8fbeb66b5e0c1cd75b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:46.152451 2169 ts_tablet_manager.cc:1929] T 34315f64b89148efbf1f20a43d5c5efb P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:46.152659 2169 log.cc:1199] T 34315f64b89148efbf1f20a43d5c5efb P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/34315f64b89148efbf1f20a43d5c5efb
I20251025 07:52:46.153090 2169 ts_tablet_manager.cc:1950] T 34315f64b89148efbf1f20a43d5c5efb P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:46.154001 1166 tablet_bootstrap.cc:492] T 9a8e2658a81e436298fe2190726080d4 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:46.154484 1166 ts_tablet_manager.cc:1403] T 9a8e2658a81e436298fe2190726080d4 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.252s user 0.006s sys 0.004s
I20251025 07:52:46.155694 1166 raft_consensus.cc:359] T 9a8e2658a81e436298fe2190726080d4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.155923 1166 raft_consensus.cc:385] T 9a8e2658a81e436298fe2190726080d4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:46.156015 1166 raft_consensus.cc:740] T 9a8e2658a81e436298fe2190726080d4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:46.156297 1166 consensus_queue.cc:260] T 9a8e2658a81e436298fe2190726080d4 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.157070 1166 ts_tablet_manager.cc:1434] T 9a8e2658a81e436298fe2190726080d4 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:46.157967 2146 ts_tablet_manager.cc:1929] T a4e4b99be0434baaa75d988281797846 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:46.158588 2146 log.cc:1199] T a4e4b99be0434baaa75d988281797846 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/a4e4b99be0434baaa75d988281797846
I20251025 07:52:46.159040 2146 ts_tablet_manager.cc:1950] T a4e4b99be0434baaa75d988281797846 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:46.159379 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 136) took 7588 ms. Trace:
I20251025 07:52:46.160153 1166 tablet_bootstrap.cc:492] T 682c1e22d7764dce862d970b33b88cb3 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:46.162382 1166 tablet_bootstrap.cc:654] T 682c1e22d7764dce862d970b33b88cb3 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:46.159479 2146 rpcz_store.cc:276] 1025 07:52:38.571159 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.571206 (+ 47us) service_pool.cc:225] Handling call
1025 07:52:46.017386 (+7446180us) ts_tablet_manager.cc:1151] Deleting tablet a4e4b99be0434baaa75d988281797846
1025 07:52:46.017398 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:46.123919 (+106521us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.157846 (+ 33927us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.159365 (+ 1519us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7250318}
I20251025 07:52:46.163549 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet a4e4b99be0434baaa75d988281797846 (table default.Table_dac528b081844b7c86027169c6234c27 [id=b0caa78ee8c84aa8bea4818cf9feb70c]) successfully deleted
I20251025 07:52:46.163936 2146 tablet_replica.cc:333] T 92aa605fabbc4257a6a204c087e7cc7f P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:46.164165 2146 raft_consensus.cc:2243] T 92aa605fabbc4257a6a204c087e7cc7f P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:46.164366 2146 raft_consensus.cc:2272] T 92aa605fabbc4257a6a204c087e7cc7f P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:46.164698 2166 ts_tablet_manager.cc:1950] T c4610b0c62e741eca28c21f63cc3de81 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:46.165287 2146 ts_tablet_manager.cc:1916] T 92aa605fabbc4257a6a204c087e7cc7f P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:46.165526 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a1ec8c2ed7f41abb588f592f62097d1" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:46.166285 768 raft_consensus.cc:2468] T 4a1ec8c2ed7f41abb588f592f62097d1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:46.166906 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 121) took 7484 ms. Trace:
I20251025 07:52:46.166996 2166 rpcz_store.cc:276] 1025 07:52:38.682401 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.682445 (+ 44us) service_pool.cc:225] Handling call
1025 07:52:45.961515 (+7279070us) ts_tablet_manager.cc:1151] Deleting tablet c4610b0c62e741eca28c21f63cc3de81
1025 07:52:45.961532 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:46.114653 (+153121us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.117699 (+ 3046us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.166891 (+ 49192us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7278850}
I20251025 07:52:46.167305 2166 tablet_replica.cc:333] T 2b594d4718344aafbdaec34040695589 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:46.167486 2166 raft_consensus.cc:2243] T 2b594d4718344aafbdaec34040695589 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:46.167629 2166 raft_consensus.cc:2272] T 2b594d4718344aafbdaec34040695589 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:46.168965 2166 ts_tablet_manager.cc:1916] T 2b594d4718344aafbdaec34040695589 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:46.170001 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0ea1edeca7347fc844faa5e4be1ae94" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:46.170614 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dff7f86ae8924685b26011e504e2247b" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:46.171186 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet c4610b0c62e741eca28c21f63cc3de81 (table default.Table_c9bc68f9b5c54a33b854f9ee8a1ad311 [id=d8f616c04cc54b86af0af3cc073c5e2d]) successfully deleted
I20251025 07:52:46.164664 1166 tablet_bootstrap.cc:492] T 682c1e22d7764dce862d970b33b88cb3 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:46.178313 1166 ts_tablet_manager.cc:1403] T 682c1e22d7764dce862d970b33b88cb3 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20251025 07:52:46.179178 1166 raft_consensus.cc:359] T 682c1e22d7764dce862d970b33b88cb3 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.180079 1167 tablet_bootstrap.cc:492] T 6a33cfb2f2de48d4a5e9e39730d4900e P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:46.180207 1166 raft_consensus.cc:385] T 682c1e22d7764dce862d970b33b88cb3 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:46.180240 1167 ts_tablet_manager.cc:1403] T 6a33cfb2f2de48d4a5e9e39730d4900e P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.211s user 0.036s sys 0.035s
I20251025 07:52:46.180315 1166 raft_consensus.cc:740] T 682c1e22d7764dce862d970b33b88cb3 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:46.180980 1167 raft_consensus.cc:359] T 6a33cfb2f2de48d4a5e9e39730d4900e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.181135 1167 raft_consensus.cc:385] T 6a33cfb2f2de48d4a5e9e39730d4900e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:46.181198 1167 raft_consensus.cc:740] T 6a33cfb2f2de48d4a5e9e39730d4900e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:46.181334 1166 consensus_queue.cc:260] T 682c1e22d7764dce862d970b33b88cb3 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.181439 1167 consensus_queue.cc:260] T 6a33cfb2f2de48d4a5e9e39730d4900e P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.181917 1167 ts_tablet_manager.cc:1434] T 6a33cfb2f2de48d4a5e9e39730d4900e P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 07:52:46.182354 1167 tablet_bootstrap.cc:492] T fd265cdbf6a8440393a2b5d8240d1872 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:46.183825 1167 tablet_bootstrap.cc:654] T fd265cdbf6a8440393a2b5d8240d1872 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:46.184840 1166 ts_tablet_manager.cc:1434] T 682c1e22d7764dce862d970b33b88cb3 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20251025 07:52:46.187422 1166 tablet_bootstrap.cc:492] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:46.190861 1166 tablet_bootstrap.cc:654] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:46.187094 2146 ts_tablet_manager.cc:1929] T 92aa605fabbc4257a6a204c087e7cc7f P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:46.195044 2146 log.cc:1199] T 92aa605fabbc4257a6a204c087e7cc7f P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/92aa605fabbc4257a6a204c087e7cc7f
I20251025 07:52:46.195416 2146 ts_tablet_manager.cc:1950] T 92aa605fabbc4257a6a204c087e7cc7f P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:46.195762 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 137) took 7624 ms. Trace:
I20251025 07:52:46.195874 2146 rpcz_store.cc:276] 1025 07:52:38.571355 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.571404 (+ 49us) service_pool.cc:225] Handling call
1025 07:52:46.163884 (+7592480us) ts_tablet_manager.cc:1151] Deleting tablet 92aa605fabbc4257a6a204c087e7cc7f
1025 07:52:46.163900 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:46.179400 (+ 15500us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.186988 (+ 7588us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.195746 (+ 8758us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7396711}
I20251025 07:52:46.196380 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 92aa605fabbc4257a6a204c087e7cc7f (table default.Table_78724078b4574f35879fd2db388fc4f4 [id=187bde879b044237ada58158eb000123]) successfully deleted
I20251025 07:52:46.196728 2146 tablet_replica.cc:333] T 3abfd52327a04ba88625f391a8658fbb P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:46.196949 2146 raft_consensus.cc:2243] T 3abfd52327a04ba88625f391a8658fbb P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:46.197151 2146 raft_consensus.cc:2272] T 3abfd52327a04ba88625f391a8658fbb P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:46.197731 2146 ts_tablet_manager.cc:1916] T 3abfd52327a04ba88625f391a8658fbb P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:46.198544 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 137) took 7356 ms. Trace:
I20251025 07:52:46.201107 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c187a40f8ff14c6a8c184fbeb030ad13" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:46.203236 847 leader_election.cc:304] T 2e2e6893a48449969b71ee1c08d1960e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:46.203758 2698 raft_consensus.cc:2804] T 2e2e6893a48449969b71ee1c08d1960e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:46.203823 2698 raft_consensus.cc:493] T 2e2e6893a48449969b71ee1c08d1960e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:46.203886 2698 raft_consensus.cc:3060] T 2e2e6893a48449969b71ee1c08d1960e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:46.205142 2166 ts_tablet_manager.cc:1929] T 2b594d4718344aafbdaec34040695589 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:46.205240 2166 log.cc:1199] T 2b594d4718344aafbdaec34040695589 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/2b594d4718344aafbdaec34040695589
I20251025 07:52:46.205503 2166 ts_tablet_manager.cc:1950] T 2b594d4718344aafbdaec34040695589 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
W20251025 07:52:46.206750 983 consensus_peers.cc:597] T 4b28215f61574cf48c447dd01955ac77 P db42be7ad3c741a0aa427f27e245993b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b28215f61574cf48c447dd01955ac77. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:46.206965 2698 raft_consensus.cc:515] T 2e2e6893a48449969b71ee1c08d1960e P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.207715 2698 leader_election.cc:290] T 2e2e6893a48449969b71ee1c08d1960e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:46.198648 2169 rpcz_store.cc:276] 1025 07:52:38.843150 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.843233 (+ 83us) service_pool.cc:225] Handling call
1025 07:52:46.071740 (+7228507us) ts_tablet_manager.cc:1151] Deleting tablet 34315f64b89148efbf1f20a43d5c5efb
1025 07:52:46.071750 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:46.137513 (+ 65763us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.152333 (+ 14820us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.198514 (+ 46181us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7228349}
I20251025 07:52:46.209637 2169 tablet_replica.cc:333] T 50ae9bca6bed4f749d54191ac1543410 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:46.209842 2169 raft_consensus.cc:2243] T 50ae9bca6bed4f749d54191ac1543410 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:46.210005 2169 raft_consensus.cc:2272] T 50ae9bca6bed4f749d54191ac1543410 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:46.210589 2169 ts_tablet_manager.cc:1916] T 50ae9bca6bed4f749d54191ac1543410 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:46.211678 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 34315f64b89148efbf1f20a43d5c5efb (table default.Table_4b3d7b3a731942189cdc214938a45820 [id=88f1e8b1c5fa4913a5f7b5b2c6cbf6ec]) successfully deleted
I20251025 07:52:46.211824 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e2e6893a48449969b71ee1c08d1960e" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:46.212219 2686 consensus_queue.cc:1048] T 4678f7eab5e8497db308e8a38e387618 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:46.212291 1167 tablet_bootstrap.cc:492] T fd265cdbf6a8440393a2b5d8240d1872 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:46.212417 1167 ts_tablet_manager.cc:1403] T fd265cdbf6a8440393a2b5d8240d1872 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.007s
I20251025 07:52:46.213078 1167 raft_consensus.cc:359] T fd265cdbf6a8440393a2b5d8240d1872 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:46.213253 1167 raft_consensus.cc:385] T fd265cdbf6a8440393a2b5d8240d1872 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251025 07:52:46.213228 850 leader_election.cc:343] T 2e2e6893a48449969b71ee1c08d1960e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:46.213305 1167 raft_consensus.cc:740] T fd265cdbf6a8440393a2b5d8240d1872 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:46.213483 1167 consensus_queue.cc:260] T fd265cdbf6a8440393a2b5d8240d1872 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:46.214059 1167 ts_tablet_manager.cc:1434] T fd265cdbf6a8440393a2b5d8240d1872 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:46.214339 1167 tablet_bootstrap.cc:492] T fce535a0c0bd45c6bf17f81add297f40 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:46.216220 1167 tablet_bootstrap.cc:654] T fce535a0c0bd45c6bf17f81add297f40 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:46.219200 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 122) took 7536 ms. Trace:
I20251025 07:52:46.219298 2166 rpcz_store.cc:276] 1025 07:52:38.683125 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.683167 (+ 42us) service_pool.cc:225] Handling call
1025 07:52:46.167257 (+7484090us) ts_tablet_manager.cc:1151] Deleting tablet 2b594d4718344aafbdaec34040695589
1025 07:52:46.167266 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:46.201699 (+ 34433us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.205049 (+ 3350us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.219179 (+ 14130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7483930}
I20251025 07:52:46.220320 1166 tablet_bootstrap.cc:492] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:46.220523 1166 ts_tablet_manager.cc:1403] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.002s
I20251025 07:52:46.220757 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 2b594d4718344aafbdaec34040695589 (table default.Table_e08a3ff70f3a411e82b4bb292d768961 [id=8d075a6652b14a37a2bfbfddc6139eb9]) successfully deleted
I20251025 07:52:46.221325 1166 raft_consensus.cc:359] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.221541 1166 raft_consensus.cc:385] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:46.221661 1166 raft_consensus.cc:740] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:46.221860 2166 tablet_replica.cc:333] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:46.221985 1166 consensus_queue.cc:260] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.222730 1166 ts_tablet_manager.cc:1434] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251025 07:52:46.223169 1166 tablet_bootstrap.cc:492] T 34c63b69904d499bb0d286941a8303f2 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:46.224560 2117 raft_consensus.cc:493] T 71432a6848944a0da06ef9b81f160dc4 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db42be7ad3c741a0aa427f27e245993b)
I20251025 07:52:46.224701 2117 raft_consensus.cc:515] T 71432a6848944a0da06ef9b81f160dc4 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:46.225309 2117 leader_election.cc:290] T 71432a6848944a0da06ef9b81f160dc4 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
W20251025 07:52:46.226259 712 leader_election.cc:343] T 71432a6848944a0da06ef9b81f160dc4 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 71432a6848944a0da06ef9b81f160dc4
I20251025 07:52:46.228540 1167 tablet_bootstrap.cc:492] T fce535a0c0bd45c6bf17f81add297f40 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:46.228715 1167 ts_tablet_manager.cc:1403] T fce535a0c0bd45c6bf17f81add297f40 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.001s
I20251025 07:52:46.229475 1167 raft_consensus.cc:359] T fce535a0c0bd45c6bf17f81add297f40 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.229678 1167 raft_consensus.cc:385] T fce535a0c0bd45c6bf17f81add297f40 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:46.229771 1167 raft_consensus.cc:740] T fce535a0c0bd45c6bf17f81add297f40 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:46.230031 1167 consensus_queue.cc:260] T fce535a0c0bd45c6bf17f81add297f40 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.230588 1166 tablet_bootstrap.cc:654] T 34c63b69904d499bb0d286941a8303f2 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:46.230603 1167 ts_tablet_manager.cc:1434] T fce535a0c0bd45c6bf17f81add297f40 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:46.230933 1167 tablet_bootstrap.cc:492] T 29dbaf9149134f548a974a5779f7da67 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:46.232542 1167 tablet_bootstrap.cc:654] T 29dbaf9149134f548a974a5779f7da67 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:46.235549 2117 raft_consensus.cc:493] T 6a33cfb2f2de48d4a5e9e39730d4900e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:46.235709 2117 raft_consensus.cc:515] T 6a33cfb2f2de48d4a5e9e39730d4900e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.236366 2117 leader_election.cc:290] T 6a33cfb2f2de48d4a5e9e39730d4900e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:46.237111 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a33cfb2f2de48d4a5e9e39730d4900e" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:46.237713 712 leader_election.cc:343] T 6a33cfb2f2de48d4a5e9e39730d4900e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:46.238571 1167 tablet_bootstrap.cc:492] T 29dbaf9149134f548a974a5779f7da67 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:46.238699 1167 ts_tablet_manager.cc:1403] T 29dbaf9149134f548a974a5779f7da67 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20251025 07:52:46.239387 1167 raft_consensus.cc:359] T 29dbaf9149134f548a974a5779f7da67 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:46.239537 1167 raft_consensus.cc:385] T 29dbaf9149134f548a974a5779f7da67 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:46.239593 1167 raft_consensus.cc:740] T 29dbaf9149134f548a974a5779f7da67 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:46.222170 2166 raft_consensus.cc:2243] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:46.242594 2166 raft_consensus.cc:2272] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:46.242882 1166 tablet_bootstrap.cc:492] T 34c63b69904d499bb0d286941a8303f2 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:46.243068 1166 ts_tablet_manager.cc:1403] T 34c63b69904d499bb0d286941a8303f2 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.001s
I20251025 07:52:46.243795 1166 raft_consensus.cc:359] T 34c63b69904d499bb0d286941a8303f2 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.244001 1166 raft_consensus.cc:385] T 34c63b69904d499bb0d286941a8303f2 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:46.243342 748 tablet_service.cc:1552] Processing DeleteTablet for tablet ac0ea298bc114c56a8093e12def21ce1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:46.244099 1166 raft_consensus.cc:740] T 34c63b69904d499bb0d286941a8303f2 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:46.244287 1166 consensus_queue.cc:260] T 34c63b69904d499bb0d286941a8303f2 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.244592 2166 ts_tablet_manager.cc:1916] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:46.244894 1166 ts_tablet_manager.cc:1434] T 34c63b69904d499bb0d286941a8303f2 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:46.245224 1166 tablet_bootstrap.cc:492] T 9f039d9107ae493486d799ab6a1d731d P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:46.245771 1167 consensus_queue.cc:260] T 29dbaf9149134f548a974a5779f7da67 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:46.246284 1167 ts_tablet_manager.cc:1434] T 29dbaf9149134f548a974a5779f7da67 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20251025 07:52:46.246551 1167 tablet_bootstrap.cc:492] T d77704b4fe6c4456a2ba40ffec0c60fc P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:46.247495 2117 raft_consensus.cc:493] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:46.247642 2117 raft_consensus.cc:515] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.248274 2117 leader_election.cc:290] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:46.248739 1167 tablet_bootstrap.cc:654] T d77704b4fe6c4456a2ba40ffec0c60fc P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:46.248929 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aeba18be0cb4eb9949aeaf6ad73a812" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:46.249621 712 leader_election.cc:343] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:46.250265 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:46231, user_credentials={real_user=slave}} blocked reactor thread for 47104.3us
I20251025 07:52:46.250487 847 leader_election.cc:304] T c0c6324cbae34dcebade3ef9f89a7150 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:46.250712 2698 raft_consensus.cc:2804] T c0c6324cbae34dcebade3ef9f89a7150 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:46.250813 2698 raft_consensus.cc:697] T c0c6324cbae34dcebade3ef9f89a7150 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:46.251036 2698 consensus_queue.cc:237] T c0c6324cbae34dcebade3ef9f89a7150 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.273068 1166 tablet_bootstrap.cc:654] T 9f039d9107ae493486d799ab6a1d731d P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:46.274775 2117 raft_consensus.cc:493] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:46.274921 2117 raft_consensus.cc:515] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.275672 2117 leader_election.cc:290] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:46.276382 1044 raft_consensus.cc:3060] T 91728987303b41fd840eb56f3a311c5f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:46.281086 2709 consensus_queue.cc:1048] T 11bfe0c87ca941a3846462c8a4512d83 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:46.282490 2117 raft_consensus.cc:493] T e8402fd33ba842e981c6c68787c7c548 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:46.282683 1167 tablet_bootstrap.cc:492] T d77704b4fe6c4456a2ba40ffec0c60fc P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:46.282629 2117 raft_consensus.cc:515] T e8402fd33ba842e981c6c68787c7c548 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:46.283041 1167 ts_tablet_manager.cc:1403] T d77704b4fe6c4456a2ba40ffec0c60fc P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.003s
I20251025 07:52:46.283289 2117 leader_election.cc:290] T e8402fd33ba842e981c6c68787c7c548 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:46.283403 1166 tablet_bootstrap.cc:492] T 9f039d9107ae493486d799ab6a1d731d P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:46.283586 1166 ts_tablet_manager.cc:1403] T 9f039d9107ae493486d799ab6a1d731d P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.005s
I20251025 07:52:46.284042 1167 raft_consensus.cc:359] T d77704b4fe6c4456a2ba40ffec0c60fc P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.284554 1167 raft_consensus.cc:385] T d77704b4fe6c4456a2ba40ffec0c60fc P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:46.284482 1166 raft_consensus.cc:359] T 9f039d9107ae493486d799ab6a1d731d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.284643 1167 raft_consensus.cc:740] T d77704b4fe6c4456a2ba40ffec0c60fc P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:46.284765 1166 raft_consensus.cc:385] T 9f039d9107ae493486d799ab6a1d731d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:46.284873 1166 raft_consensus.cc:740] T 9f039d9107ae493486d799ab6a1d731d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:46.284897 1167 consensus_queue.cc:260] T d77704b4fe6c4456a2ba40ffec0c60fc P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.285136 1166 consensus_queue.cc:260] T 9f039d9107ae493486d799ab6a1d731d P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.285668 1167 ts_tablet_manager.cc:1434] T d77704b4fe6c4456a2ba40ffec0c60fc P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:46.285955 1167 tablet_bootstrap.cc:492] T a7ab3150a35b4caca9119a1374d3f2f7 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:46.285869 1166 ts_tablet_manager.cc:1434] T 9f039d9107ae493486d799ab6a1d731d P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251025 07:52:46.286674 1166 tablet_bootstrap.cc:492] T 141e413d2faa4c67b4c822192e631417 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:46.287585 1167 tablet_bootstrap.cc:654] T a7ab3150a35b4caca9119a1374d3f2f7 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:46.287644 2146 ts_tablet_manager.cc:1929] T 3abfd52327a04ba88625f391a8658fbb P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:46.287952 2146 log.cc:1199] T 3abfd52327a04ba88625f391a8658fbb P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/3abfd52327a04ba88625f391a8658fbb
I20251025 07:52:46.288566 1166 tablet_bootstrap.cc:654] T 141e413d2faa4c67b4c822192e631417 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:46.288821 2146 ts_tablet_manager.cc:1950] T 3abfd52327a04ba88625f391a8658fbb P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:46.290248 847 leader_election.cc:304] T 143aa3fce9ba4850b000a94054e5dc3a P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:46.290351 1169 tablet_bootstrap.cc:492] T ebc824ec4c60411dab1a2ea4a385148e P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:46.290499 2615 raft_consensus.cc:2804] T 143aa3fce9ba4850b000a94054e5dc3a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:46.290627 2615 raft_consensus.cc:697] T 143aa3fce9ba4850b000a94054e5dc3a P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:46.290912 2166 ts_tablet_manager.cc:1929] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:46.290881 2615 consensus_queue.cc:237] T 143aa3fce9ba4850b000a94054e5dc3a P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.290999 2166 log.cc:1199] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/02e4c4d6ffb34cecaf17beb5ba197e81
I20251025 07:52:46.291222 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 133) took 7720 ms. Trace:
I20251025 07:52:46.291409 2146 rpcz_store.cc:276] 1025 07:52:38.570566 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.574194 (+ 3628us) service_pool.cc:225] Handling call
1025 07:52:46.196681 (+7622487us) ts_tablet_manager.cc:1151] Deleting tablet 3abfd52327a04ba88625f391a8658fbb
1025 07:52:46.196691 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:46.274017 (+ 77326us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.287436 (+ 13419us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.291201 (+ 3765us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7429265}
I20251025 07:52:46.292090 1169 ts_tablet_manager.cc:1403] T ebc824ec4c60411dab1a2ea4a385148e P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.168s user 0.005s sys 0.017s
I20251025 07:52:46.292155 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 3abfd52327a04ba88625f391a8658fbb (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]) successfully deleted
I20251025 07:52:46.293124 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8402fd33ba842e981c6c68787c7c548" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:46.293936 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d3c9954693d4e8782ee0128ef066aa9" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:46.294523 983 consensus_peers.cc:597] T 92aa605fabbc4257a6a204c087e7cc7f P db42be7ad3c741a0aa427f27e245993b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 92aa605fabbc4257a6a204c087e7cc7f. This is attempt 1: this message will repeat every 5th retry.
W20251025 07:52:46.295141 712 leader_election.cc:343] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:46.296556 2146 tablet_replica.cc:333] T d66ad049d5484228a268b0facb00ecf1 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:46.297235 2146 raft_consensus.cc:2243] T d66ad049d5484228a268b0facb00ecf1 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:46.297387 2686 consensus_queue.cc:1048] T ce1ebb1ec14b4801930a72765bca1a7d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:46.297580 847 leader_election.cc:304] T 4a1ec8c2ed7f41abb588f592f62097d1 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:46.298241 2166 ts_tablet_manager.cc:1950] T 02e4c4d6ffb34cecaf17beb5ba197e81 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:46.298360 2169 ts_tablet_manager.cc:1929] T 50ae9bca6bed4f749d54191ac1543410 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:46.299278 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 123) took 7597 ms. Trace:
I20251025 07:52:46.299398 2166 rpcz_store.cc:276] 1025 07:52:38.701430 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.701495 (+ 65us) service_pool.cc:225] Handling call
1025 07:52:46.221713 (+7520218us) ts_tablet_manager.cc:1151] Deleting tablet 02e4c4d6ffb34cecaf17beb5ba197e81
1025 07:52:46.221725 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:46.271241 (+ 49516us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.290803 (+ 19562us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.299254 (+ 8451us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7500263}
I20251025 07:52:46.299947 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 02e4c4d6ffb34cecaf17beb5ba197e81 (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]) successfully deleted
I20251025 07:52:46.300695 2169 log.cc:1199] T 50ae9bca6bed4f749d54191ac1543410 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/50ae9bca6bed4f749d54191ac1543410
I20251025 07:52:46.301172 2169 ts_tablet_manager.cc:1950] T 50ae9bca6bed4f749d54191ac1543410 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:46.303714 2166 tablet_replica.cc:333] T 7c7a9d7a4fb3449bb57eafa83230a839 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:46.303952 2166 raft_consensus.cc:2243] T 7c7a9d7a4fb3449bb57eafa83230a839 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:46.304109 2166 raft_consensus.cc:2272] T 7c7a9d7a4fb3449bb57eafa83230a839 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:46.304836 2166 ts_tablet_manager.cc:1916] T 7c7a9d7a4fb3449bb57eafa83230a839 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:46.305088 712 leader_election.cc:343] T e8402fd33ba842e981c6c68787c7c548 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:46.305994 2117 raft_consensus.cc:493] T c501ef5d70e742598c1a902efe47feb1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:46.306231 2117 raft_consensus.cc:515] T c501ef5d70e742598c1a902efe47feb1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.306802 2117 leader_election.cc:290] T c501ef5d70e742598c1a902efe47feb1 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:46.307332 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 138) took 7458 ms. Trace:
I20251025 07:52:46.307469 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c501ef5d70e742598c1a902efe47feb1" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:46.307435 2169 rpcz_store.cc:276] 1025 07:52:38.849385 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.849597 (+ 212us) service_pool.cc:225] Handling call
1025 07:52:46.209582 (+7359985us) ts_tablet_manager.cc:1151] Deleting tablet 50ae9bca6bed4f749d54191ac1543410
1025 07:52:46.209591 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:46.294659 (+ 85068us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.298273 (+ 3614us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.307313 (+ 9040us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7359800}
W20251025 07:52:46.307948 712 leader_election.cc:343] T c501ef5d70e742598c1a902efe47feb1 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:46.308012 2169 tablet_replica.cc:333] T a4e4b99be0434baaa75d988281797846 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:46.308120 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 50ae9bca6bed4f749d54191ac1543410 (table default.Table_b42187d20a7e4f8783a8d25acb249350 [id=a5c73cb1e5a349ca8675f4ba4cb8a82d]) successfully deleted
I20251025 07:52:46.308249 2169 raft_consensus.cc:2243] T a4e4b99be0434baaa75d988281797846 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:46.308388 2169 raft_consensus.cc:2272] T a4e4b99be0434baaa75d988281797846 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:46.312628 1169 raft_consensus.cc:359] T ebc824ec4c60411dab1a2ea4a385148e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.313759 2169 ts_tablet_manager.cc:1916] T a4e4b99be0434baaa75d988281797846 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:46.313797 1169 raft_consensus.cc:385] T ebc824ec4c60411dab1a2ea4a385148e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:46.313990 1169 raft_consensus.cc:740] T ebc824ec4c60411dab1a2ea4a385148e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:46.314384 1169 consensus_queue.cc:260] T ebc824ec4c60411dab1a2ea4a385148e P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.318681 1169 ts_tablet_manager.cc:1434] T ebc824ec4c60411dab1a2ea4a385148e P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20251025 07:52:46.318727 2166 ts_tablet_manager.cc:1929] T 7c7a9d7a4fb3449bb57eafa83230a839 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:46.319088 1169 tablet_bootstrap.cc:492] T 2e2e6893a48449969b71ee1c08d1960e P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:46.325007 1169 tablet_bootstrap.cc:654] T 2e2e6893a48449969b71ee1c08d1960e P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:46.326668 2166 log.cc:1199] T 7c7a9d7a4fb3449bb57eafa83230a839 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/7c7a9d7a4fb3449bb57eafa83230a839
I20251025 07:52:46.327461 2709 raft_consensus.cc:2804] T 4a1ec8c2ed7f41abb588f592f62097d1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:46.327342 1167 tablet_bootstrap.cc:492] T a7ab3150a35b4caca9119a1374d3f2f7 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:46.327548 2146 raft_consensus.cc:2272] T d66ad049d5484228a268b0facb00ecf1 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:46.327720 1167 ts_tablet_manager.cc:1403] T a7ab3150a35b4caca9119a1374d3f2f7 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.042s user 0.007s sys 0.001s
I20251025 07:52:46.328118 2146 ts_tablet_manager.cc:1916] T d66ad049d5484228a268b0facb00ecf1 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:46.328474 1167 raft_consensus.cc:359] T a7ab3150a35b4caca9119a1374d3f2f7 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.328639 1167 raft_consensus.cc:385] T a7ab3150a35b4caca9119a1374d3f2f7 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:46.328689 1167 raft_consensus.cc:740] T a7ab3150a35b4caca9119a1374d3f2f7 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:46.328915 1167 consensus_queue.cc:260] T a7ab3150a35b4caca9119a1374d3f2f7 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.331490 2710 consensus_queue.cc:1048] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:46.338233 451 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 106) took 1228 ms. Trace:
I20251025 07:52:46.338332 878 tablet_service.cc:1505] Processing CreateTablet for tablet 77867309b24c47c89da7296dde89c183 (DEFAULT_TABLE table=default.Table_0c3a15ef44874a679e54197a65dc10c2 [id=6429a7cc57b4477f9c49d8c06d032904]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:46.338379 451 rpcz_store.cc:276] 1025 07:52:45.109813 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.722547 (+612734us) service_pool.cc:225] Handling call
1025 07:52:45.905059 (+182512us) spinlock_profiling.cc:243] Waited 182 ms on lock 0x616000041254. stack: 00007fb90f7779f6 00007fb90f7777b0 00007fb90e7533ad 0000000000983f04 0000000000983ecd 00007fb91ea756bd 00007fb91ea757b3 00007fb91b640441 00007fb9232f69dd 00007fb923483263 00007fb919f997c6 00007fb919f994f5 00007fb91671327e 00007fb91671215e 00007fb916718395 00007fb916718cbf
1025 07:52:45.905672 (+ 613us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:45.906033 (+ 361us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:45.960330 (+ 54297us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:46.338000 (+377670us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:46.338148 (+ 148us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:45.908600 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:45.908684 (+ 84us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:45.908688 (+ 4us) write_op.cc:435] Acquired schema lock
1025 07:52:45.908691 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:45.908758 (+ 67us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:45.908766 (+ 8us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:45.908770 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:45.908830 (+ 60us) tablet.cc:699] Row locks acquired
1025 07:52:45.908832 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:45.908855 (+ 23us) write_op.cc:270] Start()
1025 07:52:45.908877 (+ 22us) write_op.cc:276] Timestamp: P: 1761378765906334 usec, L: 0
1025 07:52:45.908880 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:45.909000 (+ 120us) log.cc:844] Serialized 433 byte log entry
1025 07:52:45.959772 (+ 50772us) op_driver.cc:464] REPLICATION: finished
1025 07:52:45.959889 (+ 117us) write_op.cc:301] APPLY: starting
1025 07:52:45.959898 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:45.959938 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:45.959957 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:45.960056 (+ 99us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:45.960093 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:45.960102 (+ 9us) write_op.cc:312] APPLY: finished
1025 07:52:45.960159 (+ 57us) log.cc:844] Serialized 29 byte log entry
1025 07:52:45.960198 (+ 39us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.960215 (+ 17us) write_op.cc:454] Released schema lock
1025 07:52:45.960221 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":419119488,"child_traces":[["op",{"apply.queue_time_us":50,"apply.run_cpu_time_us":386,"apply.run_wall_time_us":386,"num_ops":2,"prepare.queue_time_us":2136,"prepare.run_cpu_time_us":479,"prepare.run_wall_time_us":477,"replication_time_us":50872,"spinlock_wait_cycles":39680}]]}
I20251025 07:52:46.339368 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 87a93778620f4c84b3b3f7a3d7d7e87f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:46.339687 455 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 107) took 1216 ms. Trace:
I20251025 07:52:46.339792 743 tablet_service.cc:1505] Processing CreateTablet for tablet 77867309b24c47c89da7296dde89c183 (DEFAULT_TABLE table=default.Table_0c3a15ef44874a679e54197a65dc10c2 [id=6429a7cc57b4477f9c49d8c06d032904]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:46.340006 1167 ts_tablet_manager.cc:1434] T a7ab3150a35b4caca9119a1374d3f2f7 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20251025 07:52:46.340169 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77867309b24c47c89da7296dde89c183. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:46.340306 1167 tablet_bootstrap.cc:492] T 645263a61b654f04985c40d04506c087 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:46.339793 455 rpcz_store.cc:276] 1025 07:52:45.122727 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.726503 (+603776us) service_pool.cc:225] Handling call
1025 07:52:45.902283 (+175780us) spinlock_profiling.cc:243] Waited 169 ms on lock 0x616000041254. stack: 00007fb90f7779f6 00007fb90f7777b0 00007fb90e7533ad 0000000000983f04 0000000000983ecd 00007fb91ea756bd 00007fb91ea757b3 00007fb91b640441 00007fb9232f69dd 00007fb923483263 00007fb919f997c6 00007fb919f994f5 00007fb91671327e 00007fb91671215e 00007fb916718395 00007fb916718cbf
1025 07:52:45.903137 (+ 854us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:45.903600 (+ 463us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:45.959576 (+ 55976us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:46.334767 (+375191us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:46.339617 (+ 4850us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:45.908066 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:45.908152 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:45.908157 (+ 5us) write_op.cc:435] Acquired schema lock
1025 07:52:45.908159 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:45.908229 (+ 70us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:45.908237 (+ 8us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:45.908241 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:45.908299 (+ 58us) tablet.cc:699] Row locks acquired
1025 07:52:45.908303 (+ 4us) write_op.cc:260] PREPARE: finished
1025 07:52:45.908329 (+ 26us) write_op.cc:270] Start()
1025 07:52:45.908352 (+ 23us) write_op.cc:276] Timestamp: P: 1761378765905809 usec, L: 0
1025 07:52:45.908355 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:45.908494 (+ 139us) log.cc:844] Serialized 433 byte log entry
1025 07:52:45.958332 (+ 49838us) op_driver.cc:464] REPLICATION: finished
1025 07:52:45.959097 (+ 765us) write_op.cc:301] APPLY: starting
1025 07:52:45.959107 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:45.959138 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:45.959166 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:45.959317 (+ 151us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:45.959345 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:45.959352 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:45.959408 (+ 56us) log.cc:844] Serialized 29 byte log entry
1025 07:52:45.959444 (+ 36us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.959459 (+ 15us) write_op.cc:454] Released schema lock
1025 07:52:45.959464 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":389075840,"child_traces":[["op",{"apply.queue_time_us":734,"apply.run_cpu_time_us":418,"apply.run_wall_time_us":418,"num_ops":2,"prepare.queue_time_us":4031,"prepare.run_cpu_time_us":493,"prepare.run_wall_time_us":492,"replication_time_us":49962}]]}
I20251025 07:52:46.340641 455 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_735de8c152fe443991bb9516d9a56d8d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:46.341226 455 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_735de8c152fe443991bb9516d9a56d8d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:46.341869 1167 tablet_bootstrap.cc:654] T 645263a61b654f04985c40d04506c087 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:46.341847 448 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:35777 (request call id 195) took 2756 ms (client timeout 1000 ms). Trace:
I20251025 07:52:46.343251 2166 ts_tablet_manager.cc:1950] T 7c7a9d7a4fb3449bb57eafa83230a839 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:46.342667 451 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_f34197ad40904305af4d5568e865489b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:46.343976 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 2aa522477e934c89834026dc480fe7c0 with delete_type TABLET_DATA_DELETED (replaced by tablet 1068f566477e4c6db7dde566fff4c2fb) from {username='slave'} at 127.0.0.1:58412
W20251025 07:52:46.344347 451 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f34197ad40904305af4d5568e865489b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:46.344658 2420 raft_consensus.cc:493] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:46.344762 749 tablet_service.cc:1505] Processing CreateTablet for tablet dafcad0b32de46b9899bd30a87b28be2 (DEFAULT_TABLE table=default.Table_c4fe5959f4f242eeb407926c8f97e810 [id=d0cc97df1dd44dcd8ce2f7881efdd4c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:46.344777 2420 raft_consensus.cc:515] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.345691 2420 leader_election.cc:290] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:46.346235 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 35ba9d23262248cda6fdf03c51be11a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 01f82980559b4662bdc09786ed429ddc) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:46.346328 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 74e2d78c024849b0af5c0bde1f48c36f with delete_type TABLET_DATA_DELETED (replaced by tablet fd265cdbf6a8440393a2b5d8240d1872) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:46.346529 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 5fca357d7e24494fb6af4bd73d98833c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:46.346735 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 02e4c4d6ffb34cecaf17beb5ba197e81 with delete_type TABLET_DATA_DELETED (replaced by tablet 3abfd52327a04ba88625f391a8658fbb) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:46.346735 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b594d4718344aafbdaec34040695589 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:46.346920 754 tablet_service.cc:1552] Processing DeleteTablet for tablet c4610b0c62e741eca28c21f63cc3de81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:58412
W20251025 07:52:46.342761 453 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 110a8e9e4c6c434284c0ae14dd6987f7 (table default.Table_6c5a24747ba14352b0285a41ec2985a4 [id=28bbaad474164a8c8c97a76bb8081791], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 110a8e9e4c6c434284c0ae14dd6987f7 consensus state has no leader
I20251025 07:52:46.349294 2597 raft_consensus.cc:493] T ebc824ec4c60411dab1a2ea4a385148e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:46.349443 2597 raft_consensus.cc:515] T ebc824ec4c60411dab1a2ea4a385148e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.350126 1017 tablet_service.cc:1505] Processing CreateTablet for tablet 77867309b24c47c89da7296dde89c183 (DEFAULT_TABLE table=default.Table_0c3a15ef44874a679e54197a65dc10c2 [id=6429a7cc57b4477f9c49d8c06d032904]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:46.350374 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "141e413d2faa4c67b4c822192e631417" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:46.351078 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77867309b24c47c89da7296dde89c183. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:46.351127 1019 tablet_service.cc:1505] Processing CreateTablet for tablet c5b4b02705ec47b3a6b55de712d2dc8f (DEFAULT_TABLE table=default.Table_9ce0483829814297b8bb466be9cf1ce6 [id=998590d0f3f048819d44ce730549572c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:46.351210 2597 leader_election.cc:290] T ebc824ec4c60411dab1a2ea4a385148e P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
W20251025 07:52:46.351366 712 leader_election.cc:343] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:46.351797 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc824ec4c60411dab1a2ea4a385148e" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:46.352999 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5b4b02705ec47b3a6b55de712d2dc8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:46.327559 2709 raft_consensus.cc:493] T 4a1ec8c2ed7f41abb588f592f62097d1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:46.357420 2709 raft_consensus.cc:3060] T 4a1ec8c2ed7f41abb588f592f62097d1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:46.357513 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77867309b24c47c89da7296dde89c183. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:46.358235 894 tablet_service.cc:1505] Processing CreateTablet for tablet dafcad0b32de46b9899bd30a87b28be2 (DEFAULT_TABLE table=default.Table_c4fe5959f4f242eeb407926c8f97e810 [id=d0cc97df1dd44dcd8ce2f7881efdd4c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:46.358575 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dafcad0b32de46b9899bd30a87b28be2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:46.358578 883 tablet_service.cc:1505] Processing CreateTablet for tablet c5b4b02705ec47b3a6b55de712d2dc8f (DEFAULT_TABLE table=default.Table_9ce0483829814297b8bb466be9cf1ce6 [id=998590d0f3f048819d44ce730549572c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:46.359223 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5b4b02705ec47b3a6b55de712d2dc8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:46.361464 2169 ts_tablet_manager.cc:1929] T a4e4b99be0434baaa75d988281797846 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:46.361549 2169 log.cc:1199] T a4e4b99be0434baaa75d988281797846 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/a4e4b99be0434baaa75d988281797846
I20251025 07:52:46.361850 2169 ts_tablet_manager.cc:1950] T a4e4b99be0434baaa75d988281797846 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:46.362884 2467 raft_consensus.cc:493] T 91728987303b41fd840eb56f3a311c5f P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:46.363003 2467 raft_consensus.cc:515] T 91728987303b41fd840eb56f3a311c5f P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.363834 2467 leader_election.cc:290] T 91728987303b41fd840eb56f3a311c5f P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:46.364523 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "91728987303b41fd840eb56f3a311c5f" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:46.364761 776 raft_consensus.cc:2468] T 91728987303b41fd840eb56f3a311c5f P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:46.365327 982 leader_election.cc:304] T 91728987303b41fd840eb56f3a311c5f P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:46.365602 2467 raft_consensus.cc:2804] T 91728987303b41fd840eb56f3a311c5f P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 07:52:46.365685 2467 raft_consensus.cc:493] T 91728987303b41fd840eb56f3a311c5f P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:46.365760 2467 raft_consensus.cc:3060] T 91728987303b41fd840eb56f3a311c5f P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Advancing to term 2
I20251025 07:52:46.350857 1018 tablet_service.cc:1505] Processing CreateTablet for tablet dafcad0b32de46b9899bd30a87b28be2 (DEFAULT_TABLE table=default.Table_c4fe5959f4f242eeb407926c8f97e810 [id=d0cc97df1dd44dcd8ce2f7881efdd4c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:46.377194 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dafcad0b32de46b9899bd30a87b28be2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:46.378532 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 124) took 7655 ms. Trace:
I20251025 07:52:46.378697 2166 rpcz_store.cc:276] 1025 07:52:38.722666 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.722761 (+ 95us) service_pool.cc:225] Handling call
1025 07:52:46.303665 (+7580904us) ts_tablet_manager.cc:1151] Deleting tablet 7c7a9d7a4fb3449bb57eafa83230a839
1025 07:52:46.303675 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:46.315731 (+ 12056us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.318637 (+ 2906us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.378512 (+ 59875us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7580663}
I20251025 07:52:46.379361 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 7c7a9d7a4fb3449bb57eafa83230a839 (table default.Table_192549a3e67840f3aefc31029cf2ff52 [id=a37be02f8e994f82baf2f4f5ec800926]) successfully deleted
I20251025 07:52:46.379711 2166 tablet_replica.cc:333] T 943ec703e65f4fd3b0c690e08c0bd226 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:46.379963 2166 raft_consensus.cc:2243] T 943ec703e65f4fd3b0c690e08c0bd226 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:46.380318 2166 raft_consensus.cc:2272] T 943ec703e65f4fd3b0c690e08c0bd226 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:46.380892 455 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 109) took 1160 ms. Trace:
I20251025 07:52:46.380996 455 rpcz_store.cc:276] 1025 07:52:45.220159 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:46.340420 (+1120261us) service_pool.cc:225] Handling call
1025 07:52:46.341313 (+ 893us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:46.341790 (+ 477us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:46.379066 (+ 37276us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:46.379075 (+ 9us) write_op.cc:454] Released schema lock
1025 07:52:46.379155 (+ 80us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:46.380723 (+ 1568us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:46.380817 (+ 94us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:46.354855 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:46.355017 (+ 162us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:46.355027 (+ 10us) write_op.cc:435] Acquired schema lock
1025 07:52:46.355031 (+ 4us) tablet.cc:660] Decoding operations
1025 07:52:46.355159 (+ 128us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:46.355180 (+ 21us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:46.355187 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:46.355276 (+ 89us) tablet.cc:699] Row locks acquired
1025 07:52:46.355279 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:46.355318 (+ 39us) write_op.cc:270] Start()
1025 07:52:46.355357 (+ 39us) write_op.cc:276] Timestamp: P: 1761378766355302 usec, L: 0
1025 07:52:46.355362 (+ 5us) op_driver.cc:348] REPLICATION: starting
1025 07:52:46.355637 (+ 275us) log.cc:844] Serialized 433 byte log entry
1025 07:52:46.360660 (+ 5023us) op_driver.cc:464] REPLICATION: finished
1025 07:52:46.363503 (+ 2843us) write_op.cc:301] APPLY: starting
1025 07:52:46.363540 (+ 37us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:46.363586 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:46.363611 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:46.363747 (+ 136us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:46.363785 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:46.363793 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:46.363886 (+ 93us) log.cc:844] Serialized 29 byte log entry
1025 07:52:46.363955 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:46.363978 (+ 23us) write_op.cc:454] Released schema lock
1025 07:52:46.363985 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2779,"apply.run_cpu_time_us":542,"apply.run_wall_time_us":541,"num_ops":2,"prepare.queue_time_us":1519,"prepare.run_cpu_time_us":911,"prepare.run_wall_time_us":911,"raft.queue_time_us":1294,"raft.run_cpu_time_us":754,"raft.run_wall_time_us":7494,"replication_time_us":5261,"spinlock_wait_cycles":426368}]]}
I20251025 07:52:46.380955 2166 ts_tablet_manager.cc:1916] T 943ec703e65f4fd3b0c690e08c0bd226 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:46.341977 448 rpcz_store.cc:269] 1025 07:52:43.585505 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:43.595280 (+ 9775us) service_pool.cc:225] Handling call
1025 07:52:44.595011 (+999731us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.595017 (+ 6us) write_op.cc:454] Released schema lock
1025 07:52:46.341788 (+1746771us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:44.588081 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:44.588201 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:44.588207 (+ 6us) write_op.cc:435] Acquired schema lock
1025 07:52:44.588210 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:44.588323 (+ 113us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:44.588337 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:44.588342 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:44.588405 (+ 63us) tablet.cc:699] Row locks acquired
1025 07:52:44.588408 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:44.588448 (+ 40us) write_op.cc:270] Start()
1025 07:52:44.588485 (+ 37us) write_op.cc:276] Timestamp: P: 1761378764588429 usec, L: 0
1025 07:52:44.588489 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:44.588721 (+ 232us) log.cc:844] Serialized 527 byte log entry
1025 07:52:44.591949 (+ 3228us) op_driver.cc:464] REPLICATION: finished
1025 07:52:44.592052 (+ 103us) write_op.cc:301] APPLY: starting
1025 07:52:44.592067 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:44.592110 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:44.592141 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:44.592192 (+ 51us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:44.592222 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:44.592230 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:44.592300 (+ 70us) log.cc:844] Serialized 23 byte log entry
1025 07:52:44.592362 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:44.592381 (+ 19us) write_op.cc:454] Released schema lock
1025 07:52:44.592388 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":733986,"reported_tablets":247,"child_traces":[["op",{"apply.queue_time_us":56,"apply.run_cpu_time_us":387,"apply.run_wall_time_us":386,"num_ops":1,"prepare.queue_time_us":5916,"prepare.run_cpu_time_us":756,"prepare.run_wall_time_us":756,"raft.queue_time_us":415,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":201,"replication_time_us":3432}]]}
I20251025 07:52:46.383023 448 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_967b302c337a43aeb6158503e776a618"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:46.381281 743 tablet_service.cc:1505] Processing CreateTablet for tablet c5b4b02705ec47b3a6b55de712d2dc8f (DEFAULT_TABLE table=default.Table_9ce0483829814297b8bb466be9cf1ce6 [id=998590d0f3f048819d44ce730549572c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:46.383540 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5b4b02705ec47b3a6b55de712d2dc8f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:46.384661 448 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_967b302c337a43aeb6158503e776a618 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:46.391929 1167 tablet_bootstrap.cc:492] T 645263a61b654f04985c40d04506c087 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:46.392124 1167 ts_tablet_manager.cc:1403] T 645263a61b654f04985c40d04506c087 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.000s
I20251025 07:52:46.393435 1167 raft_consensus.cc:359] T 645263a61b654f04985c40d04506c087 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.393611 1167 raft_consensus.cc:385] T 645263a61b654f04985c40d04506c087 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:46.393682 1167 raft_consensus.cc:740] T 645263a61b654f04985c40d04506c087 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:46.393975 1167 consensus_queue.cc:260] T 645263a61b654f04985c40d04506c087 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.394570 1167 ts_tablet_manager.cc:1434] T 645263a61b654f04985c40d04506c087 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251025 07:52:46.394833 1167 tablet_bootstrap.cc:492] T 3b940fa5ae26427dafbcfd26372754b3 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:46.396385 1167 tablet_bootstrap.cc:654] T 3b940fa5ae26427dafbcfd26372754b3 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:46.397794 850 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:45417, user_credentials={real_user=slave}} blocked reactor thread for 67312.5us
I20251025 07:52:46.400640 1166 tablet_bootstrap.cc:492] T 141e413d2faa4c67b4c822192e631417 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:46.400842 1166 ts_tablet_manager.cc:1403] T 141e413d2faa4c67b4c822192e631417 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.114s user 0.006s sys 0.006s
I20251025 07:52:46.402192 1166 raft_consensus.cc:359] T 141e413d2faa4c67b4c822192e631417 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.402428 1166 raft_consensus.cc:385] T 141e413d2faa4c67b4c822192e631417 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:46.402530 1166 raft_consensus.cc:740] T 141e413d2faa4c67b4c822192e631417 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:46.402827 1166 consensus_queue.cc:260] T 141e413d2faa4c67b4c822192e631417 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.403360 1166 ts_tablet_manager.cc:1434] T 141e413d2faa4c67b4c822192e631417 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:46.403702 1166 tablet_bootstrap.cc:492] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:46.405372 1166 tablet_bootstrap.cc:654] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:46.406355 448 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 110) took 1186 ms. Trace:
I20251025 07:52:46.407754 448 rpcz_store.cc:276] 1025 07:52:45.220295 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:46.382828 (+1162533us) service_pool.cc:225] Handling call
1025 07:52:46.385305 (+ 2477us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:46.385755 (+ 450us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:46.406029 (+ 20274us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:46.406199 (+ 170us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:46.406297 (+ 98us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:46.394011 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:46.394125 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:46.394159 (+ 34us) write_op.cc:435] Acquired schema lock
1025 07:52:46.394162 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:46.394257 (+ 95us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:46.394284 (+ 27us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:46.394288 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:46.394359 (+ 71us) tablet.cc:699] Row locks acquired
1025 07:52:46.394362 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:46.394394 (+ 32us) write_op.cc:270] Start()
1025 07:52:46.394422 (+ 28us) write_op.cc:276] Timestamp: P: 1761378766394378 usec, L: 0
1025 07:52:46.394426 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:46.394586 (+ 160us) log.cc:844] Serialized 433 byte log entry
1025 07:52:46.397632 (+ 3046us) op_driver.cc:464] REPLICATION: finished
1025 07:52:46.398277 (+ 645us) write_op.cc:301] APPLY: starting
1025 07:52:46.398290 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:46.398328 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:46.398360 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:46.398466 (+ 106us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:46.398501 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:46.398507 (+ 6us) write_op.cc:312] APPLY: finished
1025 07:52:46.398577 (+ 70us) log.cc:844] Serialized 29 byte log entry
1025 07:52:46.398646 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:46.398664 (+ 18us) write_op.cc:454] Released schema lock
1025 07:52:46.398671 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:46.398740 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:46.398744 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":593,"apply.run_cpu_time_us":454,"apply.run_wall_time_us":454,"mutex_wait_us":659,"num_ops":2,"prepare.queue_time_us":7756,"prepare.run_cpu_time_us":777,"prepare.run_wall_time_us":1430,"raft.queue_time_us":750,"raft.run_cpu_time_us":315,"raft.run_wall_time_us":1803,"replication_time_us":3168,"spinlock_wait_cycles":447104}]]}
I20251025 07:52:46.407546 451 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 108) took 1282 ms. Trace:
I20251025 07:52:46.410557 448 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_913d48d5976244519b5f452c92b02b14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:46.409649 451 rpcz_store.cc:276] 1025 07:52:45.125445 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:46.339208 (+1213763us) service_pool.cc:225] Handling call
1025 07:52:46.345415 (+ 6207us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:46.345899 (+ 484us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:46.407267 (+ 61368us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:46.407273 (+ 6us) write_op.cc:454] Released schema lock
1025 07:52:46.407335 (+ 62us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:46.407416 (+ 81us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:46.407492 (+ 76us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:46.382263 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:46.382402 (+ 139us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:46.382416 (+ 14us) write_op.cc:435] Acquired schema lock
1025 07:52:46.382420 (+ 4us) tablet.cc:660] Decoding operations
1025 07:52:46.382574 (+ 154us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:46.382591 (+ 17us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:46.382599 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:46.382684 (+ 85us) tablet.cc:699] Row locks acquired
1025 07:52:46.382686 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:46.382728 (+ 42us) write_op.cc:270] Start()
1025 07:52:46.382767 (+ 39us) write_op.cc:276] Timestamp: P: 1761378766382709 usec, L: 0
1025 07:52:46.382773 (+ 6us) op_driver.cc:348] REPLICATION: starting
1025 07:52:46.384066 (+ 1293us) log.cc:844] Serialized 433 byte log entry
1025 07:52:46.387541 (+ 3475us) op_driver.cc:464] REPLICATION: finished
1025 07:52:46.389134 (+ 1593us) write_op.cc:301] APPLY: starting
1025 07:52:46.389152 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:46.389196 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:46.389228 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:46.389325 (+ 97us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:46.389358 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:46.389367 (+ 9us) write_op.cc:312] APPLY: finished
1025 07:52:46.389451 (+ 84us) log.cc:844] Serialized 29 byte log entry
1025 07:52:46.389526 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:46.389550 (+ 24us) write_op.cc:454] Released schema lock
1025 07:52:46.389558 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1540,"apply.run_cpu_time_us":481,"apply.run_wall_time_us":481,"num_ops":2,"prepare.queue_time_us":1787,"prepare.run_cpu_time_us":878,"prepare.run_wall_time_us":1952,"raft.queue_time_us":1529,"raft.run_cpu_time_us":522,"raft.run_wall_time_us":2761,"replication_time_us":3665}]]}
I20251025 07:52:46.412118 2420 raft_consensus.cc:493] T d77704b4fe6c4456a2ba40ffec0c60fc P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:46.412292 2420 raft_consensus.cc:515] T d77704b4fe6c4456a2ba40ffec0c60fc P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.413123 2420 leader_election.cc:290] T d77704b4fe6c4456a2ba40ffec0c60fc P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:46.413784 982 consensus_peers.cc:597] T 6dcab53a020847a1aa27930265ba0205 P db42be7ad3c741a0aa427f27e245993b -> Peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Couldn't send request to peer 23eeffd3de254ce18db5febfde971c6b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6dcab53a020847a1aa27930265ba0205. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:46.413901 883 tablet_service.cc:1505] Processing CreateTablet for tablet 325571291cfa46669cfbb208b7b11da6 (DEFAULT_TABLE table=default.Table_7d1445fd52a84e80b40235d993a1306a [id=e9154ad93a02422482de1fbd4e4970ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:46.414391 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 325571291cfa46669cfbb208b7b11da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:46.415421 749 tablet_service.cc:1505] Processing CreateTablet for tablet 325571291cfa46669cfbb208b7b11da6 (DEFAULT_TABLE table=default.Table_7d1445fd52a84e80b40235d993a1306a [id=e9154ad93a02422482de1fbd4e4970ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:46.416014 451 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_f4321d2891254856a874d7c806c4ad60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:46.416947 875 tablet_service.cc:1505] Processing CreateTablet for tablet 5c9c9b5777274d87a3be873bcc3a97eb (DEFAULT_TABLE table=default.Table_9726fef71da14752a13971fe2ddfa9e3 [id=a152772ca6f546898c0a5d057d35fb11]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:46.417210 448 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_913d48d5976244519b5f452c92b02b14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:46.417270 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c9c9b5777274d87a3be873bcc3a97eb. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:46.418889 451 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4321d2891254856a874d7c806c4ad60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:46.418985 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebc824ec4c60411dab1a2ea4a385148e" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:46.432029 2166 ts_tablet_manager.cc:1929] T 943ec703e65f4fd3b0c690e08c0bd226 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:46.432755 448 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 111) took 1172 ms. Trace:
I20251025 07:52:46.438025 2146 ts_tablet_manager.cc:1929] T d66ad049d5484228a268b0facb00ecf1 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:46.438906 2709 raft_consensus.cc:515] T 4a1ec8c2ed7f41abb588f592f62097d1 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.439561 2709 leader_election.cc:290] T 4a1ec8c2ed7f41abb588f592f62097d1 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:46.441637 2146 log.cc:1199] T d66ad049d5484228a268b0facb00ecf1 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/d66ad049d5484228a268b0facb00ecf1
I20251025 07:52:46.442057 2146 ts_tablet_manager.cc:1950] T d66ad049d5484228a268b0facb00ecf1 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:46.442380 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 132) took 7871 ms. Trace:
I20251025 07:52:46.442584 2146 rpcz_store.cc:276] 1025 07:52:38.570453 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.574297 (+ 3844us) service_pool.cc:225] Handling call
1025 07:52:46.296421 (+7722124us) ts_tablet_manager.cc:1151] Deleting tablet d66ad049d5484228a268b0facb00ecf1
1025 07:52:46.296432 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:46.421406 (+124974us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.437925 (+ 16519us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.442364 (+ 4439us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29456,"tablet-delete.queue_time_us":7528474}
I20251025 07:52:46.443329 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 138) took 7534 ms. Trace:
I20251025 07:52:46.443399 2146 rpcz_store.cc:276] 1025 07:52:38.909215 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.909264 (+ 49us) service_pool.cc:225] Handling call
1025 07:52:46.443163 (+7533899us) ts_tablet_manager.cc:1151] Deleting tablet 062c0f3e51a3487880cea373f8cf1f54
1025 07:52:46.443182 (+ 19us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:46.443314 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7528235}
I20251025 07:52:46.443954 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 139) took 7289 ms. Trace:
I20251025 07:52:46.444026 2146 rpcz_store.cc:276] 1025 07:52:39.154829 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:39.155124 (+ 295us) service_pool.cc:225] Handling call
1025 07:52:46.443855 (+7288731us) ts_tablet_manager.cc:1151] Deleting tablet 3c4970c8a491460e8b4b1c29fde83e4c
1025 07:52:46.443864 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:46.443940 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7241422}
I20251025 07:52:46.444356 2146 tablet_replica.cc:333] T 36c8fcafd5464b4e999276a8062a812a P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:46.451704 983 leader_election.cc:304] T ebc824ec4c60411dab1a2ea4a385148e P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:46.452154 2671 raft_consensus.cc:2749] T ebc824ec4c60411dab1a2ea4a385148e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:46.452895 2467 raft_consensus.cc:515] T 91728987303b41fd840eb56f3a311c5f P db42be7ad3c741a0aa427f27e245993b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.453495 2467 leader_election.cc:290] T 91728987303b41fd840eb56f3a311c5f P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 2 election: Requested vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:46.454360 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dafcad0b32de46b9899bd30a87b28be2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:46.455458 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d77704b4fe6c4456a2ba40ffec0c60fc" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:46.443192 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet d66ad049d5484228a268b0facb00ecf1 (table default.Table_9f52d94037424befbdea5c711068396a [id=55440328803042a28df1b6869d279f59]) successfully deleted
I20251025 07:52:46.456593 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "91728987303b41fd840eb56f3a311c5f" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
W20251025 07:52:46.456894 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 062c0f3e51a3487880cea373f8cf1f54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 062c0f3e51a3487880cea373f8cf1f54 already in progress: creating tablet
W20251025 07:52:46.457129 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 3c4970c8a491460e8b4b1c29fde83e4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c4970c8a491460e8b4b1c29fde83e4c already in progress: creating tablet
I20251025 07:52:46.457346 448 rpcz_store.cc:276] 1025 07:52:45.260364 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:46.410351 (+1149987us) service_pool.cc:225] Handling call
1025 07:52:46.417307 (+ 6956us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:46.417748 (+ 441us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:46.432430 (+ 14682us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:46.432436 (+ 6us) write_op.cc:454] Released schema lock
1025 07:52:46.432508 (+ 72us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:46.432618 (+ 110us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:46.432712 (+ 94us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:46.420992 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:46.421090 (+ 98us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:46.421096 (+ 6us) write_op.cc:435] Acquired schema lock
1025 07:52:46.421098 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:46.421194 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:46.421211 (+ 17us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:46.421219 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:46.421309 (+ 90us) tablet.cc:699] Row locks acquired
1025 07:52:46.421313 (+ 4us) write_op.cc:260] PREPARE: finished
1025 07:52:46.421352 (+ 39us) write_op.cc:270] Start()
1025 07:52:46.421408 (+ 56us) write_op.cc:276] Timestamp: P: 1761378766421335 usec, L: 0
1025 07:52:46.421412 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:46.421595 (+ 183us) log.cc:844] Serialized 433 byte log entry
1025 07:52:46.424710 (+ 3115us) op_driver.cc:464] REPLICATION: finished
1025 07:52:46.425398 (+ 688us) write_op.cc:301] APPLY: starting
1025 07:52:46.425416 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:46.425460 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:46.425483 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:46.425595 (+ 112us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:46.425638 (+ 43us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:46.425650 (+ 12us) write_op.cc:312] APPLY: finished
1025 07:52:46.425732 (+ 82us) log.cc:844] Serialized 29 byte log entry
1025 07:52:46.425799 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:46.425822 (+ 23us) write_op.cc:454] Released schema lock
1025 07:52:46.425829 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":620,"apply.run_cpu_time_us":498,"apply.run_wall_time_us":498,"num_ops":2,"prepare.queue_time_us":2678,"prepare.run_cpu_time_us":734,"prepare.run_wall_time_us":734,"raft.queue_time_us":679,"raft.run_cpu_time_us":713,"raft.run_wall_time_us":9730,"replication_time_us":3271}]]}
I20251025 07:52:46.458933 2146 raft_consensus.cc:2243] T 36c8fcafd5464b4e999276a8062a812a P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:46.447010 455 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_976b27c0f51e4b4b871ac24a5a320b08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:46.458510 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "91728987303b41fd840eb56f3a311c5f" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:46.432368 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 325571291cfa46669cfbb208b7b11da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:46.434995 1167 tablet_bootstrap.cc:492] T 3b940fa5ae26427dafbcfd26372754b3 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
W20251025 07:52:46.456986 712 leader_election.cc:343] T d77704b4fe6c4456a2ba40ffec0c60fc P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:46.461359 1167 ts_tablet_manager.cc:1403] T 3b940fa5ae26427dafbcfd26372754b3 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.067s user 0.003s sys 0.000s
I20251025 07:52:46.453965 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "91728987303b41fd840eb56f3a311c5f" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:46.461715 776 raft_consensus.cc:3060] T 91728987303b41fd840eb56f3a311c5f P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Advancing to term 2
I20251025 07:52:46.464006 754 tablet_service.cc:1505] Processing CreateTablet for tablet d914d324a4464354975014dc4bffd641 (DEFAULT_TABLE table=default.Table_5cce578756b643d8ba30a65ed7e621fb [id=b2e012bc7210498aa09f71851d375f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:46.464350 776 raft_consensus.cc:2468] T 91728987303b41fd840eb56f3a311c5f P 23eeffd3de254ce18db5febfde971c6b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 2.
I20251025 07:52:46.464473 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d914d324a4464354975014dc4bffd641. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:46.464761 982 leader_election.cc:304] T 91728987303b41fd840eb56f3a311c5f P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters: 3aea2869ff814cc7b142012faa274382
I20251025 07:52:46.443789 2420 raft_consensus.cc:493] T e9e8fffb8e984463b5fa2c7dd9e93227 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:46.465219 2420 raft_consensus.cc:515] T e9e8fffb8e984463b5fa2c7dd9e93227 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.465754 2420 leader_election.cc:290] T e9e8fffb8e984463b5fa2c7dd9e93227 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:46.466078 2467 raft_consensus.cc:2804] T 91728987303b41fd840eb56f3a311c5f P db42be7ad3c741a0aa427f27e245993b [term 2 FOLLOWER]: Leader election won for term 2
I20251025 07:52:46.466221 2467 raft_consensus.cc:697] T 91728987303b41fd840eb56f3a311c5f P db42be7ad3c741a0aa427f27e245993b [term 2 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:46.466446 2467 consensus_queue.cc:237] T 91728987303b41fd840eb56f3a311c5f P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.467083 1167 raft_consensus.cc:359] T 3b940fa5ae26427dafbcfd26372754b3 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:46.467226 1167 raft_consensus.cc:385] T 3b940fa5ae26427dafbcfd26372754b3 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:46.467281 1167 raft_consensus.cc:740] T 3b940fa5ae26427dafbcfd26372754b3 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:46.467450 1167 consensus_queue.cc:260] T 3b940fa5ae26427dafbcfd26372754b3 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:46.467927 1167 ts_tablet_manager.cc:1434] T 3b940fa5ae26427dafbcfd26372754b3 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20251025 07:52:46.468215 1167 tablet_bootstrap.cc:492] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:46.469918 1167 tablet_bootstrap.cc:654] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:46.470769 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9e8fffb8e984463b5fa2c7dd9e93227" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:46.471438 712 leader_election.cc:343] T e9e8fffb8e984463b5fa2c7dd9e93227 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:46.473501 1019 tablet_service.cc:1505] Processing CreateTablet for tablet 325571291cfa46669cfbb208b7b11da6 (DEFAULT_TABLE table=default.Table_7d1445fd52a84e80b40235d993a1306a [id=e9154ad93a02422482de1fbd4e4970ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:46.476495 1021 tablet_service.cc:1505] Processing CreateTablet for tablet d914d324a4464354975014dc4bffd641 (DEFAULT_TABLE table=default.Table_5cce578756b643d8ba30a65ed7e621fb [id=b2e012bc7210498aa09f71851d375f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:46.476850 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d914d324a4464354975014dc4bffd641. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:46.482396 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 5c9c9b5777274d87a3be873bcc3a97eb (DEFAULT_TABLE table=default.Table_9726fef71da14752a13971fe2ddfa9e3 [id=a152772ca6f546898c0a5d057d35fb11]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:46.482760 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c9c9b5777274d87a3be873bcc3a97eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:46.486589 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 139) took 7636 ms. Trace:
I20251025 07:52:46.486719 2169 rpcz_store.cc:276] 1025 07:52:38.851187 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.851377 (+ 190us) service_pool.cc:225] Handling call
1025 07:52:46.307960 (+7456583us) ts_tablet_manager.cc:1151] Deleting tablet a4e4b99be0434baaa75d988281797846
1025 07:52:46.307971 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:46.321890 (+ 13919us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.361359 (+ 39469us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.486562 (+125203us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7456415}
I20251025 07:52:46.487259 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 140) took 7586 ms. Trace:
I20251025 07:52:46.487354 2169 rpcz_store.cc:276] 1025 07:52:38.900803 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.900867 (+ 64us) service_pool.cc:225] Handling call
1025 07:52:46.487115 (+7586248us) ts_tablet_manager.cc:1151] Deleting tablet 062c0f3e51a3487880cea373f8cf1f54
1025 07:52:46.487128 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:46.487241 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7586055}
I20251025 07:52:46.490913 2420 raft_consensus.cc:493] T a7ab3150a35b4caca9119a1374d3f2f7 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:46.491052 2420 raft_consensus.cc:515] T a7ab3150a35b4caca9119a1374d3f2f7 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.491600 2420 leader_election.cc:290] T a7ab3150a35b4caca9119a1374d3f2f7 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:46.494004 1167 tablet_bootstrap.cc:492] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:46.494189 1167 ts_tablet_manager.cc:1403] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.000s
I20251025 07:52:46.494493 2146 raft_consensus.cc:2272] T 36c8fcafd5464b4e999276a8062a812a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:46.494448 894 tablet_service.cc:1505] Processing CreateTablet for tablet d914d324a4464354975014dc4bffd641 (DEFAULT_TABLE table=default.Table_5cce578756b643d8ba30a65ed7e621fb [id=b2e012bc7210498aa09f71851d375f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:46.494938 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d914d324a4464354975014dc4bffd641. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:46.495371 2146 ts_tablet_manager.cc:1916] T 36c8fcafd5464b4e999276a8062a812a P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:46.496152 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 325571291cfa46669cfbb208b7b11da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:46.499675 1167 raft_consensus.cc:359] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:46.499850 1167 raft_consensus.cc:385] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:46.499902 1167 raft_consensus.cc:740] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:46.500016 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7ab3150a35b4caca9119a1374d3f2f7" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:46.500154 1167 consensus_queue.cc:260] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:46.500653 1167 ts_tablet_manager.cc:1434] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
W20251025 07:52:46.500762 712 leader_election.cc:343] T a7ab3150a35b4caca9119a1374d3f2f7 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:46.461095 455 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_976b27c0f51e4b4b871ac24a5a320b08 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:46.503607 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet a4e4b99be0434baaa75d988281797846 (table default.Table_dac528b081844b7c86027169c6234c27 [id=b0caa78ee8c84aa8bea4818cf9feb70c]) successfully deleted
W20251025 07:52:46.503922 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 062c0f3e51a3487880cea373f8cf1f54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 062c0f3e51a3487880cea373f8cf1f54 already in progress: creating tablet
I20251025 07:52:46.504537 2698 consensus_queue.cc:1048] T bdaeedbcf35a4aad97f9ef192991978a P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:46.504938 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 142) took 7350 ms. Trace:
I20251025 07:52:46.505030 2169 rpcz_store.cc:276] 1025 07:52:39.154356 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:39.154694 (+ 338us) service_pool.cc:225] Handling call
1025 07:52:46.487598 (+7332904us) ts_tablet_manager.cc:1151] Deleting tablet 3c4970c8a491460e8b4b1c29fde83e4c
1025 07:52:46.487609 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:46.504921 (+ 17312us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7299422}
I20251025 07:52:46.505338 2169 tablet_replica.cc:333] T 36c8fcafd5464b4e999276a8062a812a P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:46.505508 2169 raft_consensus.cc:2243] T 36c8fcafd5464b4e999276a8062a812a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:46.505656 2169 raft_consensus.cc:2272] T 36c8fcafd5464b4e999276a8062a812a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:46.506523 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 3c4970c8a491460e8b4b1c29fde83e4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c4970c8a491460e8b4b1c29fde83e4c already in progress: creating tablet
I20251025 07:52:46.506889 2169 ts_tablet_manager.cc:1916] T 36c8fcafd5464b4e999276a8062a812a P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:46.509369 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:46231, user_credentials={real_user=slave}} blocked reactor thread for 123789us
I20251025 07:52:46.523128 1167 tablet_bootstrap.cc:492] T 437da5ded9b648abb668a8cb4896452c P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:46.524982 1167 tablet_bootstrap.cc:654] T 437da5ded9b648abb668a8cb4896452c P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:46.536844 2728 consensus_queue.cc:1048] T 6dcab53a020847a1aa27930265ba0205 P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:46.541229 2709 raft_consensus.cc:493] T 141e413d2faa4c67b4c822192e631417 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:46.541404 2709 raft_consensus.cc:515] T 141e413d2faa4c67b4c822192e631417 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.542057 2709 leader_election.cc:290] T 141e413d2faa4c67b4c822192e631417 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:46.531845 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a1ec8c2ed7f41abb588f592f62097d1" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:46.561033 2146 ts_tablet_manager.cc:1929] T 36c8fcafd5464b4e999276a8062a812a P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:46.577117 1169 tablet_bootstrap.cc:492] T 2e2e6893a48449969b71ee1c08d1960e P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:46.577376 1169 ts_tablet_manager.cc:1403] T 2e2e6893a48449969b71ee1c08d1960e P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.258s user 0.001s sys 0.004s
I20251025 07:52:46.577405 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "141e413d2faa4c67b4c822192e631417" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:46.578715 1169 raft_consensus.cc:359] T 2e2e6893a48449969b71ee1c08d1960e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.578991 1169 raft_consensus.cc:385] T 2e2e6893a48449969b71ee1c08d1960e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:46.579085 1169 raft_consensus.cc:740] T 2e2e6893a48449969b71ee1c08d1960e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:46.579334 1169 consensus_queue.cc:260] T 2e2e6893a48449969b71ee1c08d1960e P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.579926 1169 ts_tablet_manager.cc:1434] T 2e2e6893a48449969b71ee1c08d1960e P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:46.580214 1169 tablet_bootstrap.cc:492] T 5b6383c99e9e4c90a374da9ccd830a6f P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:46.581908 1169 tablet_bootstrap.cc:654] T 5b6383c99e9e4c90a374da9ccd830a6f P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:46.588181 2169 ts_tablet_manager.cc:1929] T 36c8fcafd5464b4e999276a8062a812a P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251025 07:52:46.448509 453 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:35777 (request call id 196) took 1815 ms (client timeout 1000 ms). Trace:
I20251025 07:52:46.456542 451 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 112) took 1176 ms. Trace:
I20251025 07:52:46.459805 448 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_915572e49894471d8933816077123d6c" } new_table_name: "default.Table_09095c0829ed4d1cb8b763fc695e7fa8" modify_external_catalogs: true
I20251025 07:52:46.598282 451 rpcz_store.cc:276] 1025 07:52:45.279978 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:46.415802 (+1135824us) service_pool.cc:225] Handling call
1025 07:52:46.418997 (+ 3195us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:46.420235 (+ 1238us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:46.456132 (+ 35897us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:46.456139 (+ 7us) write_op.cc:454] Released schema lock
1025 07:52:46.456221 (+ 82us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:46.456385 (+ 164us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:46.456481 (+ 96us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:46.433173 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:46.433281 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:46.433286 (+ 5us) write_op.cc:435] Acquired schema lock
1025 07:52:46.433288 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:46.433367 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:46.433379 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:46.433384 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:46.433450 (+ 66us) tablet.cc:699] Row locks acquired
1025 07:52:46.433453 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:46.433479 (+ 26us) write_op.cc:270] Start()
1025 07:52:46.433505 (+ 26us) write_op.cc:276] Timestamp: P: 1761378766433465 usec, L: 0
1025 07:52:46.433508 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:46.433654 (+ 146us) log.cc:844] Serialized 433 byte log entry
1025 07:52:46.437145 (+ 3491us) op_driver.cc:464] REPLICATION: finished
1025 07:52:46.437826 (+ 681us) write_op.cc:301] APPLY: starting
1025 07:52:46.437844 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:46.437914 (+ 70us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:46.437947 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:46.438048 (+ 101us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:46.438083 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:46.438089 (+ 6us) write_op.cc:312] APPLY: finished
1025 07:52:46.438285 (+ 196us) log.cc:844] Serialized 29 byte log entry
1025 07:52:46.438344 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:46.438365 (+ 21us) write_op.cc:454] Released schema lock
1025 07:52:46.438372 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":618,"apply.run_cpu_time_us":506,"apply.run_wall_time_us":601,"mutex_wait_us":12388,"num_ops":2,"prepare.queue_time_us":12492,"prepare.run_cpu_time_us":581,"prepare.run_wall_time_us":581,"raft.queue_time_us":800,"raft.run_cpu_time_us":508,"raft.run_wall_time_us":5364,"replication_time_us":3610}]]}
I20251025 07:52:46.590812 1046 raft_consensus.cc:1240] T 91728987303b41fd840eb56f3a311c5f P db42be7ad3c741a0aa427f27e245993b [term 2 LEADER]: Rejecting Update request from peer 3aea2869ff814cc7b142012faa274382 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20251025 07:52:46.604715 2169 log.cc:1199] T 36c8fcafd5464b4e999276a8062a812a P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/36c8fcafd5464b4e999276a8062a812a
I20251025 07:52:46.605271 2169 ts_tablet_manager.cc:1950] T 36c8fcafd5464b4e999276a8062a812a P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:46.609092 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 143) took 7424 ms. Trace:
I20251025 07:52:46.609297 2169 rpcz_store.cc:276] 1025 07:52:39.184172 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:39.184263 (+ 91us) service_pool.cc:225] Handling call
1025 07:52:46.505286 (+7321023us) ts_tablet_manager.cc:1151] Deleting tablet 36c8fcafd5464b4e999276a8062a812a
1025 07:52:46.505296 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:46.559258 (+ 53962us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.588056 (+ 28798us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.609066 (+ 21010us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7313647}
I20251025 07:52:46.610379 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 36c8fcafd5464b4e999276a8062a812a (table default.Table_19a3326e4da345c1bbeb2cf083d28e24 [id=11c088570f6940b1895d1691465c10b7]) successfully deleted
I20251025 07:52:46.615983 2467 raft_consensus.cc:493] T 2e2e6893a48449969b71ee1c08d1960e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:46.616184 2467 raft_consensus.cc:515] T 2e2e6893a48449969b71ee1c08d1960e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.616513 1169 tablet_bootstrap.cc:492] T 5b6383c99e9e4c90a374da9ccd830a6f P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:46.616688 1169 ts_tablet_manager.cc:1403] T 5b6383c99e9e4c90a374da9ccd830a6f P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.012s
I20251025 07:52:46.617460 1169 raft_consensus.cc:359] T 5b6383c99e9e4c90a374da9ccd830a6f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.618412 743 tablet_service.cc:1505] Processing CreateTablet for tablet 5c9c9b5777274d87a3be873bcc3a97eb (DEFAULT_TABLE table=default.Table_9726fef71da14752a13971fe2ddfa9e3 [id=a152772ca6f546898c0a5d057d35fb11]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:46.618911 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c9c9b5777274d87a3be873bcc3a97eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:46.619190 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e2e6893a48449969b71ee1c08d1960e" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:46.619407 776 raft_consensus.cc:2468] T 2e2e6893a48449969b71ee1c08d1960e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:46.619017 2467 leader_election.cc:290] T 2e2e6893a48449969b71ee1c08d1960e P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:46.619786 1169 raft_consensus.cc:385] T 5b6383c99e9e4c90a374da9ccd830a6f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:46.619987 1169 raft_consensus.cc:740] T 5b6383c99e9e4c90a374da9ccd830a6f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:46.619855 982 leader_election.cc:304] T 2e2e6893a48449969b71ee1c08d1960e P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:46.622376 1169 consensus_queue.cc:260] T 5b6383c99e9e4c90a374da9ccd830a6f P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.623111 2597 raft_consensus.cc:2804] T 2e2e6893a48449969b71ee1c08d1960e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:46.623186 2597 raft_consensus.cc:493] T 2e2e6893a48449969b71ee1c08d1960e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:46.623247 2597 raft_consensus.cc:3060] T 2e2e6893a48449969b71ee1c08d1960e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:46.625926 2597 raft_consensus.cc:515] T 2e2e6893a48449969b71ee1c08d1960e P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
W20251025 07:52:46.598282 453 rpcz_store.cc:269] 1025 07:52:44.633515 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.685607 (+ 52092us) service_pool.cc:225] Handling call
1025 07:52:46.448456 (+1762849us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":112166,"reported_tablets":250}
I20251025 07:52:46.626819 453 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_87d38832f85e4965bfab1a077990114f" } new_table_name: "default.Table_d11bba62f65740319701b4337a5c255a" modify_external_catalogs: true
I20251025 07:52:46.627862 2597 leader_election.cc:290] T 2e2e6893a48449969b71ee1c08d1960e P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:46.628160 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e2e6893a48449969b71ee1c08d1960e" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:46.628377 776 raft_consensus.cc:3060] T 2e2e6893a48449969b71ee1c08d1960e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:46.631420 776 raft_consensus.cc:2468] T 2e2e6893a48449969b71ee1c08d1960e P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:46.631837 982 leader_election.cc:304] T 2e2e6893a48449969b71ee1c08d1960e P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:46.632112 2597 raft_consensus.cc:2804] T 2e2e6893a48449969b71ee1c08d1960e P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:46.632267 2597 raft_consensus.cc:697] T 2e2e6893a48449969b71ee1c08d1960e P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:46.632525 2597 consensus_queue.cc:237] T 2e2e6893a48449969b71ee1c08d1960e P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.610090 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 146) took 6467 ms. Trace:
I20251025 07:52:46.634325 2169 rpcz_store.cc:276] 1025 07:52:40.142717 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:40.142881 (+ 164us) service_pool.cc:225] Handling call
1025 07:52:46.609918 (+6467037us) ts_tablet_manager.cc:1151] Deleting tablet ff3daed6c9af47e3976b82b4b9a41086
1025 07:52:46.609931 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:46.610076 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6466821}
I20251025 07:52:46.635059 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 158) took 6223 ms. Trace:
W20251025 07:52:46.635185 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet ff3daed6c9af47e3976b82b4b9a41086 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff3daed6c9af47e3976b82b4b9a41086 already in progress: creating tablet
I20251025 07:52:46.635326 2169 rpcz_store.cc:276] 1025 07:52:40.412000 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:40.430196 (+ 18196us) service_pool.cc:225] Handling call
1025 07:52:46.634911 (+6204715us) ts_tablet_manager.cc:1151] Deleting tablet b02f5c0ab0b24288bb50a3ba0e6bd78a
1025 07:52:46.634925 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:46.635042 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6146165}
I20251025 07:52:46.635834 2169 tablet_replica.cc:333] T a77eb82f26ae43b48de1e85693372d34 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
W20251025 07:52:46.635787 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet b02f5c0ab0b24288bb50a3ba0e6bd78a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b02f5c0ab0b24288bb50a3ba0e6bd78a already in progress: creating tablet
I20251025 07:52:46.636166 2169 raft_consensus.cc:2243] T a77eb82f26ae43b48de1e85693372d34 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:46.636361 2169 raft_consensus.cc:2272] T a77eb82f26ae43b48de1e85693372d34 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:46.636831 1169 ts_tablet_manager.cc:1434] T 5b6383c99e9e4c90a374da9ccd830a6f P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20251025 07:52:46.637125 1169 tablet_bootstrap.cc:492] T 4a1ec8c2ed7f41abb588f592f62097d1 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:46.637410 2169 ts_tablet_manager.cc:1916] T a77eb82f26ae43b48de1e85693372d34 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:46.638180 442 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_fa83d0528b4b481989bfc0ca35c6de44 [id=bf77b5ff6f1048dd996d948437f9ff5e], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet d46673a283fe42b78ddeb99fbd2678eb consensus state has no leader
I20251025 07:52:46.561968 2615 raft_consensus.cc:493] T 34c63b69904d499bb0d286941a8303f2 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 07:52:46.563642 850 leader_election.cc:343] T 4a1ec8c2ed7f41abb588f592f62097d1 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:46.639307 1169 tablet_bootstrap.cc:654] T 4a1ec8c2ed7f41abb588f592f62097d1 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:46.639446 850 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:45417, user_credentials={real_user=slave}} blocked reactor thread for 75793.2us
W20251025 07:52:46.640913 850 leader_election.cc:343] T 141e413d2faa4c67b4c822192e631417 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:46.641709 2687 consensus_queue.cc:1059] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:46.642020 2687 consensus_queue.cc:1243] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382 [LEADER]: Peer db42be7ad3c741a0aa427f27e245993b log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20251025 07:52:46.642205 2687 raft_consensus.cc:3055] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Stepping down as leader of term 1
I20251025 07:52:46.642321 2687 raft_consensus.cc:740] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:46.642652 2687 consensus_queue.cc:260] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.643011 2687 raft_consensus.cc:3060] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Advancing to term 2
I20251025 07:52:46.647012 1167 tablet_bootstrap.cc:492] T 437da5ded9b648abb668a8cb4896452c P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:46.647133 1167 ts_tablet_manager.cc:1403] T 437da5ded9b648abb668a8cb4896452c P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.124s user 0.014s sys 0.021s
I20251025 07:52:46.648044 1167 raft_consensus.cc:359] T 437da5ded9b648abb668a8cb4896452c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:46.648255 1167 raft_consensus.cc:385] T 437da5ded9b648abb668a8cb4896452c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:46.648322 1167 raft_consensus.cc:740] T 437da5ded9b648abb668a8cb4896452c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:46.648124 2615 raft_consensus.cc:515] T 34c63b69904d499bb0d286941a8303f2 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.648555 1167 consensus_queue.cc:260] T 437da5ded9b648abb668a8cb4896452c P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:46.650434 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34c63b69904d499bb0d286941a8303f2" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:46.651197 2615 leader_election.cc:290] T 34c63b69904d499bb0d286941a8303f2 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:46.581099 2795 raft_consensus.cc:493] T 7e6544202d4f489d82f22d6073ef240d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:46.596609 1166 tablet_bootstrap.cc:492] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:46.651942 1166 ts_tablet_manager.cc:1403] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.248s user 0.032s sys 0.050s
I20251025 07:52:46.651908 2795 raft_consensus.cc:515] T 7e6544202d4f489d82f22d6073ef240d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.652616 2795 leader_election.cc:290] T 7e6544202d4f489d82f22d6073ef240d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:46.652748 1166 raft_consensus.cc:359] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:46.652915 1166 raft_consensus.cc:385] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:46.652992 1166 raft_consensus.cc:740] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:46.653046 1167 ts_tablet_manager.cc:1434] T 437da5ded9b648abb668a8cb4896452c P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251025 07:52:46.653210 2420 raft_consensus.cc:493] T 29dbaf9149134f548a974a5779f7da67 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:46.653187 1166 consensus_queue.cc:260] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:46.653321 1167 tablet_bootstrap.cc:492] T d46673a283fe42b78ddeb99fbd2678eb P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:46.653311 2420 raft_consensus.cc:515] T 29dbaf9149134f548a974a5779f7da67 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:46.581280 2823 raft_consensus.cc:493] T 6d3c9954693d4e8782ee0128ef066aa9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:46.627255 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e2e6893a48449969b71ee1c08d1960e" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:46.617313 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e2e6893a48449969b71ee1c08d1960e" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:46.653714 902 raft_consensus.cc:2393] T 2e2e6893a48449969b71ee1c08d1960e P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
W20251025 07:52:46.651486 850 leader_election.cc:343] T 34c63b69904d499bb0d286941a8303f2 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:46.653880 2420 leader_election.cc:290] T 29dbaf9149134f548a974a5779f7da67 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:46.653739 1166 ts_tablet_manager.cc:1434] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:46.654353 1166 tablet_bootstrap.cc:492] T a0ea1edeca7347fc844faa5e4be1ae94 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:46.654366 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e6544202d4f489d82f22d6073ef240d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:46.635234 2824 raft_consensus.cc:493] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 07:52:46.654798 850 leader_election.cc:343] T 7e6544202d4f489d82f22d6073ef240d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:46.654871 2824 raft_consensus.cc:515] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.655181 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29dbaf9149134f548a974a5779f7da67" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:46.655450 2824 leader_election.cc:290] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
W20251025 07:52:46.656108 712 leader_election.cc:343] T 29dbaf9149134f548a974a5779f7da67 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:46.656872 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "48ff4ce7846b4e56a37eee9bcb17bd5e" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:46.657050 1167 tablet_bootstrap.cc:654] T d46673a283fe42b78ddeb99fbd2678eb P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:46.657529 850 leader_election.cc:343] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:46.653628 2823 raft_consensus.cc:515] T 6d3c9954693d4e8782ee0128ef066aa9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.658654 2823 leader_election.cc:290] T 6d3c9954693d4e8782ee0128ef066aa9 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:46.659938 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d3c9954693d4e8782ee0128ef066aa9" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:46.660477 850 leader_election.cc:343] T 6d3c9954693d4e8782ee0128ef066aa9 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:46.660800 2146 log.cc:1199] T 36c8fcafd5464b4e999276a8062a812a P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/36c8fcafd5464b4e999276a8062a812a
I20251025 07:52:46.661381 2146 ts_tablet_manager.cc:1950] T 36c8fcafd5464b4e999276a8062a812a P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:46.663106 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 140) took 7487 ms. Trace:
I20251025 07:52:46.663341 2146 rpcz_store.cc:276] 1025 07:52:39.175228 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:39.175412 (+ 184us) service_pool.cc:225] Handling call
1025 07:52:46.444318 (+7268906us) ts_tablet_manager.cc:1151] Deleting tablet 36c8fcafd5464b4e999276a8062a812a
1025 07:52:46.444325 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:46.556683 (+112358us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.560874 (+ 4191us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.663053 (+102179us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7241702}
I20251025 07:52:46.664413 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 36c8fcafd5464b4e999276a8062a812a (table default.Table_19a3326e4da345c1bbeb2cf083d28e24 [id=11c088570f6940b1895d1691465c10b7]) successfully deleted
I20251025 07:52:46.668881 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 143) took 6526 ms. Trace:
I20251025 07:52:46.669235 2146 rpcz_store.cc:276] 1025 07:52:40.142435 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:40.142488 (+ 53us) service_pool.cc:225] Handling call
1025 07:52:46.668488 (+6526000us) ts_tablet_manager.cc:1151] Deleting tablet ff3daed6c9af47e3976b82b4b9a41086
1025 07:52:46.668529 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:46.668805 (+ 276us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6525788}
W20251025 07:52:46.670439 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet ff3daed6c9af47e3976b82b4b9a41086 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff3daed6c9af47e3976b82b4b9a41086 already in progress: creating tablet
W20251025 07:52:46.671824 710 leader_election.cc:343] T a0ea1edeca7347fc844faa5e4be1ae94 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:46.671988 710 leader_election.cc:304] T a0ea1edeca7347fc844faa5e4be1ae94 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:46.672560 2420 raft_consensus.cc:2749] T a0ea1edeca7347fc844faa5e4be1ae94 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:46.673236 710 leader_election.cc:343] T dff7f86ae8924685b26011e504e2247b P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:46.673380 710 leader_election.cc:304] T dff7f86ae8924685b26011e504e2247b P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:46.673822 2420 raft_consensus.cc:2749] T dff7f86ae8924685b26011e504e2247b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:46.675072 710 leader_election.cc:343] T c187a40f8ff14c6a8c184fbeb030ad13 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:46.675197 710 leader_election.cc:304] T c187a40f8ff14c6a8c184fbeb030ad13 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:46.675587 2420 raft_consensus.cc:2749] T c187a40f8ff14c6a8c184fbeb030ad13 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:46.676057 710 leader_election.cc:343] T 70fe9aa2f58d4abea3b3b71066a29653 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 70fe9aa2f58d4abea3b3b71066a29653
I20251025 07:52:46.676174 710 leader_election.cc:304] T 70fe9aa2f58d4abea3b3b71066a29653 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:46.676641 2420 raft_consensus.cc:2749] T 70fe9aa2f58d4abea3b3b71066a29653 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 07:52:46.678013 771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e2e6893a48449969b71ee1c08d1960e" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:46.678249 771 raft_consensus.cc:2393] T 2e2e6893a48449969b71ee1c08d1960e P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3aea2869ff814cc7b142012faa274382 in current term 1: Already voted for candidate db42be7ad3c741a0aa427f27e245993b in this term.
I20251025 07:52:46.679527 771 raft_consensus.cc:1240] T 91728987303b41fd840eb56f3a311c5f P 23eeffd3de254ce18db5febfde971c6b [term 2 FOLLOWER]: Rejecting Update request from peer 3aea2869ff814cc7b142012faa274382 for earlier term 1. Current term is 2. Ops: []
I20251025 07:52:46.680050 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a1ec8c2ed7f41abb588f592f62097d1" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:46.680200 774 raft_consensus.cc:3060] T 4a1ec8c2ed7f41abb588f592f62097d1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:46.681704 2420 raft_consensus.cc:493] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:46.681847 2420 raft_consensus.cc:515] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:46.682611 2420 leader_election.cc:290] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:46.684141 774 raft_consensus.cc:2468] T 4a1ec8c2ed7f41abb588f592f62097d1 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:46.685333 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc4a788dca9d4a8ba40f8fdf1a4b915e" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:46.685981 712 leader_election.cc:343] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:46.432155 2166 log.cc:1199] T 943ec703e65f4fd3b0c690e08c0bd226 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/943ec703e65f4fd3b0c690e08c0bd226
I20251025 07:52:46.690709 2166 ts_tablet_manager.cc:1950] T 943ec703e65f4fd3b0c690e08c0bd226 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:46.691684 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 125) took 7966 ms. Trace:
I20251025 07:52:46.691839 2166 rpcz_store.cc:276] 1025 07:52:38.724746 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.724794 (+ 48us) service_pool.cc:225] Handling call
1025 07:52:46.379667 (+7654873us) ts_tablet_manager.cc:1151] Deleting tablet 943ec703e65f4fd3b0c690e08c0bd226
1025 07:52:46.379676 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:46.425976 (+ 46300us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.431940 (+ 5964us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.691663 (+259723us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7654647}
I20251025 07:52:46.692515 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 943ec703e65f4fd3b0c690e08c0bd226 (table default.Table_db98182d07a54725823623925f51a0cb [id=62cf364c073b47a1939f41dd69222582]) successfully deleted
I20251025 07:52:46.693068 2166 tablet_replica.cc:333] T 70fe9aa2f58d4abea3b3b71066a29653 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:46.693552 2166 raft_consensus.cc:2243] T 70fe9aa2f58d4abea3b3b71066a29653 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:46.693993 2166 raft_consensus.cc:2272] T 70fe9aa2f58d4abea3b3b71066a29653 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:46.703400 1042 raft_consensus.cc:3060] T 5b6383c99e9e4c90a374da9ccd830a6f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:46.709563 2687 consensus_queue.cc:1048] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:46.713203 1167 tablet_bootstrap.cc:492] T d46673a283fe42b78ddeb99fbd2678eb P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:46.713392 1167 ts_tablet_manager.cc:1403] T d46673a283fe42b78ddeb99fbd2678eb P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.060s user 0.003s sys 0.003s
I20251025 07:52:46.716928 1042 raft_consensus.cc:3060] T ebc824ec4c60411dab1a2ea4a385148e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:46.735199 1167 raft_consensus.cc:359] T d46673a283fe42b78ddeb99fbd2678eb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:46.735417 1167 raft_consensus.cc:385] T d46673a283fe42b78ddeb99fbd2678eb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:46.735471 1167 raft_consensus.cc:740] T d46673a283fe42b78ddeb99fbd2678eb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:46.735649 1167 consensus_queue.cc:260] T d46673a283fe42b78ddeb99fbd2678eb P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:46.736101 1167 ts_tablet_manager.cc:1434] T d46673a283fe42b78ddeb99fbd2678eb P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20251025 07:52:46.739518 2169 ts_tablet_manager.cc:1929] T a77eb82f26ae43b48de1e85693372d34 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251025 07:52:46.746277 850 consensus_peers.cc:597] T 36c414bc700341188b7d85c14173c3e0 P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:46.748060 2823 raft_consensus.cc:493] T 570201e7bf05406a97c81dde071d894b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:46.748284 2823 raft_consensus.cc:515] T 570201e7bf05406a97c81dde071d894b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.748986 2823 leader_election.cc:290] T 570201e7bf05406a97c81dde071d894b P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:46.750337 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 155) took 6341 ms. Trace:
I20251025 07:52:46.750483 2146 rpcz_store.cc:276] 1025 07:52:40.408583 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:40.408642 (+ 59us) service_pool.cc:225] Handling call
1025 07:52:46.750158 (+6341516us) ts_tablet_manager.cc:1151] Deleting tablet b02f5c0ab0b24288bb50a3ba0e6bd78a
1025 07:52:46.750171 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:46.750290 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6341289}
W20251025 07:52:46.751179 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet b02f5c0ab0b24288bb50a3ba0e6bd78a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b02f5c0ab0b24288bb50a3ba0e6bd78a already in progress: creating tablet
I20251025 07:52:46.751652 2146 tablet_replica.cc:333] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:46.753340 2823 raft_consensus.cc:493] T c7691c7bdd89437c8db060254c0db25d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:46.753549 2823 raft_consensus.cc:515] T c7691c7bdd89437c8db060254c0db25d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.754705 2823 leader_election.cc:290] T c7691c7bdd89437c8db060254c0db25d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
W20251025 07:52:46.761523 850 consensus_peers.cc:597] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:46.762319 2117 raft_consensus.cc:493] T 37c7938e8c5e41f08506aff6747801c9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:46.762498 2117 raft_consensus.cc:515] T 37c7938e8c5e41f08506aff6747801c9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.763192 2117 leader_election.cc:290] T 37c7938e8c5e41f08506aff6747801c9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:46.767026 2795 raft_consensus.cc:493] T 37c7938e8c5e41f08506aff6747801c9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:46.767132 2795 raft_consensus.cc:515] T 37c7938e8c5e41f08506aff6747801c9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.767606 2795 leader_election.cc:290] T 37c7938e8c5e41f08506aff6747801c9 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:46.780046 455 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 113) took 1082 ms. Trace:
I20251025 07:52:46.780836 447 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f]). Expected version 1 got 0
I20251025 07:52:46.781437 2166 ts_tablet_manager.cc:1916] T 70fe9aa2f58d4abea3b3b71066a29653 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:46.782955 2146 raft_consensus.cc:2243] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:46.783391 907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a33cfb2f2de48d4a5e9e39730d4900e" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:46.794193 2146 pending_rounds.cc:70] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382: Trying to abort 1 pending ops.
I20251025 07:52:46.794267 2146 pending_rounds.cc:77] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214607421888602112 op_type: NO_OP noop_request { }
I20251025 07:52:46.794350 2146 raft_consensus.cc:2889] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 07:52:46.794430 2146 raft_consensus.cc:2272] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:46.783946 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aeba18be0cb4eb9949aeaf6ad73a812" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:46.785310 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d3c9954693d4e8782ee0128ef066aa9" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:46.794860 910 raft_consensus.cc:2468] T 6d3c9954693d4e8782ee0128ef066aa9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:46.794979 2146 ts_tablet_manager.cc:1916] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:46.800495 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskRBHlYt/build/asan/bin/kudu pid 1830
I20251025 07:52:46.807476 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8402fd33ba842e981c6c68787c7c548" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:46.809731 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c501ef5d70e742598c1a902efe47feb1" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:46.812737 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "141e413d2faa4c67b4c822192e631417" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:46.812913 909 raft_consensus.cc:2468] T 141e413d2faa4c67b4c822192e631417 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:46.813989 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d77704b4fe6c4456a2ba40ffec0c60fc" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:46.819523 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9e8fffb8e984463b5fa2c7dd9e93227" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:46.819944 2146 ts_tablet_manager.cc:1929] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:46.820058 2146 log.cc:1199] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/a77eb82f26ae43b48de1e85693372d34
I20251025 07:52:46.820425 2146 ts_tablet_manager.cc:1950] T a77eb82f26ae43b48de1e85693372d34 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:46.820487 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7ab3150a35b4caca9119a1374d3f2f7" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:46.821231 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 156) took 6388 ms. Trace:
I20251025 07:52:46.821311 2146 rpcz_store.cc:276] 1025 07:52:40.433050 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:40.433104 (+ 54us) service_pool.cc:225] Handling call
1025 07:52:46.751568 (+6318464us) ts_tablet_manager.cc:1151] Deleting tablet a77eb82f26ae43b48de1e85693372d34
1025 07:52:46.751578 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:46.816586 (+ 65008us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.819859 (+ 3273us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.821217 (+ 1358us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6318252}
I20251025 07:52:46.824039 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29dbaf9149134f548a974a5779f7da67" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:46.824908 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc4a788dca9d4a8ba40f8fdf1a4b915e" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:46.825712 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37c7938e8c5e41f08506aff6747801c9" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:46.825850 909 raft_consensus.cc:2468] T 37c7938e8c5e41f08506aff6747801c9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:46.828429 2169 log.cc:1199] T a77eb82f26ae43b48de1e85693372d34 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/a77eb82f26ae43b48de1e85693372d34
I20251025 07:52:46.828783 2169 ts_tablet_manager.cc:1950] T a77eb82f26ae43b48de1e85693372d34 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:46.749886 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "570201e7bf05406a97c81dde071d894b" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:46.754633 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7691c7bdd89437c8db060254c0db25d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:46.829574 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 159) took 6396 ms. Trace:
I20251025 07:52:46.764333 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37c7938e8c5e41f08506aff6747801c9" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:46.829658 2169 rpcz_store.cc:276] 1025 07:52:40.433043 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:40.452991 (+ 19948us) service_pool.cc:225] Handling call
1025 07:52:46.635769 (+6182778us) ts_tablet_manager.cc:1151] Deleting tablet a77eb82f26ae43b48de1e85693372d34
1025 07:52:46.635781 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:46.736315 (+100534us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.739420 (+ 3105us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.829534 (+ 90114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6102703}
I20251025 07:52:46.829963 2169 tablet_replica.cc:333] T 6897405bbc1c47a4ab0aa3af80e60e3d P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
W20251025 07:52:46.830024 850 leader_election.cc:343] T 570201e7bf05406a97c81dde071d894b P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:46.830142 2169 raft_consensus.cc:2243] T 6897405bbc1c47a4ab0aa3af80e60e3d P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
W20251025 07:52:46.830242 850 leader_election.cc:343] T c7691c7bdd89437c8db060254c0db25d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:46.830307 2169 raft_consensus.cc:2272] T 6897405bbc1c47a4ab0aa3af80e60e3d P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:46.768252 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37c7938e8c5e41f08506aff6747801c9" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:46.830798 2169 ts_tablet_manager.cc:1916] T 6897405bbc1c47a4ab0aa3af80e60e3d P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:46.831310 712 leader_election.cc:343] T 37c7938e8c5e41f08506aff6747801c9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:46.831964 850 leader_election.cc:343] T 37c7938e8c5e41f08506aff6747801c9 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:46.782512 1169 tablet_bootstrap.cc:492] T 4a1ec8c2ed7f41abb588f592f62097d1 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:46.832269 1169 ts_tablet_manager.cc:1403] T 4a1ec8c2ed7f41abb588f592f62097d1 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.195s user 0.008s sys 0.004s
I20251025 07:52:46.833133 1169 raft_consensus.cc:359] T 4a1ec8c2ed7f41abb588f592f62097d1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.833326 960 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
I20251025 07:52:46.833339 1169 raft_consensus.cc:385] T 4a1ec8c2ed7f41abb588f592f62097d1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:46.833464 1169 raft_consensus.cc:740] T 4a1ec8c2ed7f41abb588f592f62097d1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:46.833725 1169 consensus_queue.cc:260] T 4a1ec8c2ed7f41abb588f592f62097d1 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.834354 1169 ts_tablet_manager.cc:1434] T 4a1ec8c2ed7f41abb588f592f62097d1 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:46.834606 2858 tablet.cc:1722] T 292fe04055954afd8c08ac2636b68191 P db42be7ad3c741a0aa427f27e245993b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 07:52:46.838551 2860 tablet.cc:1722] T 36262ab8c6964dd684c01b4db6219a4f P 3aea2869ff814cc7b142012faa274382: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 07:52:46.834712 1169 tablet_bootstrap.cc:492] T 37c7938e8c5e41f08506aff6747801c9 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:46.840070 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "141e413d2faa4c67b4c822192e631417" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:46.840291 768 raft_consensus.cc:2468] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:46.841243 1169 tablet_bootstrap.cc:654] T 37c7938e8c5e41f08506aff6747801c9 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:46.843863 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34c63b69904d499bb0d286941a8303f2" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:46.844038 768 raft_consensus.cc:2468] T 34c63b69904d499bb0d286941a8303f2 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:46.844208 1166 tablet_bootstrap.cc:654] T a0ea1edeca7347fc844faa5e4be1ae94 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:46.845045 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e6544202d4f489d82f22d6073ef240d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:46.845217 768 raft_consensus.cc:2468] T 7e6544202d4f489d82f22d6073ef240d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:46.849419 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "48ff4ce7846b4e56a37eee9bcb17bd5e" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:46.849606 768 raft_consensus.cc:2468] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:46.850513 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d3c9954693d4e8782ee0128ef066aa9" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:46.850687 768 raft_consensus.cc:2468] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:46.850764 2146 tablet_replica.cc:333] T 6897405bbc1c47a4ab0aa3af80e60e3d P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:46.850992 2146 raft_consensus.cc:2243] T 6897405bbc1c47a4ab0aa3af80e60e3d P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:46.854467 2146 raft_consensus.cc:2272] T 6897405bbc1c47a4ab0aa3af80e60e3d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:46.780354 455 rpcz_store.cc:276] 1025 07:52:45.697961 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:46.446699 (+748738us) service_pool.cc:225] Handling call
1025 07:52:46.614386 (+167687us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:46.614970 (+ 584us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:46.635309 (+ 20339us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:46.779778 (+144469us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:46.779930 (+ 152us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:46.615566 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:46.615716 (+ 150us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:46.615726 (+ 10us) write_op.cc:435] Acquired schema lock
1025 07:52:46.615730 (+ 4us) tablet.cc:660] Decoding operations
1025 07:52:46.615855 (+ 125us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:46.615875 (+ 20us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:46.615882 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:46.616007 (+ 125us) tablet.cc:699] Row locks acquired
1025 07:52:46.616011 (+ 4us) write_op.cc:260] PREPARE: finished
1025 07:52:46.616056 (+ 45us) write_op.cc:270] Start()
1025 07:52:46.616101 (+ 45us) write_op.cc:276] Timestamp: P: 1761378766616038 usec, L: 0
1025 07:52:46.616107 (+ 6us) op_driver.cc:348] REPLICATION: starting
1025 07:52:46.616366 (+ 259us) log.cc:844] Serialized 433 byte log entry
1025 07:52:46.620684 (+ 4318us) op_driver.cc:464] REPLICATION: finished
1025 07:52:46.620783 (+ 99us) write_op.cc:301] APPLY: starting
1025 07:52:46.620798 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:46.620864 (+ 66us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:46.620887 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:46.620998 (+ 111us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:46.621035 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:46.621042 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:46.621115 (+ 73us) log.cc:844] Serialized 29 byte log entry
1025 07:52:46.621178 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:46.621198 (+ 20us) write_op.cc:454] Released schema lock
1025 07:52:46.621204 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:46.621270 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:46.621274 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":18048,"child_traces":[["op",{"apply.queue_time_us":55,"apply.run_cpu_time_us":477,"apply.run_wall_time_us":476,"num_ops":2,"prepare.queue_time_us":104,"prepare.run_cpu_time_us":950,"prepare.run_wall_time_us":951,"raft.queue_time_us":359,"raft.run_cpu_time_us":405,"raft.run_wall_time_us":403,"replication_time_us":4547,"spinlock_wait_cycles":1920}]]}
I20251025 07:52:46.854969 455 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_4d4a21cfe7034318b758a28cc3471403"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:46.855638 455 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d4a21cfe7034318b758a28cc3471403 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:46.855826 2146 ts_tablet_manager.cc:1916] T 6897405bbc1c47a4ab0aa3af80e60e3d P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:46.856719 1167 tablet_bootstrap.cc:492] T 5dab439bc50d4e869548154e1d222ad6 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:46.821703 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet a77eb82f26ae43b48de1e85693372d34 (table default.Table_bcd5afa1072f47898d67504fc72b84c4 [id=3d6f0426d32f481ba8152eaacb298784]) successfully deleted
I20251025 07:52:46.831682 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet a77eb82f26ae43b48de1e85693372d34 (table default.Table_bcd5afa1072f47898d67504fc72b84c4 [id=3d6f0426d32f481ba8152eaacb298784]) successfully deleted
I20251025 07:52:46.860146 1167 tablet_bootstrap.cc:654] T 5dab439bc50d4e869548154e1d222ad6 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:46.861722 822 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
I20251025 07:52:46.867321 455 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 116) took 1121 ms. Trace:
I20251025 07:52:46.867493 455 rpcz_store.cc:276] 1025 07:52:45.745778 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:46.854726 (+1108948us) service_pool.cc:225] Handling call
1025 07:52:46.855783 (+ 1057us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:46.856353 (+ 570us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:46.866983 (+ 10630us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:46.866990 (+ 7us) write_op.cc:454] Released schema lock
1025 07:52:46.867068 (+ 78us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:46.867176 (+ 108us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:46.867261 (+ 85us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:46.858716 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:46.858824 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:46.858831 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:46.858833 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:46.858924 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:46.858939 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:46.858944 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:46.859024 (+ 80us) tablet.cc:699] Row locks acquired
1025 07:52:46.859026 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:46.859066 (+ 40us) write_op.cc:270] Start()
1025 07:52:46.859106 (+ 40us) write_op.cc:276] Timestamp: P: 1761378766859050 usec, L: 0
1025 07:52:46.859111 (+ 5us) op_driver.cc:348] REPLICATION: starting
1025 07:52:46.859322 (+ 211us) log.cc:844] Serialized 433 byte log entry
1025 07:52:46.863431 (+ 4109us) op_driver.cc:464] REPLICATION: finished
1025 07:52:46.863566 (+ 135us) write_op.cc:301] APPLY: starting
1025 07:52:46.863582 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:46.863625 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:46.863649 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:46.863776 (+ 127us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:46.863811 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:46.863819 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:46.863891 (+ 72us) log.cc:844] Serialized 29 byte log entry
1025 07:52:46.863948 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:46.863968 (+ 20us) write_op.cc:454] Released schema lock
1025 07:52:46.863975 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":91,"apply.run_cpu_time_us":456,"apply.run_wall_time_us":455,"num_ops":2,"prepare.queue_time_us":1801,"prepare.run_cpu_time_us":746,"prepare.run_wall_time_us":745,"raft.queue_time_us":413,"raft.run_cpu_time_us":373,"raft.run_wall_time_us":372,"replication_time_us":4292}]]}
I20251025 07:52:46.868858 455 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_ca479cb5c1184c5391b7e1c9d8a08937" } new_table_name: "default.Table_cea16af97a2242968d72900d3a84a86d" modify_external_catalogs: true
I20251025 07:52:46.872344 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "570201e7bf05406a97c81dde071d894b" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:46.872601 768 raft_consensus.cc:2468] T 570201e7bf05406a97c81dde071d894b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:46.874861 2166 ts_tablet_manager.cc:1929] T 70fe9aa2f58d4abea3b3b71066a29653 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:46.874993 2166 log.cc:1199] T 70fe9aa2f58d4abea3b3b71066a29653 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/70fe9aa2f58d4abea3b3b71066a29653
I20251025 07:52:46.875336 2166 ts_tablet_manager.cc:1950] T 70fe9aa2f58d4abea3b3b71066a29653 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:46.875648 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 126) took 8148 ms. Trace:
I20251025 07:52:46.875741 2166 rpcz_store.cc:276] 1025 07:52:38.727200 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.727251 (+ 51us) service_pool.cc:225] Handling call
1025 07:52:46.692969 (+7965718us) ts_tablet_manager.cc:1151] Deleting tablet 70fe9aa2f58d4abea3b3b71066a29653
1025 07:52:46.692983 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:46.816678 (+123695us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.874775 (+ 58097us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.875634 (+ 859us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7965483}
I20251025 07:52:46.876183 2166 tablet_replica.cc:333] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:46.876261 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 70fe9aa2f58d4abea3b3b71066a29653 (table default.Table_517752d265854c7cbd706e2742536620 [id=aa0642dd1fa34cf8bd1fe2070037a55a]) successfully deleted
I20251025 07:52:46.876385 2166 raft_consensus.cc:2243] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:46.879211 767 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7691c7bdd89437c8db060254c0db25d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:46.879388 767 raft_consensus.cc:2468] T c7691c7bdd89437c8db060254c0db25d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:46.879838 2864 tablet.cc:1722] T 292fe04055954afd8c08ac2636b68191 P 23eeffd3de254ce18db5febfde971c6b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 07:52:46.881731 447 catalog_manager.cc:5649] T 30418858bf95438686bdfe64ac733144 P 23eeffd3de254ce18db5febfde971c6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } }
I20251025 07:52:46.886338 447 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7]). Expected version 1 got 0
I20251025 07:52:46.888871 447 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 04d27d54fdb7483b869408af65ff85c9 (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]). Expected version 2 got 0
I20251025 07:52:46.889690 447 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 872cb7512d104827b1f1f186d33a4a44 (table default.Table_d5c65dce8b1b4214ba519d964012bfc0 [id=d4be274b36a845a99a4b96b8846248fd]). Expected version 1 got 0
I20251025 07:52:46.889670 2859 tablet.cc:1722] T 36262ab8c6964dd684c01b4db6219a4f P db42be7ad3c741a0aa427f27e245993b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 07:52:46.892318 1169 tablet_bootstrap.cc:492] T 37c7938e8c5e41f08506aff6747801c9 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:46.892521 1169 ts_tablet_manager.cc:1403] T 37c7938e8c5e41f08506aff6747801c9 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.058s user 0.003s sys 0.000s
I20251025 07:52:46.893426 1169 raft_consensus.cc:359] T 37c7938e8c5e41f08506aff6747801c9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.896438 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37c7938e8c5e41f08506aff6747801c9" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:46.896720 772 raft_consensus.cc:2468] T 37c7938e8c5e41f08506aff6747801c9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:46.899471 2795 raft_consensus.cc:493] T dda0b26e4f1a44928bc4b84f333901c0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:46.899653 2795 raft_consensus.cc:515] T dda0b26e4f1a44928bc4b84f333901c0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.900249 2795 leader_election.cc:290] T dda0b26e4f1a44928bc4b84f333901c0 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:46.900789 2146 ts_tablet_manager.cc:1929] T 6897405bbc1c47a4ab0aa3af80e60e3d P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 07:52:46.901165 2146 log.cc:1199] T 6897405bbc1c47a4ab0aa3af80e60e3d P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/6897405bbc1c47a4ab0aa3af80e60e3d
I20251025 07:52:46.901675 2146 ts_tablet_manager.cc:1950] T 6897405bbc1c47a4ab0aa3af80e60e3d P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:46.902428 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 157) took 6421 ms. Trace:
I20251025 07:52:46.903365 2146 rpcz_store.cc:276] 1025 07:52:40.481141 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:40.481192 (+ 51us) service_pool.cc:225] Handling call
1025 07:52:46.850707 (+6369515us) ts_tablet_manager.cc:1151] Deleting tablet 6897405bbc1c47a4ab0aa3af80e60e3d
1025 07:52:46.850721 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:46.873646 (+ 22925us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.900686 (+ 27040us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.902414 (+ 1728us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6369343}
I20251025 07:52:46.904183 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 6897405bbc1c47a4ab0aa3af80e60e3d (table default.Table_92c0120dd29348618040150dedb14c43 [id=ea892976886644d8aa025ae2edef718d]) successfully deleted
I20251025 07:52:46.904404 2857 tablet.cc:1722] T 292fe04055954afd8c08ac2636b68191 P 3aea2869ff814cc7b142012faa274382: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 07:52:46.904721 2146 tablet_replica.cc:333] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
W20251025 07:52:46.905207 850 consensus_peers.cc:597] T 627a9cfd398346f2aa34c8932a11b5a8 P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:46.906103 2146 raft_consensus.cc:2243] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:46.906548 2146 raft_consensus.cc:2272] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:46.907256 2146 ts_tablet_manager.cc:1916] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:46.907667 447 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet aa9d9ab8073945d28d52fb02213f201a (table default.Table_4791672658f043f6b97b9320a531ebb5 [id=dfd20cf9c3e541c4abe23d59e0aa7f2a]). Expected version 1 got 0
I20251025 07:52:46.907976 447 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]). Expected version 1 got 0
I20251025 07:52:46.908295 447 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]). Expected version 1 got 0
I20251025 07:52:46.908519 447 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_fa83d0528b4b481989bfc0ca35c6de44 [id=bf77b5ff6f1048dd996d948437f9ff5e]). Expected version 3 got 0
I20251025 07:52:46.908941 447 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet bc2d16b33ca8426abbd7bd38339c9f1d (table default.Table_4ea985df4c5b415c9570c64506a98cc4 [id=98d2b7797e33485e8872054aa2eb8ec9]). Expected version 1 got 0
I20251025 07:52:46.909153 447 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e]). Expected version 1 got 0
I20251025 07:52:46.909626 447 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 5dab439bc50d4e869548154e1d222ad6 (table default.Table_fb9def2302784c999bf0f5707c60eb4c [id=45325c188eac4d34bb016d01de091f87]). Expected version 3 got 0
I20251025 07:52:46.909970 447 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet f70091024c034c118fb86578db19a31c (table default.Table_b7da40b96d6c4359bed6f898e8cf6275 [id=e041946c093b4c789ce175c736cc230b]). Expected version 2 got 0
I20251025 07:52:46.910893 447 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]). Expected version 3 got 0
I20251025 07:52:46.911710 2698 consensus_queue.cc:1048] T 11bfe0c87ca941a3846462c8a4512d83 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:46.913712 2166 raft_consensus.cc:2272] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:46.913997 847 leader_election.cc:304] T 2e2e6893a48449969b71ee1c08d1960e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:46.914651 2117 raft_consensus.cc:493] T fd265cdbf6a8440393a2b5d8240d1872 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:46.914857 1166 tablet_bootstrap.cc:492] T a0ea1edeca7347fc844faa5e4be1ae94 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:46.914906 2795 raft_consensus.cc:2749] T 2e2e6893a48449969b71ee1c08d1960e P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251025 07:52:46.914968 1166 ts_tablet_manager.cc:1403] T a0ea1edeca7347fc844faa5e4be1ae94 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.261s user 0.029s sys 0.001s
I20251025 07:52:46.915557 1166 raft_consensus.cc:359] T a0ea1edeca7347fc844faa5e4be1ae94 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.915756 1166 raft_consensus.cc:385] T a0ea1edeca7347fc844faa5e4be1ae94 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:46.915850 1166 raft_consensus.cc:740] T a0ea1edeca7347fc844faa5e4be1ae94 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:46.914014 2663 raft_consensus.cc:493] T d5e5e5bbcf9f43118eea502b69880af5 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db42be7ad3c741a0aa427f27e245993b)
I20251025 07:52:46.916115 1166 consensus_queue.cc:260] T a0ea1edeca7347fc844faa5e4be1ae94 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.916688 1166 ts_tablet_manager.cc:1434] T a0ea1edeca7347fc844faa5e4be1ae94 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:46.917014 1166 tablet_bootstrap.cc:492] T 716fb61312e94a4d9eb8ab8336ca140c P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:46.917187 2795 consensus_queue.cc:1048] T 36c414bc700341188b7d85c14173c3e0 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:46.914772 2117 raft_consensus.cc:515] T fd265cdbf6a8440393a2b5d8240d1872 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:46.917999 2117 leader_election.cc:290] T fd265cdbf6a8440393a2b5d8240d1872 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:46.915082 2420 raft_consensus.cc:493] T fa826b85f0c94150a17d4e417dd3d62f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:46.918267 2420 raft_consensus.cc:515] T fa826b85f0c94150a17d4e417dd3d62f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.918740 2420 leader_election.cc:290] T fa826b85f0c94150a17d4e417dd3d62f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:46.918823 2860 tablet.cc:1722] T 0325d462441349ef81841f1b0cab2a20 P 3aea2869ff814cc7b142012faa274382: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251025 07:52:46.915979 710 leader_election.cc:343] T 71432a6848944a0da06ef9b81f160dc4 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 71432a6848944a0da06ef9b81f160dc4
I20251025 07:52:46.919178 710 leader_election.cc:304] T 71432a6848944a0da06ef9b81f160dc4 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:46.919514 710 leader_election.cc:343] T 6a33cfb2f2de48d4a5e9e39730d4900e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:46.919597 710 leader_election.cc:304] T 6a33cfb2f2de48d4a5e9e39730d4900e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:46.919864 710 leader_election.cc:343] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:46.919934 710 leader_election.cc:304] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:46.916057 2663 raft_consensus.cc:515] T d5e5e5bbcf9f43118eea502b69880af5 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:46.920977 2420 raft_consensus.cc:2749] T 71432a6848944a0da06ef9b81f160dc4 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 07:52:46.921103 2420 raft_consensus.cc:2749] T 6a33cfb2f2de48d4a5e9e39730d4900e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:46.921190 2420 raft_consensus.cc:2749] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:46.894320 2872 tablet.cc:1722] T 0325d462441349ef81841f1b0cab2a20 P db42be7ad3c741a0aa427f27e245993b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 07:52:46.914698 2166 ts_tablet_manager.cc:1916] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:46.923795 1166 tablet_bootstrap.cc:654] T 716fb61312e94a4d9eb8ab8336ca140c P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:46.924052 1169 raft_consensus.cc:385] T 37c7938e8c5e41f08506aff6747801c9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:46.924168 1169 raft_consensus.cc:740] T 37c7938e8c5e41f08506aff6747801c9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:46.924432 1169 consensus_queue.cc:260] T 37c7938e8c5e41f08506aff6747801c9 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.900821 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dda0b26e4f1a44928bc4b84f333901c0" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:46.925029 850 leader_election.cc:343] T dda0b26e4f1a44928bc4b84f333901c0 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:46.926803 847 leader_election.cc:304] T 4a1ec8c2ed7f41abb588f592f62097d1 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:46.927057 2795 raft_consensus.cc:2804] T 4a1ec8c2ed7f41abb588f592f62097d1 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:46.927155 2795 raft_consensus.cc:697] T 4a1ec8c2ed7f41abb588f592f62097d1 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:46.927377 2795 consensus_queue.cc:237] T 4a1ec8c2ed7f41abb588f592f62097d1 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.902786 2875 tablet.cc:1722] T aa9d9ab8073945d28d52fb02213f201a P db42be7ad3c741a0aa427f27e245993b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 07:52:46.920543 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa826b85f0c94150a17d4e417dd3d62f" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:46.931208 712 leader_election.cc:343] T fa826b85f0c94150a17d4e417dd3d62f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:46.933670 2871 tablet.cc:1722] T aa9d9ab8073945d28d52fb02213f201a P 3aea2869ff814cc7b142012faa274382: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 07:52:46.934330 2874 tablet.cc:1722] T bc2d16b33ca8426abbd7bd38339c9f1d P 3aea2869ff814cc7b142012faa274382: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 07:52:46.921438 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd265cdbf6a8440393a2b5d8240d1872" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:46.936262 712 leader_election.cc:343] T fd265cdbf6a8440393a2b5d8240d1872 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:46.943975 2146 ts_tablet_manager.cc:1929] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:46.947414 2698 consensus_queue.cc:1048] T bdaeedbcf35a4aad97f9ef192991978a P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:46.923403 2858 tablet.cc:1722] T bc2d16b33ca8426abbd7bd38339c9f1d P db42be7ad3c741a0aa427f27e245993b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 07:52:46.950307 1169 ts_tablet_manager.cc:1434] T 37c7938e8c5e41f08506aff6747801c9 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.058s user 0.001s sys 0.001s
I20251025 07:52:46.950645 1169 tablet_bootstrap.cc:492] T c7691c7bdd89437c8db060254c0db25d P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:46.952548 1169 tablet_bootstrap.cc:654] T c7691c7bdd89437c8db060254c0db25d P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:46.954347 1167 tablet_bootstrap.cc:492] T 5dab439bc50d4e869548154e1d222ad6 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:46.954563 1167 ts_tablet_manager.cc:1403] T 5dab439bc50d4e869548154e1d222ad6 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.098s user 0.004s sys 0.001s
I20251025 07:52:46.956903 2146 log.cc:1199] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/fa998b9ca6724c4cb1eb68a3eecb0b11
I20251025 07:52:46.958452 2146 ts_tablet_manager.cc:1950] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:46.958935 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 161) took 5589 ms. Trace:
I20251025 07:52:46.958001 1166 tablet_bootstrap.cc:492] T 716fb61312e94a4d9eb8ab8336ca140c P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:46.961822 1166 ts_tablet_manager.cc:1403] T 716fb61312e94a4d9eb8ab8336ca140c P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.002s
I20251025 07:52:46.962622 1166 raft_consensus.cc:359] T 716fb61312e94a4d9eb8ab8336ca140c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.962818 1166 raft_consensus.cc:385] T 716fb61312e94a4d9eb8ab8336ca140c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:46.962910 1166 raft_consensus.cc:740] T 716fb61312e94a4d9eb8ab8336ca140c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:46.963130 1166 consensus_queue.cc:260] T 716fb61312e94a4d9eb8ab8336ca140c P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.963320 2146 rpcz_store.cc:276] 1025 07:52:41.369265 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.369333 (+ 68us) service_pool.cc:225] Handling call
1025 07:52:46.904655 (+5535322us) ts_tablet_manager.cc:1151] Deleting tablet fa998b9ca6724c4cb1eb68a3eecb0b11
1025 07:52:46.904667 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:46.913990 (+ 9323us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.943876 (+ 29886us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.958916 (+ 15040us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5535094}
I20251025 07:52:46.964039 2146 tablet_replica.cc:333] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:46.964257 2146 raft_consensus.cc:2243] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:46.966645 2169 ts_tablet_manager.cc:1929] T 6897405bbc1c47a4ab0aa3af80e60e3d P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 07:52:46.966821 2169 log.cc:1199] T 6897405bbc1c47a4ab0aa3af80e60e3d P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/6897405bbc1c47a4ab0aa3af80e60e3d
I20251025 07:52:46.967376 2663 leader_election.cc:290] T d5e5e5bbcf9f43118eea502b69880af5 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:46.969638 1167 raft_consensus.cc:359] T 5dab439bc50d4e869548154e1d222ad6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:46.969882 1167 raft_consensus.cc:385] T 5dab439bc50d4e869548154e1d222ad6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:46.969980 1167 raft_consensus.cc:740] T 5dab439bc50d4e869548154e1d222ad6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
W20251025 07:52:46.970613 712 leader_election.cc:343] T d5e5e5bbcf9f43118eea502b69880af5 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: d5e5e5bbcf9f43118eea502b69880af5
I20251025 07:52:46.970714 2151 generic_service.cc:196] Checking for leaks (request via RPC)
I20251025 07:52:46.965034 1166 ts_tablet_manager.cc:1434] T 716fb61312e94a4d9eb8ab8336ca140c P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251025 07:52:46.971043 1166 tablet_bootstrap.cc:492] T c11f674385184ae6b235f1db8bf1fb91 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:46.972479 1166 tablet_bootstrap.cc:654] T c11f674385184ae6b235f1db8bf1fb91 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:46.975911 2146 raft_consensus.cc:2272] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:46.976822 2146 ts_tablet_manager.cc:1916] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:46.976850 847 leader_election.cc:304] T 141e413d2faa4c67b4c822192e631417 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:46.977063 2687 raft_consensus.cc:2804] T 141e413d2faa4c67b4c822192e631417 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:46.977116 2687 raft_consensus.cc:493] T 141e413d2faa4c67b4c822192e631417 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:46.977169 2687 raft_consensus.cc:3060] T 141e413d2faa4c67b4c822192e631417 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:46.970276 1167 consensus_queue.cc:260] T 5dab439bc50d4e869548154e1d222ad6 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:46.963809 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet fa998b9ca6724c4cb1eb68a3eecb0b11 (table default.Table_4976d4f5d05540258d144751d129bdb8 [id=8cde66661cf749919839da894ed4dc05]) successfully deleted
I20251025 07:52:46.979846 447 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_87d38832f85e4965bfab1a077990114f [id=ad812862698d421184ebf501ce79b7be]). Expected version 2 got 0
I20251025 07:52:46.979969 2687 raft_consensus.cc:515] T 141e413d2faa4c67b4c822192e631417 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.980114 447 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]). Expected version 1 got 0
I20251025 07:52:46.980345 447 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 22f7db660616457b83b24e947a0b0bae (table default.Table_4a86191163f2408f9bcc795cd8d1bbf6 [id=bb316a7e000748ca902a9bfb494802d7]). Expected version 1 got 0
I20251025 07:52:46.980589 447 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 5fc092c2bb2d41ac8c5b4fc07460c8ec (table default.Table_cd20e202397645f8821dcde13b5c0ce3 [id=630c274339664b9db18ab4ef7c29c16f]). Expected version 2 got 0
I20251025 07:52:46.981287 2687 leader_election.cc:290] T 141e413d2faa4c67b4c822192e631417 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:46.983503 1166 tablet_bootstrap.cc:492] T c11f674385184ae6b235f1db8bf1fb91 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:46.983670 1166 ts_tablet_manager.cc:1403] T c11f674385184ae6b235f1db8bf1fb91 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.001s
I20251025 07:52:46.984345 1166 raft_consensus.cc:359] T c11f674385184ae6b235f1db8bf1fb91 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:46.992422 2864 tablet.cc:1722] T 04d27d54fdb7483b869408af65ff85c9 P 23eeffd3de254ce18db5febfde971c6b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251025 07:52:46.992864 1167 ts_tablet_manager.cc:1434] T 5dab439bc50d4e869548154e1d222ad6 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.038s user 0.002s sys 0.000s
I20251025 07:52:46.993218 1167 tablet_bootstrap.cc:492] T 1c40b2e06d894a06af9e58285892ea8d P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:46.995522 1167 tablet_bootstrap.cc:654] T 1c40b2e06d894a06af9e58285892ea8d P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:46.989431 2146 ts_tablet_manager.cc:1929] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:46.996583 1166 raft_consensus.cc:385] T c11f674385184ae6b235f1db8bf1fb91 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:46.996645 2146 log.cc:1199] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/4f0c4f0e8f014e59be9e46d2afe0be9d
I20251025 07:52:46.996968 2146 ts_tablet_manager.cc:1950] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:46.997231 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 162) took 5581 ms. Trace:
I20251025 07:52:46.997298 2146 rpcz_store.cc:276] 1025 07:52:41.415584 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.415650 (+ 66us) service_pool.cc:225] Handling call
1025 07:52:46.963994 (+5548344us) ts_tablet_manager.cc:1151] Deleting tablet 4f0c4f0e8f014e59be9e46d2afe0be9d
1025 07:52:46.964004 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:46.980192 (+ 16188us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.989332 (+ 9140us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.997215 (+ 7883us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6999,"tablet-delete.queue_time_us":5548158}
I20251025 07:52:46.997655 2146 tablet_replica.cc:333] T 7e6544202d4f489d82f22d6073ef240d P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:46.997822 2146 raft_consensus.cc:2243] T 7e6544202d4f489d82f22d6073ef240d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:46.997850 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 4f0c4f0e8f014e59be9e46d2afe0be9d (table default.Table_c902adc24467448f9b5d4f585cf614f5 [id=56e169f79a37465d82dd1d1c3c285740]) successfully deleted
I20251025 07:52:46.997942 2146 raft_consensus.cc:2272] T 7e6544202d4f489d82f22d6073ef240d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:46.998608 2146 ts_tablet_manager.cc:1916] T 7e6544202d4f489d82f22d6073ef240d P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:46.981118 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "141e413d2faa4c67b4c822192e631417" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:46.992725 2875 tablet.cc:1722] T 04d27d54fdb7483b869408af65ff85c9 P db42be7ad3c741a0aa427f27e245993b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251025 07:52:46.999270 850 leader_election.cc:343] T 141e413d2faa4c67b4c822192e631417 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:46.996615 1169 tablet_bootstrap.cc:492] T c7691c7bdd89437c8db060254c0db25d P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:47.003007 1169 ts_tablet_manager.cc:1403] T c7691c7bdd89437c8db060254c0db25d P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.000s
I20251025 07:52:47.004401 1169 raft_consensus.cc:359] T c7691c7bdd89437c8db060254c0db25d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.004673 1169 raft_consensus.cc:385] T c7691c7bdd89437c8db060254c0db25d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.004755 2146 ts_tablet_manager.cc:1929] T 7e6544202d4f489d82f22d6073ef240d P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:47.004787 1169 raft_consensus.cc:740] T c7691c7bdd89437c8db060254c0db25d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.005064 1169 consensus_queue.cc:260] T c7691c7bdd89437c8db060254c0db25d P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.005558 1169 ts_tablet_manager.cc:1434] T c7691c7bdd89437c8db060254c0db25d P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:47.005873 1169 tablet_bootstrap.cc:492] T 36c414bc700341188b7d85c14173c3e0 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:47.022235 447 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 0325d462441349ef81841f1b0cab2a20 (table default.Table_750ecb905839434fa9a1aa72f87394be [id=9ce3ba77c56a4ef09cb3b40ea0e488f6]). Expected version 1 got 0
I20251025 07:52:47.022725 447 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 3a0f50c8000144a4acb5219484b01800 (table default.Table_c62e0dc1ae504f9bb912013506ff3031 [id=6f395892b09240fea02482b3199cfb5e]). Expected version 1 got 0
I20251025 07:52:47.023154 447 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525]). Expected version 1 got 0
I20251025 07:52:47.023383 447 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_67b99749f3bf43228b40d8b9807e69b4 [id=fc8180c09d7b4e74b2e161f7acb8fe0e]). Expected version 2 got 0
I20251025 07:52:47.023679 447 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 8707b93d491e4e4386d148087a2f0870 (table default.Table_799dcb1cef2f4fe6b9b8f17f9f1026c5 [id=59840a567e5244e8a9024cbad42471e4]). Expected version 1 got 0
I20251025 07:52:47.024238 447 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236]). Expected version 1 got 0
I20251025 07:52:47.024569 447 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5]). Expected version 1 got 0
I20251025 07:52:47.024984 447 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3]). Expected version 1 got 0
I20251025 07:52:47.025178 447 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c]). Expected version 1 got 0
I20251025 07:52:47.025499 447 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 755f0d6c87044e20a8020f8410940123 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 4 got 0
I20251025 07:52:47.026154 447 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet b56209e4934e4407a8b4518ca4af0d46 (table default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de]). Expected version 1 got 0
I20251025 07:52:47.026410 447 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 830fdbc6812e416690d0db42a73af503 (table default.Table_92824b3f2a774564b3aea04aaa25d204 [id=fb21a45b86ab446e8c3dae4512a9598f]). Expected version 1 got 0
I20251025 07:52:47.003520 2169 ts_tablet_manager.cc:1950] T 6897405bbc1c47a4ab0aa3af80e60e3d P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:47.031049 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 161) took 6549 ms. Trace:
I20251025 07:52:47.031165 2169 rpcz_store.cc:276] 1025 07:52:40.481186 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:40.481273 (+ 87us) service_pool.cc:225] Handling call
1025 07:52:46.829917 (+6348644us) ts_tablet_manager.cc:1151] Deleting tablet 6897405bbc1c47a4ab0aa3af80e60e3d
1025 07:52:46.829926 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:46.833797 (+ 3871us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:46.966540 (+132743us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.031025 (+ 64485us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6293630}
I20251025 07:52:47.031637 2169 tablet_replica.cc:333] T fa998b9ca6724c4cb1eb68a3eecb0b11 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:47.016691 2795 raft_consensus.cc:493] T 9a8e2658a81e436298fe2190726080d4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:47.031821 2146 log.cc:1199] T 7e6544202d4f489d82f22d6073ef240d P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/7e6544202d4f489d82f22d6073ef240d
I20251025 07:52:47.031878 2795 raft_consensus.cc:515] T 9a8e2658a81e436298fe2190726080d4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.032135 2146 ts_tablet_manager.cc:1950] T 7e6544202d4f489d82f22d6073ef240d P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:47.032516 2795 leader_election.cc:290] T 9a8e2658a81e436298fe2190726080d4 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:47.032536 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 164) took 5616 ms. Trace:
I20251025 07:52:47.032608 2146 rpcz_store.cc:276] 1025 07:52:41.415809 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.415920 (+ 111us) service_pool.cc:225] Handling call
1025 07:52:46.997595 (+5581675us) ts_tablet_manager.cc:1151] Deleting tablet 7e6544202d4f489d82f22d6073ef240d
1025 07:52:46.997604 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.001637 (+ 4033us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.004654 (+ 3017us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.032519 (+ 27865us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5581520}
I20251025 07:52:46.994521 847 leader_election.cc:304] T 34c63b69904d499bb0d286941a8303f2 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.032917 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a8e2658a81e436298fe2190726080d4" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:47.033260 847 leader_election.cc:304] T 7e6544202d4f489d82f22d6073ef240d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.033324 2795 raft_consensus.cc:2804] T 34c63b69904d499bb0d286941a8303f2 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20251025 07:52:47.033373 850 leader_election.cc:343] T 9a8e2658a81e436298fe2190726080d4 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.033411 2795 raft_consensus.cc:493] T 34c63b69904d499bb0d286941a8303f2 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251025 07:52:47.033442 847 raft_consensus.cc:2686] T 7e6544202d4f489d82f22d6073ef240d P 3aea2869ff814cc7b142012faa274382: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251025 07:52:47.033469 2795 raft_consensus.cc:3060] T 34c63b69904d499bb0d286941a8303f2 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.033505 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 165) took 5617 ms. Trace:
I20251025 07:52:47.033563 2146 rpcz_store.cc:276] 1025 07:52:41.415937 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.416022 (+ 85us) service_pool.cc:225] Handling call
1025 07:52:47.033375 (+5617353us) ts_tablet_manager.cc:1151] Deleting tablet ffe7c8e1d6c44872b7b65e512764a56a
1025 07:52:47.033385 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.033491 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5617213}
I20251025 07:52:47.033818 2146 tablet_replica.cc:333] T 80fe3255ed614270bd8352a2853fbc43 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:47.033973 2146 raft_consensus.cc:2243] T 80fe3255ed614270bd8352a2853fbc43 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:47.034214 2146 raft_consensus.cc:2272] T 80fe3255ed614270bd8352a2853fbc43 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:46.996649 1166 raft_consensus.cc:740] T c11f674385184ae6b235f1db8bf1fb91 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.034258 2687 consensus_queue.cc:1048] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:47.034433 1166 consensus_queue.cc:260] T c11f674385184ae6b235f1db8bf1fb91 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.034750 2146 ts_tablet_manager.cc:1916] T 80fe3255ed614270bd8352a2853fbc43 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.034276 847 leader_election.cc:304] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.034925 1166 ts_tablet_manager.cc:1434] T c11f674385184ae6b235f1db8bf1fb91 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.051s user 0.001s sys 0.001s
I20251025 07:52:47.035157 1166 tablet_bootstrap.cc:492] T 2b594d4718344aafbdaec34040695589 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:47.035336 2698 raft_consensus.cc:2804] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:47.035395 2698 raft_consensus.cc:493] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:47.035447 2698 raft_consensus.cc:3060] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.036629 1166 tablet_bootstrap.cc:654] T 2b594d4718344aafbdaec34040695589 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.035115 847 leader_election.cc:304] T 6d3c9954693d4e8782ee0128ef066aa9 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.037907 2826 raft_consensus.cc:2804] T 6d3c9954693d4e8782ee0128ef066aa9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:47.037979 2826 raft_consensus.cc:493] T 6d3c9954693d4e8782ee0128ef066aa9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:47.038019 847 leader_election.cc:304] T 570201e7bf05406a97c81dde071d894b P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.038064 2826 raft_consensus.cc:3060] T 6d3c9954693d4e8782ee0128ef066aa9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.038985 2824 raft_consensus.cc:2804] T 570201e7bf05406a97c81dde071d894b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:47.039058 2824 raft_consensus.cc:493] T 570201e7bf05406a97c81dde071d894b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:47.039112 2824 raft_consensus.cc:3060] T 570201e7bf05406a97c81dde071d894b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.040750 2826 raft_consensus.cc:515] T 6d3c9954693d4e8782ee0128ef066aa9 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.041236 2826 leader_election.cc:290] T 6d3c9954693d4e8782ee0128ef066aa9 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:47.041507 2824 raft_consensus.cc:515] T 570201e7bf05406a97c81dde071d894b P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.041883 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d3c9954693d4e8782ee0128ef066aa9" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:47.038079 2698 raft_consensus.cc:515] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.042037 2824 leader_election.cc:290] T 570201e7bf05406a97c81dde071d894b P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:47.042426 2698 leader_election.cc:290] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:47.042932 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "570201e7bf05406a97c81dde071d894b" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:47.042951 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "48ff4ce7846b4e56a37eee9bcb17bd5e" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:47.043282 850 leader_election.cc:343] T 6d3c9954693d4e8782ee0128ef066aa9 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:47.043759 850 leader_election.cc:343] T 570201e7bf05406a97c81dde071d894b P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:47.043941 850 leader_election.cc:343] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.043918 2795 raft_consensus.cc:515] T 34c63b69904d499bb0d286941a8303f2 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.044416 2795 leader_election.cc:290] T 34c63b69904d499bb0d286941a8303f2 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:47.044730 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34c63b69904d499bb0d286941a8303f2" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:47.045181 850 leader_election.cc:343] T 34c63b69904d499bb0d286941a8303f2 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:46.995762 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:43563, user_credentials={real_user=slave}} blocked reactor thread for 75901.3us
I20251025 07:52:47.047205 710 leader_election.cc:304] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:47.047609 710 leader_election.cc:343] T e8402fd33ba842e981c6c68787c7c548 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.047695 710 leader_election.cc:304] T e8402fd33ba842e981c6c68787c7c548 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:47.048051 710 leader_election.cc:343] T c501ef5d70e742598c1a902efe47feb1 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.048126 710 leader_election.cc:304] T c501ef5d70e742598c1a902efe47feb1 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.048460 710 leader_election.cc:304] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.048801 2117 raft_consensus.cc:2804] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:47.048868 2117 raft_consensus.cc:493] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:47.048921 2117 raft_consensus.cc:3060] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.051599 2117 raft_consensus.cc:515] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.052310 2663 raft_consensus.cc:2749] T e8402fd33ba842e981c6c68787c7c548 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:47.052453 2663 raft_consensus.cc:2749] T c501ef5d70e742598c1a902efe47feb1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:47.052659 2663 raft_consensus.cc:2804] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:47.052713 2663 raft_consensus.cc:493] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:47.052784 2663 raft_consensus.cc:3060] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.053567 2169 raft_consensus.cc:2243] T fa998b9ca6724c4cb1eb68a3eecb0b11 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:47.053923 2169 raft_consensus.cc:2272] T fa998b9ca6724c4cb1eb68a3eecb0b11 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.054401 2117 leader_election.cc:290] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:47.055056 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d3c9954693d4e8782ee0128ef066aa9" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:47.055564 712 leader_election.cc:343] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.056354 2169 ts_tablet_manager.cc:1916] T fa998b9ca6724c4cb1eb68a3eecb0b11 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.057674 2663 raft_consensus.cc:515] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.058611 1044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "141e413d2faa4c67b4c822192e631417" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:47.058852 2663 leader_election.cc:290] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:47.059289 712 leader_election.cc:343] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.070828 1169 tablet_bootstrap.cc:654] T 36c414bc700341188b7d85c14173c3e0 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.032366 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 6897405bbc1c47a4ab0aa3af80e60e3d (table default.Table_92c0120dd29348618040150dedb14c43 [id=ea892976886644d8aa025ae2edef718d]) successfully deleted
I20251025 07:52:47.072495 447 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84]). Expected version 1 got 0
I20251025 07:52:47.073385 447 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3]). Expected version 2 got 0
I20251025 07:52:47.074339 447 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 36262ab8c6964dd684c01b4db6219a4f (table default.Table_76158f8cf82e4cdb9a90580e46aaea48 [id=f249f30497bc450c887d659721469de8]). Expected version 1 got 0
I20251025 07:52:47.074671 447 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 288e543bb279410c91c01dd1623e110f (table default.Table_6e242ba55b194c23a6701f4b064fdbe5 [id=4c34b64b299f4162a53866cf01c5298f]). Expected version 1 got 0
I20251025 07:52:47.075278 447 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 292fe04055954afd8c08ac2636b68191 (table default.Table_9fa69b10b5e842639fc0b0a58cd9b6a5 [id=a253ce0253bd4da4814741c482193d8f]). Expected version 1 got 0
I20251025 07:52:47.033110 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 7e6544202d4f489d82f22d6073ef240d (table default.Table_8da73d66704143de8a4b4480c28fd1d9 [id=93fcfeca6dbf48baa3691b7939d0e4ec]) successfully deleted
I20251025 07:52:47.078336 847 leader_election.cc:304] T c7691c7bdd89437c8db060254c0db25d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:47.078409 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:43563, user_credentials={real_user=slave}} blocked reactor thread for 45288.8us
I20251025 07:52:47.078857 2795 raft_consensus.cc:2804] T c7691c7bdd89437c8db060254c0db25d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:47.078948 2795 raft_consensus.cc:493] T c7691c7bdd89437c8db060254c0db25d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:47.079023 2795 raft_consensus.cc:3060] T c7691c7bdd89437c8db060254c0db25d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:47.078995 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet ffe7c8e1d6c44872b7b65e512764a56a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ffe7c8e1d6c44872b7b65e512764a56a already in progress: creating tablet
I20251025 07:52:47.078982 847 leader_election.cc:304] T 37c7938e8c5e41f08506aff6747801c9 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.079419 2823 raft_consensus.cc:2804] T 37c7938e8c5e41f08506aff6747801c9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:47.079490 2823 raft_consensus.cc:493] T 37c7938e8c5e41f08506aff6747801c9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:47.079557 2823 raft_consensus.cc:3060] T 37c7938e8c5e41f08506aff6747801c9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.081683 2795 raft_consensus.cc:515] T c7691c7bdd89437c8db060254c0db25d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.081974 2823 raft_consensus.cc:515] T 37c7938e8c5e41f08506aff6747801c9 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.082266 2795 leader_election.cc:290] T c7691c7bdd89437c8db060254c0db25d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:47.082517 2823 leader_election.cc:290] T 37c7938e8c5e41f08506aff6747801c9 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:47.083237 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7691c7bdd89437c8db060254c0db25d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:47.083424 1046 raft_consensus.cc:3060] T c7691c7bdd89437c8db060254c0db25d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.083237 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37c7938e8c5e41f08506aff6747801c9" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:47.083652 1047 raft_consensus.cc:3060] T 37c7938e8c5e41f08506aff6747801c9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.086313 1046 raft_consensus.cc:2468] T c7691c7bdd89437c8db060254c0db25d P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:47.086776 850 leader_election.cc:304] T c7691c7bdd89437c8db060254c0db25d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:47.087018 2698 raft_consensus.cc:2804] T c7691c7bdd89437c8db060254c0db25d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:47.086966 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dda0b26e4f1a44928bc4b84f333901c0" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:47.087127 2698 raft_consensus.cc:697] T c7691c7bdd89437c8db060254c0db25d P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:47.087358 2698 consensus_queue.cc:237] T c7691c7bdd89437c8db060254c0db25d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.091961 767 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a8e2658a81e436298fe2190726080d4" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:47.092247 767 raft_consensus.cc:2468] T 9a8e2658a81e436298fe2190726080d4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:47.092567 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "48ff4ce7846b4e56a37eee9bcb17bd5e" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:47.092680 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d3c9954693d4e8782ee0128ef066aa9" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:47.092799 766 raft_consensus.cc:3060] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.092898 769 raft_consensus.cc:2393] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3aea2869ff814cc7b142012faa274382 in current term 1: Already voted for candidate 23eeffd3de254ce18db5febfde971c6b in this term.
I20251025 07:52:47.093155 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "570201e7bf05406a97c81dde071d894b" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:47.093298 776 raft_consensus.cc:3060] T 570201e7bf05406a97c81dde071d894b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.096380 766 raft_consensus.cc:2468] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:47.096581 776 raft_consensus.cc:2468] T 570201e7bf05406a97c81dde071d894b P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:47.098992 767 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34c63b69904d499bb0d286941a8303f2" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:47.099117 767 raft_consensus.cc:3060] T 34c63b69904d499bb0d286941a8303f2 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.099359 1169 tablet_bootstrap.cc:492] T 36c414bc700341188b7d85c14173c3e0 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:47.099491 1169 ts_tablet_manager.cc:1403] T 36c414bc700341188b7d85c14173c3e0 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.094s user 0.002s sys 0.002s
I20251025 07:52:47.100245 1169 raft_consensus.cc:359] T 36c414bc700341188b7d85c14173c3e0 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.100391 1169 raft_consensus.cc:385] T 36c414bc700341188b7d85c14173c3e0 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.100432 1169 raft_consensus.cc:740] T 36c414bc700341188b7d85c14173c3e0 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.101078 1047 raft_consensus.cc:2468] T 37c7938e8c5e41f08506aff6747801c9 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:47.101557 850 leader_election.cc:304] T 37c7938e8c5e41f08506aff6747801c9 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:47.101840 2886 raft_consensus.cc:2804] T 37c7938e8c5e41f08506aff6747801c9 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:47.101912 767 raft_consensus.cc:2468] T 34c63b69904d499bb0d286941a8303f2 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:47.102124 2795 raft_consensus.cc:493] T 682c1e22d7764dce862d970b33b88cb3 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:47.102177 2886 raft_consensus.cc:697] T 37c7938e8c5e41f08506aff6747801c9 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:47.102298 2795 raft_consensus.cc:515] T 682c1e22d7764dce862d970b33b88cb3 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.102546 2886 consensus_queue.cc:237] T 37c7938e8c5e41f08506aff6747801c9 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.103574 2795 leader_election.cc:290] T 682c1e22d7764dce862d970b33b88cb3 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:47.105675 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "682c1e22d7764dce862d970b33b88cb3" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:47.106264 850 leader_election.cc:343] T 682c1e22d7764dce862d970b33b88cb3 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.110612 2864 tablet.cc:1722] T 36262ab8c6964dd684c01b4db6219a4f P 23eeffd3de254ce18db5febfde971c6b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 07:52:47.111013 767 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "682c1e22d7764dce862d970b33b88cb3" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:47.111281 767 raft_consensus.cc:2468] T 682c1e22d7764dce862d970b33b88cb3 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:47.112246 847 leader_election.cc:304] T 9a8e2658a81e436298fe2190726080d4 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.113188 2900 tablet.cc:1722] T 0325d462441349ef81841f1b0cab2a20 P 23eeffd3de254ce18db5febfde971c6b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 07:52:47.113816 2795 raft_consensus.cc:2804] T 9a8e2658a81e436298fe2190726080d4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:47.113884 2795 raft_consensus.cc:493] T 9a8e2658a81e436298fe2190726080d4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:47.113935 2795 raft_consensus.cc:3060] T 9a8e2658a81e436298fe2190726080d4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.088387 764 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "141e413d2faa4c67b4c822192e631417" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:47.114912 764 raft_consensus.cc:2393] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3aea2869ff814cc7b142012faa274382 in current term 1: Already voted for candidate 23eeffd3de254ce18db5febfde971c6b in this term.
I20251025 07:52:47.096740 762 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7691c7bdd89437c8db060254c0db25d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:47.115268 762 raft_consensus.cc:3060] T c7691c7bdd89437c8db060254c0db25d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.115797 2864 tablet.cc:1722] T aa9d9ab8073945d28d52fb02213f201a P 23eeffd3de254ce18db5febfde971c6b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 07:52:47.116840 2795 raft_consensus.cc:515] T 9a8e2658a81e436298fe2190726080d4 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.117404 2795 leader_election.cc:290] T 9a8e2658a81e436298fe2190726080d4 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:47.117961 762 raft_consensus.cc:2468] T c7691c7bdd89437c8db060254c0db25d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:47.117921 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a8e2658a81e436298fe2190726080d4" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:47.097373 761 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37c7938e8c5e41f08506aff6747801c9" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:47.118299 761 raft_consensus.cc:3060] T 37c7938e8c5e41f08506aff6747801c9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:47.118487 850 leader_election.cc:343] T 9a8e2658a81e436298fe2190726080d4 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.121033 761 raft_consensus.cc:2468] T 37c7938e8c5e41f08506aff6747801c9 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:47.122280 1169 consensus_queue.cc:260] T 36c414bc700341188b7d85c14173c3e0 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.122836 1169 ts_tablet_manager.cc:1434] T 36c414bc700341188b7d85c14173c3e0 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20251025 07:52:47.123126 1169 tablet_bootstrap.cc:492] T c0c6324cbae34dcebade3ef9f89a7150 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:47.124643 1169 tablet_bootstrap.cc:654] T c0c6324cbae34dcebade3ef9f89a7150 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.125665 847 leader_election.cc:304] T 6d3c9954693d4e8782ee0128ef066aa9 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.126076 847 leader_election.cc:304] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.127357 2698 raft_consensus.cc:2749] T 6d3c9954693d4e8782ee0128ef066aa9 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251025 07:52:47.127511 2698 raft_consensus.cc:2804] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:47.127616 2698 raft_consensus.cc:697] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:47.127846 2698 consensus_queue.cc:237] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.046903 2166 ts_tablet_manager.cc:1929] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.130700 2166 log.cc:1199] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/f4578ce69a0d4322944e5ce7c8a59831
I20251025 07:52:47.132433 2166 ts_tablet_manager.cc:1950] T f4578ce69a0d4322944e5ce7c8a59831 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
W20251025 07:52:47.133706 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:43563, user_credentials={real_user=slave}} blocked reactor thread for 85259.5us
W20251025 07:52:47.135283 710 leader_election.cc:343] T d77704b4fe6c4456a2ba40ffec0c60fc P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.135597 710 leader_election.cc:304] T d77704b4fe6c4456a2ba40ffec0c60fc P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.136554 2663 raft_consensus.cc:2749] T d77704b4fe6c4456a2ba40ffec0c60fc P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:47.137881 710 leader_election.cc:343] T e9e8fffb8e984463b5fa2c7dd9e93227 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.138060 710 leader_election.cc:304] T e9e8fffb8e984463b5fa2c7dd9e93227 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.138345 847 leader_election.cc:304] T 570201e7bf05406a97c81dde071d894b P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.138574 2710 raft_consensus.cc:2804] T 570201e7bf05406a97c81dde071d894b P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:47.138676 2710 raft_consensus.cc:697] T 570201e7bf05406a97c81dde071d894b P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:47.138693 847 leader_election.cc:304] T 34c63b69904d499bb0d286941a8303f2 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.138986 847 leader_election.cc:304] T 682c1e22d7764dce862d970b33b88cb3 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.138909 2710 consensus_queue.cc:237] T 570201e7bf05406a97c81dde071d894b P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.138451 2823 consensus_queue.cc:1048] T 627a9cfd398346f2aa34c8932a11b5a8 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 07:52:47.141830 2615 raft_consensus.cc:2804] T 34c63b69904d499bb0d286941a8303f2 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:47.141949 2615 raft_consensus.cc:697] T 34c63b69904d499bb0d286941a8303f2 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:47.142244 2615 consensus_queue.cc:237] T 34c63b69904d499bb0d286941a8303f2 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.143157 2824 raft_consensus.cc:2804] T 682c1e22d7764dce862d970b33b88cb3 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:47.143231 2824 raft_consensus.cc:493] T 682c1e22d7764dce862d970b33b88cb3 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:47.143294 2824 raft_consensus.cc:3060] T 682c1e22d7764dce862d970b33b88cb3 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.144600 1167 tablet_bootstrap.cc:492] T 1c40b2e06d894a06af9e58285892ea8d P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:47.144783 1167 ts_tablet_manager.cc:1403] T 1c40b2e06d894a06af9e58285892ea8d P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.152s user 0.000s sys 0.008s
I20251025 07:52:47.145893 2824 raft_consensus.cc:515] T 682c1e22d7764dce862d970b33b88cb3 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.146425 2824 leader_election.cc:290] T 682c1e22d7764dce862d970b33b88cb3 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:47.146950 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "682c1e22d7764dce862d970b33b88cb3" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:47.147616 850 leader_election.cc:343] T 682c1e22d7764dce862d970b33b88cb3 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.147924 2663 raft_consensus.cc:2749] T e9e8fffb8e984463b5fa2c7dd9e93227 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:47.148706 710 leader_election.cc:343] T a7ab3150a35b4caca9119a1374d3f2f7 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.149243 710 leader_election.cc:304] T a7ab3150a35b4caca9119a1374d3f2f7 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.150290 2663 raft_consensus.cc:2749] T a7ab3150a35b4caca9119a1374d3f2f7 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:47.151366 710 leader_election.cc:343] T 29dbaf9149134f548a974a5779f7da67 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.151527 710 leader_election.cc:304] T 29dbaf9149134f548a974a5779f7da67 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.151801 2663 raft_consensus.cc:2749] T 29dbaf9149134f548a974a5779f7da67 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:47.152256 710 leader_election.cc:343] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.152899 710 leader_election.cc:304] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.153191 2663 raft_consensus.cc:2749] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:47.153649 710 leader_election.cc:304] T 37c7938e8c5e41f08506aff6747801c9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.154637 847 leader_election.cc:304] T 141e413d2faa4c67b4c822192e631417 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.152757 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 127) took 8423 ms. Trace:
I20251025 07:52:47.154922 2166 rpcz_store.cc:276] 1025 07:52:38.729060 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.729104 (+ 44us) service_pool.cc:225] Handling call
1025 07:52:46.876143 (+8147039us) ts_tablet_manager.cc:1151] Deleting tablet f4578ce69a0d4322944e5ce7c8a59831
1025 07:52:46.876151 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:46.991559 (+115408us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.046768 (+ 55209us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.152735 (+105967us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8146846}
I20251025 07:52:47.155236 2663 raft_consensus.cc:2764] T 37c7938e8c5e41f08506aff6747801c9 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20251025 07:52:47.155376 1166 tablet_bootstrap.cc:492] T 2b594d4718344aafbdaec34040695589 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:47.155941 2698 raft_consensus.cc:2749] T 141e413d2faa4c67b4c822192e631417 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251025 07:52:47.156931 1166 ts_tablet_manager.cc:1403] T 2b594d4718344aafbdaec34040695589 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.122s user 0.002s sys 0.015s
I20251025 07:52:47.157205 2663 raft_consensus.cc:493] T 645263a61b654f04985c40d04506c087 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:47.157326 2663 raft_consensus.cc:515] T 645263a61b654f04985c40d04506c087 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.157841 1166 raft_consensus.cc:359] T 2b594d4718344aafbdaec34040695589 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.157972 1166 raft_consensus.cc:385] T 2b594d4718344aafbdaec34040695589 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.158463 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "645263a61b654f04985c40d04506c087" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:47.159212 712 leader_election.cc:343] T 645263a61b654f04985c40d04506c087 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.159888 2888 raft_consensus.cc:493] T fce535a0c0bd45c6bf17f81add297f40 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:47.161520 1166 raft_consensus.cc:740] T 2b594d4718344aafbdaec34040695589 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.161887 1166 consensus_queue.cc:260] T 2b594d4718344aafbdaec34040695589 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.161942 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a8e2658a81e436298fe2190726080d4" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:47.162006 1167 raft_consensus.cc:359] T 1c40b2e06d894a06af9e58285892ea8d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.162153 770 raft_consensus.cc:3060] T 9a8e2658a81e436298fe2190726080d4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.162231 1167 raft_consensus.cc:385] T 1c40b2e06d894a06af9e58285892ea8d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.162293 1167 raft_consensus.cc:740] T 1c40b2e06d894a06af9e58285892ea8d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.162460 1167 consensus_queue.cc:260] T 1c40b2e06d894a06af9e58285892ea8d P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.162985 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "682c1e22d7764dce862d970b33b88cb3" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:47.163113 768 raft_consensus.cc:3060] T 682c1e22d7764dce862d970b33b88cb3 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.163153 1167 ts_tablet_manager.cc:1434] T 1c40b2e06d894a06af9e58285892ea8d P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.018s user 0.001s sys 0.000s
I20251025 07:52:47.163409 1167 tablet_bootstrap.cc:492] T e47c4f741f354a19b49777bd06e54764 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:47.171296 1169 tablet_bootstrap.cc:492] T c0c6324cbae34dcebade3ef9f89a7150 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:47.171293 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd265cdbf6a8440393a2b5d8240d1872" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:47.171445 1169 ts_tablet_manager.cc:1403] T c0c6324cbae34dcebade3ef9f89a7150 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.048s user 0.001s sys 0.001s
I20251025 07:52:47.171478 905 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa826b85f0c94150a17d4e417dd3d62f" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:47.171715 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "141e413d2faa4c67b4c822192e631417" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:47.171900 912 raft_consensus.cc:2393] T 141e413d2faa4c67b4c822192e631417 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 23eeffd3de254ce18db5febfde971c6b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:47.171939 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d3c9954693d4e8782ee0128ef066aa9" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:47.172079 1166 ts_tablet_manager.cc:1434] T 2b594d4718344aafbdaec34040695589 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20251025 07:52:47.172137 910 raft_consensus.cc:2393] T 6d3c9954693d4e8782ee0128ef066aa9 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 23eeffd3de254ce18db5febfde971c6b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:47.172267 1169 raft_consensus.cc:359] T c0c6324cbae34dcebade3ef9f89a7150 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.172379 1166 tablet_bootstrap.cc:492] T 1b5ac7e845a24b2eb5b2429d47361232 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:47.172430 1169 raft_consensus.cc:385] T c0c6324cbae34dcebade3ef9f89a7150 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.172482 1169 raft_consensus.cc:740] T c0c6324cbae34dcebade3ef9f89a7150 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.174042 1166 tablet_bootstrap.cc:654] T 1b5ac7e845a24b2eb5b2429d47361232 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.175740 2146 ts_tablet_manager.cc:1929] T 80fe3255ed614270bd8352a2853fbc43 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.175895 2146 log.cc:1199] T 80fe3255ed614270bd8352a2853fbc43 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/80fe3255ed614270bd8352a2853fbc43
I20251025 07:52:47.176661 1169 consensus_queue.cc:260] T c0c6324cbae34dcebade3ef9f89a7150 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.176833 772 raft_consensus.cc:2468] T dda0b26e4f1a44928bc4b84f333901c0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:47.177270 2663 leader_election.cc:290] T 645263a61b654f04985c40d04506c087 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:47.177383 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet f4578ce69a0d4322944e5ce7c8a59831 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]) successfully deleted
I20251025 07:52:47.175478 2169 ts_tablet_manager.cc:1929] T fa998b9ca6724c4cb1eb68a3eecb0b11 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.177824 2169 log.cc:1199] T fa998b9ca6724c4cb1eb68a3eecb0b11 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/fa998b9ca6724c4cb1eb68a3eecb0b11
I20251025 07:52:47.177938 2166 tablet_replica.cc:333] T e54fe7cbcffa40c4b04144b47bee2b48 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:47.178217 2169 ts_tablet_manager.cc:1950] T fa998b9ca6724c4cb1eb68a3eecb0b11 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:47.178203 2166 raft_consensus.cc:2243] T e54fe7cbcffa40c4b04144b47bee2b48 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:47.178442 2166 raft_consensus.cc:2272] T e54fe7cbcffa40c4b04144b47bee2b48 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.178956 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 163) took 5812 ms. Trace:
I20251025 07:52:47.179045 2169 rpcz_store.cc:276] 1025 07:52:41.366625 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.366674 (+ 49us) service_pool.cc:225] Handling call
1025 07:52:47.031591 (+5664917us) ts_tablet_manager.cc:1151] Deleting tablet fa998b9ca6724c4cb1eb68a3eecb0b11
1025 07:52:47.031601 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.079683 (+ 48082us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.175358 (+ 95675us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.178936 (+ 3578us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21644,"tablet-delete.queue_time_us":5664559}
I20251025 07:52:47.179553 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet fa998b9ca6724c4cb1eb68a3eecb0b11 (table default.Table_4976d4f5d05540258d144751d129bdb8 [id=8cde66661cf749919839da894ed4dc05]) successfully deleted
I20251025 07:52:47.179991 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 168) took 5733 ms. Trace:
I20251025 07:52:47.180163 2166 ts_tablet_manager.cc:1916] T e54fe7cbcffa40c4b04144b47bee2b48 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.180075 2169 rpcz_store.cc:276] 1025 07:52:41.446558 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.446621 (+ 63us) service_pool.cc:225] Handling call
1025 07:52:47.179824 (+5733203us) ts_tablet_manager.cc:1151] Deleting tablet dda0b26e4f1a44928bc4b84f333901c0
1025 07:52:47.179836 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.179975 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5733014}
W20251025 07:52:47.180492 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet dda0b26e4f1a44928bc4b84f333901c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dda0b26e4f1a44928bc4b84f333901c0 already in progress: creating tablet
I20251025 07:52:47.180784 2169 tablet_replica.cc:333] T 80ee75935d7e461d9d4066fac9819bef P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:47.180922 2146 ts_tablet_manager.cc:1950] T 80fe3255ed614270bd8352a2853fbc43 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:47.180959 2169 raft_consensus.cc:2243] T 80ee75935d7e461d9d4066fac9819bef P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:47.181269 2169 raft_consensus.cc:2272] T 80ee75935d7e461d9d4066fac9819bef P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.181413 1167 tablet_bootstrap.cc:654] T e47c4f741f354a19b49777bd06e54764 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.181785 2169 ts_tablet_manager.cc:1916] T 80ee75935d7e461d9d4066fac9819bef P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.182298 1169 ts_tablet_manager.cc:1434] T c0c6324cbae34dcebade3ef9f89a7150 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20251025 07:52:47.182565 1169 tablet_bootstrap.cc:492] T 570201e7bf05406a97c81dde071d894b P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:47.183302 770 raft_consensus.cc:2468] T 9a8e2658a81e436298fe2190726080d4 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:47.183764 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 167) took 5767 ms. Trace:
I20251025 07:52:47.183385 768 raft_consensus.cc:2468] T 682c1e22d7764dce862d970b33b88cb3 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:47.183928 2888 raft_consensus.cc:515] T fce535a0c0bd45c6bf17f81add297f40 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.190006 2888 leader_election.cc:290] T fce535a0c0bd45c6bf17f81add297f40 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:47.190330 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fce535a0c0bd45c6bf17f81add297f40" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:47.190824 712 leader_election.cc:343] T fce535a0c0bd45c6bf17f81add297f40 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.191080 2900 tablet.cc:1722] T bc2d16b33ca8426abbd7bd38339c9f1d P 23eeffd3de254ce18db5febfde971c6b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 07:52:47.193382 2169 ts_tablet_manager.cc:1929] T 80ee75935d7e461d9d4066fac9819bef P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.184430 847 leader_election.cc:304] T dda0b26e4f1a44928bc4b84f333901c0 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.196314 2146 rpcz_store.cc:276] 1025 07:52:41.416162 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.416198 (+ 36us) service_pool.cc:225] Handling call
1025 07:52:47.033777 (+5617579us) ts_tablet_manager.cc:1151] Deleting tablet 80fe3255ed614270bd8352a2853fbc43
1025 07:52:47.033786 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.037861 (+ 4075us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.175639 (+137778us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.183745 (+ 8106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5617424}
I20251025 07:52:47.196656 2795 raft_consensus.cc:2804] T dda0b26e4f1a44928bc4b84f333901c0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:47.196683 847 leader_election.cc:304] T 9a8e2658a81e436298fe2190726080d4 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.197024 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 80fe3255ed614270bd8352a2853fbc43 (table default.Table_d2a7e16c6f0d4b2886c16bfe6b685c00 [id=64c18a10455c4694babcc5ec0f463b85]) successfully deleted
I20251025 07:52:47.197073 847 leader_election.cc:304] T 682c1e22d7764dce862d970b33b88cb3 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.197453 2146 tablet_replica.cc:333] T ce1ebb1ec14b4801930a72765bca1a7d P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:47.197687 2698 raft_consensus.cc:2804] T 9a8e2658a81e436298fe2190726080d4 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:47.197813 2698 raft_consensus.cc:697] T 9a8e2658a81e436298fe2190726080d4 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:47.197897 2686 raft_consensus.cc:2804] T 682c1e22d7764dce862d970b33b88cb3 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:47.198012 2686 raft_consensus.cc:697] T 682c1e22d7764dce862d970b33b88cb3 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:47.198055 2698 consensus_queue.cc:237] T 9a8e2658a81e436298fe2190726080d4 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.198247 2686 consensus_queue.cc:237] T 682c1e22d7764dce862d970b33b88cb3 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.196738 2795 raft_consensus.cc:493] T dda0b26e4f1a44928bc4b84f333901c0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:47.199373 2795 raft_consensus.cc:3060] T dda0b26e4f1a44928bc4b84f333901c0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.202124 2795 raft_consensus.cc:515] T dda0b26e4f1a44928bc4b84f333901c0 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.203240 2795 leader_election.cc:290] T dda0b26e4f1a44928bc4b84f333901c0 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:47.203812 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dda0b26e4f1a44928bc4b84f333901c0" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:47.204071 770 raft_consensus.cc:3060] T dda0b26e4f1a44928bc4b84f333901c0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.204587 449 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_fa83d0528b4b481989bfc0ca35c6de44 [id=bf77b5ff6f1048dd996d948437f9ff5e]). Expected version 3 got 0
I20251025 07:52:47.204819 449 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3]). Expected version 1 got 0
I20251025 07:52:47.204988 449 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5]). Expected version 1 got 0
I20251025 07:52:47.205503 449 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f]). Expected version 1 got 0
I20251025 07:52:47.205812 449 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7]). Expected version 1 got 0
I20251025 07:52:47.197692 2146 raft_consensus.cc:2243] T ce1ebb1ec14b4801930a72765bca1a7d P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:47.206238 449 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 872cb7512d104827b1f1f186d33a4a44 (table default.Table_d5c65dce8b1b4214ba519d964012bfc0 [id=d4be274b36a845a99a4b96b8846248fd]). Expected version 1 got 0
I20251025 07:52:47.206525 449 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 04d27d54fdb7483b869408af65ff85c9 (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]). Expected version 2 got 0
I20251025 07:52:47.206516 2146 raft_consensus.cc:2272] T ce1ebb1ec14b4801930a72765bca1a7d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.206861 449 catalog_manager.cc:5649] T 4678f7eab5e8497db308e8a38e387618 P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } }
I20251025 07:52:47.207386 2146 ts_tablet_manager.cc:1916] T ce1ebb1ec14b4801930a72765bca1a7d P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.208201 770 raft_consensus.cc:2468] T dda0b26e4f1a44928bc4b84f333901c0 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:47.208896 847 leader_election.cc:304] T dda0b26e4f1a44928bc4b84f333901c0 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:47.195251 1169 tablet_bootstrap.cc:654] T 570201e7bf05406a97c81dde071d894b P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.203438 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dda0b26e4f1a44928bc4b84f333901c0" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:47.209137 2886 raft_consensus.cc:2804] T dda0b26e4f1a44928bc4b84f333901c0 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:47.209261 2886 raft_consensus.cc:697] T dda0b26e4f1a44928bc4b84f333901c0 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
W20251025 07:52:47.209453 850 leader_election.cc:343] T dda0b26e4f1a44928bc4b84f333901c0 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.209501 2886 consensus_queue.cc:237] T dda0b26e4f1a44928bc4b84f333901c0 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
W20251025 07:52:47.211148 710 leader_election.cc:343] T fd265cdbf6a8440393a2b5d8240d1872 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.211194 2169 log.cc:1199] T 80ee75935d7e461d9d4066fac9819bef P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/80ee75935d7e461d9d4066fac9819bef
I20251025 07:52:47.211308 710 leader_election.cc:304] T fd265cdbf6a8440393a2b5d8240d1872 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.211606 2169 ts_tablet_manager.cc:1950] T 80ee75935d7e461d9d4066fac9819bef P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:47.211890 2888 raft_consensus.cc:2749] T fd265cdbf6a8440393a2b5d8240d1872 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:47.213316 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 169) took 5766 ms. Trace:
W20251025 07:52:47.216131 710 leader_election.cc:343] T fa826b85f0c94150a17d4e417dd3d62f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.216246 710 leader_election.cc:304] T fa826b85f0c94150a17d4e417dd3d62f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.216509 2888 raft_consensus.cc:2749] T fa826b85f0c94150a17d4e417dd3d62f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:47.216625 2169 rpcz_store.cc:276] 1025 07:52:41.446664 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.446845 (+ 181us) service_pool.cc:225] Handling call
1025 07:52:47.180741 (+5733896us) ts_tablet_manager.cc:1151] Deleting tablet 80ee75935d7e461d9d4066fac9819bef
1025 07:52:47.180750 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.189965 (+ 9215us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.193276 (+ 3311us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.213291 (+ 20015us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5733743}
I20251025 07:52:47.217505 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 80ee75935d7e461d9d4066fac9819bef (table default.Table_bb258691f8844d85ab37eb73e9a4c314 [id=c80bab0ed06a459eb78b1d94df13f337]) successfully deleted
I20251025 07:52:47.218077 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 170) took 5771 ms. Trace:
W20251025 07:52:47.218576 710 leader_election.cc:343] T d5e5e5bbcf9f43118eea502b69880af5 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: d5e5e5bbcf9f43118eea502b69880af5
I20251025 07:52:47.218475 449 catalog_manager.cc:5649] T 36c414bc700341188b7d85c14173c3e0 P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } }
I20251025 07:52:47.218737 1167 tablet_bootstrap.cc:492] T e47c4f741f354a19b49777bd06e54764 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:47.218919 1167 ts_tablet_manager.cc:1403] T e47c4f741f354a19b49777bd06e54764 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.000s
I20251025 07:52:47.219620 1167 raft_consensus.cc:359] T e47c4f741f354a19b49777bd06e54764 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:47.219796 1167 raft_consensus.cc:385] T e47c4f741f354a19b49777bd06e54764 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.219852 1167 raft_consensus.cc:740] T e47c4f741f354a19b49777bd06e54764 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.219023 2169 rpcz_store.cc:276] 1025 07:52:41.446739 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.447025 (+ 286us) service_pool.cc:225] Handling call
1025 07:52:47.217920 (+5770895us) ts_tablet_manager.cc:1151] Deleting tablet 7e6544202d4f489d82f22d6073ef240d
1025 07:52:47.217934 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.218056 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5770775}
I20251025 07:52:47.220104 1167 consensus_queue.cc:260] T e47c4f741f354a19b49777bd06e54764 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
W20251025 07:52:47.220538 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 7e6544202d4f489d82f22d6073ef240d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e6544202d4f489d82f22d6073ef240d already in progress: creating tablet
I20251025 07:52:47.220739 1167 ts_tablet_manager.cc:1434] T e47c4f741f354a19b49777bd06e54764 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:47.221055 1167 tablet_bootstrap.cc:492] T 075f0a151e024220b51ba7fcaecdcc3d P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:47.221388 449 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 292fe04055954afd8c08ac2636b68191 (table default.Table_9fa69b10b5e842639fc0b0a58cd9b6a5 [id=a253ce0253bd4da4814741c482193d8f]). Expected version 1 got 0
I20251025 07:52:47.221642 449 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet aa9d9ab8073945d28d52fb02213f201a (table default.Table_4791672658f043f6b97b9320a531ebb5 [id=dfd20cf9c3e541c4abe23d59e0aa7f2a]). Expected version 1 got 0
I20251025 07:52:47.221840 449 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 5dab439bc50d4e869548154e1d222ad6 (table default.Table_fb9def2302784c999bf0f5707c60eb4c [id=45325c188eac4d34bb016d01de091f87]). Expected version 3 got 0
I20251025 07:52:47.222364 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 172) took 5775 ms. Trace:
I20251025 07:52:47.222728 2169 rpcz_store.cc:276] 1025 07:52:41.446920 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.447214 (+ 294us) service_pool.cc:225] Handling call
1025 07:52:47.222258 (+5775044us) ts_tablet_manager.cc:1151] Deleting tablet ffe7c8e1d6c44872b7b65e512764a56a
1025 07:52:47.222268 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.222348 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5774859}
I20251025 07:52:47.224026 2698 raft_consensus.cc:493] T 716fb61312e94a4d9eb8ab8336ca140c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:47.224195 2698 raft_consensus.cc:515] T 716fb61312e94a4d9eb8ab8336ca140c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.225433 2698 leader_election.cc:290] T 716fb61312e94a4d9eb8ab8336ca140c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:47.225147 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "716fb61312e94a4d9eb8ab8336ca140c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:47.225899 2166 ts_tablet_manager.cc:1929] T e54fe7cbcffa40c4b04144b47bee2b48 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.226012 2166 log.cc:1199] T e54fe7cbcffa40c4b04144b47bee2b48 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/e54fe7cbcffa40c4b04144b47bee2b48
I20251025 07:52:47.226059 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "716fb61312e94a4d9eb8ab8336ca140c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
W20251025 07:52:47.226217 850 leader_election.cc:343] T 716fb61312e94a4d9eb8ab8336ca140c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.226264 770 raft_consensus.cc:2468] T 716fb61312e94a4d9eb8ab8336ca140c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:47.226641 847 leader_election.cc:304] T 716fb61312e94a4d9eb8ab8336ca140c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:47.226711 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet ffe7c8e1d6c44872b7b65e512764a56a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ffe7c8e1d6c44872b7b65e512764a56a already in progress: creating tablet
I20251025 07:52:47.226987 2698 raft_consensus.cc:2804] T 716fb61312e94a4d9eb8ab8336ca140c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:47.227068 2698 raft_consensus.cc:493] T 716fb61312e94a4d9eb8ab8336ca140c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:47.227138 2698 raft_consensus.cc:3060] T 716fb61312e94a4d9eb8ab8336ca140c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.227298 2169 tablet_replica.cc:333] T 80fe3255ed614270bd8352a2853fbc43 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:47.227712 2169 raft_consensus.cc:2243] T 80fe3255ed614270bd8352a2853fbc43 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:47.227797 710 leader_election.cc:304] T d5e5e5bbcf9f43118eea502b69880af5 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.228109 2888 raft_consensus.cc:2749] T d5e5e5bbcf9f43118eea502b69880af5 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 07:52:47.228176 2169 raft_consensus.cc:2272] T 80fe3255ed614270bd8352a2853fbc43 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.229027 2169 ts_tablet_manager.cc:1916] T 80fe3255ed614270bd8352a2853fbc43 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.229643 449 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet f70091024c034c118fb86578db19a31c (table default.Table_b7da40b96d6c4359bed6f898e8cf6275 [id=e041946c093b4c789ce175c736cc230b]). Expected version 2 got 0
I20251025 07:52:47.229780 2698 raft_consensus.cc:515] T 716fb61312e94a4d9eb8ab8336ca140c P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.229709 710 leader_election.cc:304] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.230448 2888 raft_consensus.cc:2749] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251025 07:52:47.230834 710 leader_election.cc:304] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.231040 2888 raft_consensus.cc:2749] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251025 07:52:47.232636 2795 consensus_queue.cc:1048] T c0c6324cbae34dcebade3ef9f89a7150 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:47.233836 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "716fb61312e94a4d9eb8ab8336ca140c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:47.233896 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "716fb61312e94a4d9eb8ab8336ca140c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:47.234112 772 raft_consensus.cc:3060] T 716fb61312e94a4d9eb8ab8336ca140c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:47.234406 850 leader_election.cc:343] T 716fb61312e94a4d9eb8ab8336ca140c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.235806 449 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]). Expected version 3 got 0
W20251025 07:52:47.236896 850 consensus_peers.cc:597] T 143aa3fce9ba4850b000a94054e5dc3a P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:47.237555 772 raft_consensus.cc:2468] T 716fb61312e94a4d9eb8ab8336ca140c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:47.237885 847 leader_election.cc:304] T 716fb61312e94a4d9eb8ab8336ca140c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.238185 2710 raft_consensus.cc:2804] T 716fb61312e94a4d9eb8ab8336ca140c P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:47.238289 2710 raft_consensus.cc:697] T 716fb61312e94a4d9eb8ab8336ca140c P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:47.238576 2823 raft_consensus.cc:493] T 9f039d9107ae493486d799ab6a1d731d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:47.238729 2823 raft_consensus.cc:515] T 9f039d9107ae493486d799ab6a1d731d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.239871 2698 leader_election.cc:290] T 716fb61312e94a4d9eb8ab8336ca140c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:47.238557 449 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 5fc092c2bb2d41ac8c5b4fc07460c8ec (table default.Table_cd20e202397645f8821dcde13b5c0ce3 [id=630c274339664b9db18ab4ef7c29c16f]). Expected version 2 got 0
I20251025 07:52:47.240250 449 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 22f7db660616457b83b24e947a0b0bae (table default.Table_4a86191163f2408f9bcc795cd8d1bbf6 [id=bb316a7e000748ca902a9bfb494802d7]). Expected version 1 got 0
I20251025 07:52:47.238530 2710 consensus_queue.cc:237] T 716fb61312e94a4d9eb8ab8336ca140c P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.241801 449 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 0325d462441349ef81841f1b0cab2a20 (table default.Table_750ecb905839434fa9a1aa72f87394be [id=9ce3ba77c56a4ef09cb3b40ea0e488f6]). Expected version 1 got 0
I20251025 07:52:47.242410 1166 tablet_bootstrap.cc:492] T 1b5ac7e845a24b2eb5b2429d47361232 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:47.242539 2823 leader_election.cc:290] T 9f039d9107ae493486d799ab6a1d731d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:47.242542 1166 ts_tablet_manager.cc:1403] T 1b5ac7e845a24b2eb5b2429d47361232 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.070s user 0.004s sys 0.013s
I20251025 07:52:47.242859 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9f039d9107ae493486d799ab6a1d731d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:47.242996 2671 raft_consensus.cc:493] T 4a1ec8c2ed7f41abb588f592f62097d1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:47.243016 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9f039d9107ae493486d799ab6a1d731d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:47.243098 2671 raft_consensus.cc:515] T 4a1ec8c2ed7f41abb588f592f62097d1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.243161 772 raft_consensus.cc:2468] T 9f039d9107ae493486d799ab6a1d731d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
W20251025 07:52:47.243359 850 leader_election.cc:343] T 9f039d9107ae493486d799ab6a1d731d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.243502 847 leader_election.cc:304] T 9f039d9107ae493486d799ab6a1d731d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.243695 2671 leader_election.cc:290] T 4a1ec8c2ed7f41abb588f592f62097d1 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:47.243858 2823 raft_consensus.cc:2804] T 9f039d9107ae493486d799ab6a1d731d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:47.243920 2823 raft_consensus.cc:493] T 9f039d9107ae493486d799ab6a1d731d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:47.243971 2823 raft_consensus.cc:3060] T 9f039d9107ae493486d799ab6a1d731d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.244298 905 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a1ec8c2ed7f41abb588f592f62097d1" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:47.244625 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a1ec8c2ed7f41abb588f592f62097d1" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:47.244817 774 raft_consensus.cc:2393] T 4a1ec8c2ed7f41abb588f592f62097d1 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:47.245118 982 leader_election.cc:304] T 4a1ec8c2ed7f41abb588f592f62097d1 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:47.245548 2671 raft_consensus.cc:3060] T 4a1ec8c2ed7f41abb588f592f62097d1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.246601 2823 raft_consensus.cc:515] T 9f039d9107ae493486d799ab6a1d731d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.247337 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9f039d9107ae493486d799ab6a1d731d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:47.247762 850 leader_election.cc:343] T 9f039d9107ae493486d799ab6a1d731d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.248288 2823 leader_election.cc:290] T 9f039d9107ae493486d799ab6a1d731d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:47.248400 2671 raft_consensus.cc:2749] T 4a1ec8c2ed7f41abb588f592f62097d1 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:47.248383 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9f039d9107ae493486d799ab6a1d731d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:47.248525 774 raft_consensus.cc:3060] T 9f039d9107ae493486d799ab6a1d731d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.250373 1046 raft_consensus.cc:3060] T c0c6324cbae34dcebade3ef9f89a7150 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.250535 449 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 3a0f50c8000144a4acb5219484b01800 (table default.Table_c62e0dc1ae504f9bb912013506ff3031 [id=6f395892b09240fea02482b3199cfb5e]). Expected version 1 got 0
I20251025 07:52:47.250815 449 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525]). Expected version 1 got 0
I20251025 07:52:47.250983 449 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_67b99749f3bf43228b40d8b9807e69b4 [id=fc8180c09d7b4e74b2e161f7acb8fe0e]). Expected version 2 got 0
I20251025 07:52:47.251222 774 raft_consensus.cc:2468] T 9f039d9107ae493486d799ab6a1d731d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:47.251322 449 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236]). Expected version 1 got 0
I20251025 07:52:47.251493 449 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 8707b93d491e4e4386d148087a2f0870 (table default.Table_799dcb1cef2f4fe6b9b8f17f9f1026c5 [id=59840a567e5244e8a9024cbad42471e4]). Expected version 1 got 0
I20251025 07:52:47.251569 2166 ts_tablet_manager.cc:1950] T e54fe7cbcffa40c4b04144b47bee2b48 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:47.252171 449 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c]). Expected version 1 got 0
I20251025 07:52:47.252509 449 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 755f0d6c87044e20a8020f8410940123 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 4 got 0
I20251025 07:52:47.252838 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 128) took 8522 ms. Trace:
I20251025 07:52:47.253075 449 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet b56209e4934e4407a8b4518ca4af0d46 (table default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de]). Expected version 1 got 0
I20251025 07:52:47.253247 449 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 830fdbc6812e416690d0db42a73af503 (table default.Table_92824b3f2a774564b3aea04aaa25d204 [id=fb21a45b86ab446e8c3dae4512a9598f]). Expected version 1 got 0
I20251025 07:52:47.253664 2795 consensus_queue.cc:1048] T c0c6324cbae34dcebade3ef9f89a7150 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:47.254173 2166 rpcz_store.cc:276] 1025 07:52:38.730038 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.730090 (+ 52us) service_pool.cc:225] Handling call
1025 07:52:47.177766 (+8447676us) ts_tablet_manager.cc:1151] Deleting tablet e54fe7cbcffa40c4b04144b47bee2b48
1025 07:52:47.177842 (+ 76us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.204491 (+ 26649us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.225769 (+ 21278us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.252818 (+ 27049us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8447425}
I20251025 07:52:47.254837 2166 tablet_replica.cc:333] T f656945454fe4921ab819c772a089eb3 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:47.254921 847 leader_election.cc:304] T 9f039d9107ae493486d799ab6a1d731d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.255110 2857 tablet.cc:1722] T 04d27d54fdb7483b869408af65ff85c9 P 3aea2869ff814cc7b142012faa274382: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251025 07:52:47.255245 2686 raft_consensus.cc:2804] T 9f039d9107ae493486d799ab6a1d731d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:47.256321 2686 raft_consensus.cc:697] T 9f039d9107ae493486d799ab6a1d731d P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:47.256614 2686 consensus_queue.cc:237] T 9f039d9107ae493486d799ab6a1d731d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.262607 2146 ts_tablet_manager.cc:1929] T ce1ebb1ec14b4801930a72765bca1a7d P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.262786 2146 log.cc:1199] T ce1ebb1ec14b4801930a72765bca1a7d P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/ce1ebb1ec14b4801930a72765bca1a7d
I20251025 07:52:47.262996 2826 consensus_queue.cc:1048] T 143aa3fce9ba4850b000a94054e5dc3a P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:47.263289 2146 ts_tablet_manager.cc:1950] T ce1ebb1ec14b4801930a72765bca1a7d P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:47.263625 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 169) took 5847 ms. Trace:
I20251025 07:52:47.263758 2146 rpcz_store.cc:276] 1025 07:52:41.416362 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.416397 (+ 35us) service_pool.cc:225] Handling call
1025 07:52:47.197398 (+5781001us) ts_tablet_manager.cc:1151] Deleting tablet ce1ebb1ec14b4801930a72765bca1a7d
1025 07:52:47.197411 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.228558 (+ 31147us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.262500 (+ 33942us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.263611 (+ 1111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5780856}
I20251025 07:52:47.264032 2169 ts_tablet_manager.cc:1929] T 80fe3255ed614270bd8352a2853fbc43 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.264447 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet e54fe7cbcffa40c4b04144b47bee2b48 (table default.Table_590cc281d8b042b19ef12c7f6436752a [id=b1eee5c0243a48c4bbdecc0b4f38ab2d]) successfully deleted
I20251025 07:52:47.264725 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet ce1ebb1ec14b4801930a72765bca1a7d (table default.Table_d5abcbf8c8c741a2b40880a3467f2bcd [id=eab2657210e74ee6ac6f59544c9ba12e]) successfully deleted
I20251025 07:52:47.265301 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "645263a61b654f04985c40d04506c087" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:47.265622 2166 raft_consensus.cc:2243] T f656945454fe4921ab819c772a089eb3 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:47.258781 2888 raft_consensus.cc:493] T 3b940fa5ae26427dafbcfd26372754b3 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:47.265969 2888 raft_consensus.cc:515] T 3b940fa5ae26427dafbcfd26372754b3 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.267309 2888 leader_election.cc:290] T 3b940fa5ae26427dafbcfd26372754b3 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:47.268016 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fce535a0c0bd45c6bf17f81add297f40" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:47.269995 449 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]). Expected version 1 got 0
I20251025 07:52:47.270469 449 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_87d38832f85e4965bfab1a077990114f [id=ad812862698d421184ebf501ce79b7be]). Expected version 2 got 0
I20251025 07:52:47.270711 449 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84]). Expected version 1 got 0
I20251025 07:52:47.271348 449 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3]). Expected version 2 got 0
I20251025 07:52:47.271708 1167 tablet_bootstrap.cc:654] T 075f0a151e024220b51ba7fcaecdcc3d P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.271952 449 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 36262ab8c6964dd684c01b4db6219a4f (table default.Table_76158f8cf82e4cdb9a90580e46aaea48 [id=f249f30497bc450c887d659721469de8]). Expected version 1 got 0
I20251025 07:52:47.272455 2166 raft_consensus.cc:2272] T f656945454fe4921ab819c772a089eb3 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.273216 2166 ts_tablet_manager.cc:1916] T f656945454fe4921ab819c772a089eb3 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.275350 2663 raft_consensus.cc:493] T 437da5ded9b648abb668a8cb4896452c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:47.267007 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b940fa5ae26427dafbcfd26372754b3" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:47.275470 2663 raft_consensus.cc:515] T 437da5ded9b648abb668a8cb4896452c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.275669 1166 raft_consensus.cc:359] T 1b5ac7e845a24b2eb5b2429d47361232 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.275825 1166 raft_consensus.cc:385] T 1b5ac7e845a24b2eb5b2429d47361232 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.275867 1166 raft_consensus.cc:740] T 1b5ac7e845a24b2eb5b2429d47361232 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
W20251025 07:52:47.275965 712 leader_election.cc:343] T 3b940fa5ae26427dafbcfd26372754b3 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.276065 2663 leader_election.cc:290] T 437da5ded9b648abb668a8cb4896452c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:47.276037 1166 consensus_queue.cc:260] T 1b5ac7e845a24b2eb5b2429d47361232 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.276368 1166 ts_tablet_manager.cc:1434] T 1b5ac7e845a24b2eb5b2429d47361232 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.034s user 0.001s sys 0.000s
I20251025 07:52:47.276501 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "437da5ded9b648abb668a8cb4896452c" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:47.276993 712 leader_election.cc:343] T 437da5ded9b648abb668a8cb4896452c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.275460 1169 tablet_bootstrap.cc:492] T 570201e7bf05406a97c81dde071d894b P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:47.277307 1169 ts_tablet_manager.cc:1403] T 570201e7bf05406a97c81dde071d894b P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.095s user 0.015s sys 0.000s
I20251025 07:52:47.277194 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b940fa5ae26427dafbcfd26372754b3" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:47.278172 1169 raft_consensus.cc:359] T 570201e7bf05406a97c81dde071d894b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.278172 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "437da5ded9b648abb668a8cb4896452c" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:47.278934 1169 raft_consensus.cc:385] T 570201e7bf05406a97c81dde071d894b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.279038 1169 raft_consensus.cc:740] T 570201e7bf05406a97c81dde071d894b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.279338 1169 consensus_queue.cc:260] T 570201e7bf05406a97c81dde071d894b P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.279893 1169 ts_tablet_manager.cc:1434] T 570201e7bf05406a97c81dde071d894b P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:47.280299 1169 tablet_bootstrap.cc:492] T 627a9cfd398346f2aa34c8932a11b5a8 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:47.281985 1169 tablet_bootstrap.cc:654] T 627a9cfd398346f2aa34c8932a11b5a8 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.282491 449 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 288e543bb279410c91c01dd1623e110f (table default.Table_6e242ba55b194c23a6701f4b064fdbe5 [id=4c34b64b299f4162a53866cf01c5298f]). Expected version 1 got 0
I20251025 07:52:47.265355 2146 tablet_replica.cc:333] T ebc824ec4c60411dab1a2ea4a385148e P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:47.280027 2169 log.cc:1199] T 80fe3255ed614270bd8352a2853fbc43 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/80fe3255ed614270bd8352a2853fbc43
I20251025 07:52:47.284435 2169 ts_tablet_manager.cc:1950] T 80fe3255ed614270bd8352a2853fbc43 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
W20251025 07:52:47.284942 710 leader_election.cc:343] T 645263a61b654f04985c40d04506c087 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.285053 710 leader_election.cc:304] T 645263a61b654f04985c40d04506c087 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.285285 2888 raft_consensus.cc:2749] T 645263a61b654f04985c40d04506c087 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:47.285418 710 leader_election.cc:343] T fce535a0c0bd45c6bf17f81add297f40 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.285502 710 leader_election.cc:304] T fce535a0c0bd45c6bf17f81add297f40 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.285672 2888 raft_consensus.cc:2749] T fce535a0c0bd45c6bf17f81add297f40 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:47.285801 710 leader_election.cc:343] T 3b940fa5ae26427dafbcfd26372754b3 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.285965 710 leader_election.cc:304] T 3b940fa5ae26427dafbcfd26372754b3 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.286185 2888 raft_consensus.cc:2749] T 3b940fa5ae26427dafbcfd26372754b3 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:47.286294 710 leader_election.cc:343] T 437da5ded9b648abb668a8cb4896452c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.286373 710 leader_election.cc:304] T 437da5ded9b648abb668a8cb4896452c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.286564 2888 raft_consensus.cc:2749] T 437da5ded9b648abb668a8cb4896452c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:47.286844 2146 raft_consensus.cc:2243] T ebc824ec4c60411dab1a2ea4a385148e P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:47.287211 2146 raft_consensus.cc:2272] T ebc824ec4c60411dab1a2ea4a385148e P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.287679 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 174) took 5840 ms. Trace:
I20251025 07:52:47.287802 2146 ts_tablet_manager.cc:1916] T ebc824ec4c60411dab1a2ea4a385148e P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.287956 1166 tablet_bootstrap.cc:492] T 1068f566477e4c6db7dde566fff4c2fb P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:47.287843 2169 rpcz_store.cc:276] 1025 07:52:41.447106 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.447270 (+ 164us) service_pool.cc:225] Handling call
1025 07:52:47.227238 (+5779968us) ts_tablet_manager.cc:1151] Deleting tablet 80fe3255ed614270bd8352a2853fbc43
1025 07:52:47.227251 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.247398 (+ 20147us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.263906 (+ 16508us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.287643 (+ 23737us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5779743}
I20251025 07:52:47.288686 2169 tablet_replica.cc:333] T 5b6383c99e9e4c90a374da9ccd830a6f P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:47.288939 2169 raft_consensus.cc:2243] T 5b6383c99e9e4c90a374da9ccd830a6f P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:47.289132 2169 raft_consensus.cc:2272] T 5b6383c99e9e4c90a374da9ccd830a6f P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.288450 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 80fe3255ed614270bd8352a2853fbc43 (table default.Table_d2a7e16c6f0d4b2886c16bfe6b685c00 [id=64c18a10455c4694babcc5ec0f463b85]) successfully deleted
I20251025 07:52:47.289482 1166 tablet_bootstrap.cc:654] T 1068f566477e4c6db7dde566fff4c2fb P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.298595 2169 ts_tablet_manager.cc:1916] T 5b6383c99e9e4c90a374da9ccd830a6f P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.299543 1167 tablet_bootstrap.cc:492] T 075f0a151e024220b51ba7fcaecdcc3d P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:47.299732 1167 ts_tablet_manager.cc:1403] T 075f0a151e024220b51ba7fcaecdcc3d P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.079s user 0.004s sys 0.000s
I20251025 07:52:47.300617 1167 raft_consensus.cc:359] T 075f0a151e024220b51ba7fcaecdcc3d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.300848 1167 raft_consensus.cc:385] T 075f0a151e024220b51ba7fcaecdcc3d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.300923 1167 raft_consensus.cc:740] T 075f0a151e024220b51ba7fcaecdcc3d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.301211 1167 consensus_queue.cc:260] T 075f0a151e024220b51ba7fcaecdcc3d P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.301764 1167 ts_tablet_manager.cc:1434] T 075f0a151e024220b51ba7fcaecdcc3d P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:47.290273 449 catalog_manager.cc:5649] T bdaeedbcf35a4aad97f9ef192991978a P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } }
I20251025 07:52:47.302030 1167 tablet_bootstrap.cc:492] T 67cf96a7dd3f42e19b86746c87e689f6 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:47.303411 1167 tablet_bootstrap.cc:654] T 67cf96a7dd3f42e19b86746c87e689f6 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.308264 2146 ts_tablet_manager.cc:1929] T ebc824ec4c60411dab1a2ea4a385148e P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.317176 2166 ts_tablet_manager.cc:1929] T f656945454fe4921ab819c772a089eb3 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.317431 2166 log.cc:1199] T f656945454fe4921ab819c772a089eb3 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/f656945454fe4921ab819c772a089eb3
I20251025 07:52:47.318064 2166 ts_tablet_manager.cc:1950] T f656945454fe4921ab819c772a089eb3 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:47.323076 2146 log.cc:1199] T ebc824ec4c60411dab1a2ea4a385148e P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/ebc824ec4c60411dab1a2ea4a385148e
I20251025 07:52:47.323519 2146 ts_tablet_manager.cc:1950] T ebc824ec4c60411dab1a2ea4a385148e P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:47.319177 1166 tablet_bootstrap.cc:492] T 1068f566477e4c6db7dde566fff4c2fb P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:47.324007 1166 ts_tablet_manager.cc:1403] T 1068f566477e4c6db7dde566fff4c2fb P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.036s user 0.013s sys 0.007s
I20251025 07:52:47.324764 1166 raft_consensus.cc:359] T 1068f566477e4c6db7dde566fff4c2fb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.324957 1166 raft_consensus.cc:385] T 1068f566477e4c6db7dde566fff4c2fb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.325032 1166 raft_consensus.cc:740] T 1068f566477e4c6db7dde566fff4c2fb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.325169 449 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e]). Expected version 1 got 0
I20251025 07:52:47.325273 1166 consensus_queue.cc:260] T 1068f566477e4c6db7dde566fff4c2fb P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.325582 449 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet bc2d16b33ca8426abbd7bd38339c9f1d (table default.Table_4ea985df4c5b415c9570c64506a98cc4 [id=98d2b7797e33485e8872054aa2eb8ec9]). Expected version 1 got 0
I20251025 07:52:47.325820 1166 ts_tablet_manager.cc:1434] T 1068f566477e4c6db7dde566fff4c2fb P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:47.326076 1166 tablet_bootstrap.cc:492] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:47.326761 449 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]). Expected version 1 got 0
I20251025 07:52:47.327040 449 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]). Expected version 1 got 0
I20251025 07:52:47.329588 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 129) took 8594 ms. Trace:
I20251025 07:52:47.329686 2166 rpcz_store.cc:276] 1025 07:52:38.734930 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.734975 (+ 45us) service_pool.cc:225] Handling call
1025 07:52:47.254787 (+8519812us) ts_tablet_manager.cc:1151] Deleting tablet f656945454fe4921ab819c772a089eb3
1025 07:52:47.254798 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.276809 (+ 22011us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.317037 (+ 40228us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.329567 (+ 12530us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8519577}
I20251025 07:52:47.329983 2166 tablet_replica.cc:333] T 71432a6848944a0da06ef9b81f160dc4 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:47.330124 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet f656945454fe4921ab819c772a089eb3 (table default.Table_7ce7627f71134d5c989f9ba5c6766a2a [id=b5bf17121a4e4e3a8f21dccf1dba4a99]) successfully deleted
I20251025 07:52:47.330201 2166 raft_consensus.cc:2243] T 71432a6848944a0da06ef9b81f160dc4 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:47.330335 2166 raft_consensus.cc:2272] T 71432a6848944a0da06ef9b81f160dc4 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.330809 2166 ts_tablet_manager.cc:1916] T 71432a6848944a0da06ef9b81f160dc4 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.322953 1167 tablet_bootstrap.cc:492] T 67cf96a7dd3f42e19b86746c87e689f6 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:47.331903 1167 ts_tablet_manager.cc:1403] T 67cf96a7dd3f42e19b86746c87e689f6 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20251025 07:52:47.332573 1167 raft_consensus.cc:359] T 67cf96a7dd3f42e19b86746c87e689f6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.332726 1167 raft_consensus.cc:385] T 67cf96a7dd3f42e19b86746c87e689f6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.332787 1167 raft_consensus.cc:740] T 67cf96a7dd3f42e19b86746c87e689f6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.332959 1167 consensus_queue.cc:260] T 67cf96a7dd3f42e19b86746c87e689f6 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.333410 1167 ts_tablet_manager.cc:1434] T 67cf96a7dd3f42e19b86746c87e689f6 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251025 07:52:47.333658 1167 tablet_bootstrap.cc:492] T 288e543bb279410c91c01dd1623e110f P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:47.334909 1167 tablet_bootstrap.cc:654] T 288e543bb279410c91c01dd1623e110f P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.336206 1166 tablet_bootstrap.cc:654] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.347453 2166 ts_tablet_manager.cc:1929] T 71432a6848944a0da06ef9b81f160dc4 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.347659 2166 log.cc:1199] T 71432a6848944a0da06ef9b81f160dc4 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/71432a6848944a0da06ef9b81f160dc4
I20251025 07:52:47.348097 2166 ts_tablet_manager.cc:1950] T 71432a6848944a0da06ef9b81f160dc4 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:47.348976 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 130) took 8612 ms. Trace:
I20251025 07:52:47.349131 2166 rpcz_store.cc:276] 1025 07:52:38.736992 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.737037 (+ 45us) service_pool.cc:225] Handling call
1025 07:52:47.329943 (+8592906us) ts_tablet_manager.cc:1151] Deleting tablet 71432a6848944a0da06ef9b81f160dc4
1025 07:52:47.329951 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.344163 (+ 14212us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.347359 (+ 3196us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.348955 (+ 1596us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8592717}
I20251025 07:52:47.349644 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 71432a6848944a0da06ef9b81f160dc4 (table default.Table_79969e4ace6f4227b2748b59136027d6 [id=a1135611169142ba9ef987e3e9f95ce8]) successfully deleted
I20251025 07:52:47.350008 2166 tablet_replica.cc:333] T c59f6e6eb9594a8e83f109a09b034045 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:47.350256 2166 raft_consensus.cc:2243] T c59f6e6eb9594a8e83f109a09b034045 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:47.350502 2166 raft_consensus.cc:2272] T c59f6e6eb9594a8e83f109a09b034045 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.351015 2166 ts_tablet_manager.cc:1916] T c59f6e6eb9594a8e83f109a09b034045 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.334110 2169 ts_tablet_manager.cc:1929] T 5b6383c99e9e4c90a374da9ccd830a6f P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.353106 2169 log.cc:1199] T 5b6383c99e9e4c90a374da9ccd830a6f P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/5b6383c99e9e4c90a374da9ccd830a6f
I20251025 07:52:47.353361 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 171) took 5936 ms. Trace:
I20251025 07:52:47.335050 1169 tablet_bootstrap.cc:492] T 627a9cfd398346f2aa34c8932a11b5a8 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:47.353544 2146 rpcz_store.cc:276] 1025 07:52:41.416530 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.416573 (+ 43us) service_pool.cc:225] Handling call
1025 07:52:47.265295 (+5848722us) ts_tablet_manager.cc:1151] Deleting tablet ebc824ec4c60411dab1a2ea4a385148e
1025 07:52:47.265310 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.305277 (+ 39967us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.308177 (+ 2900us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.353337 (+ 45160us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3493,"tablet-delete.queue_time_us":5848591}
I20251025 07:52:47.353710 1169 ts_tablet_manager.cc:1403] T 627a9cfd398346f2aa34c8932a11b5a8 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.073s user 0.002s sys 0.002s
I20251025 07:52:47.354568 1169 raft_consensus.cc:359] T 627a9cfd398346f2aa34c8932a11b5a8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.356717 2146 tablet_replica.cc:333] T 050a5d3b5f934dd19bf2e221f2fe0780 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:47.356943 2146 raft_consensus.cc:2243] T 050a5d3b5f934dd19bf2e221f2fe0780 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:47.357510 2166 ts_tablet_manager.cc:1929] T c59f6e6eb9594a8e83f109a09b034045 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.357658 2166 log.cc:1199] T c59f6e6eb9594a8e83f109a09b034045 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/c59f6e6eb9594a8e83f109a09b034045
I20251025 07:52:47.358021 2166 ts_tablet_manager.cc:1950] T c59f6e6eb9594a8e83f109a09b034045 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:47.358386 2146 raft_consensus.cc:2272] T 050a5d3b5f934dd19bf2e221f2fe0780 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.358935 2146 ts_tablet_manager.cc:1916] T 050a5d3b5f934dd19bf2e221f2fe0780 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.358963 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 131) took 8618 ms. Trace:
I20251025 07:52:47.359130 2166 rpcz_store.cc:276] 1025 07:52:38.740143 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.740189 (+ 46us) service_pool.cc:225] Handling call
1025 07:52:47.349963 (+8609774us) ts_tablet_manager.cc:1151] Deleting tablet c59f6e6eb9594a8e83f109a09b034045
1025 07:52:47.349973 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.354371 (+ 4398us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.357430 (+ 3059us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.358940 (+ 1510us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8609580}
I20251025 07:52:47.359602 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet c59f6e6eb9594a8e83f109a09b034045 (table default.Table_500d949c884c4644b70943543bd3e0c5 [id=71ac1bb83b444af2aede5644bf5619a3]) successfully deleted
I20251025 07:52:47.359925 2166 tablet_replica.cc:333] T 8956ad28b35d45f28c683ac055bba441 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:47.360124 2166 raft_consensus.cc:2243] T 8956ad28b35d45f28c683ac055bba441 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:47.360291 2166 raft_consensus.cc:2272] T 8956ad28b35d45f28c683ac055bba441 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.360769 2166 ts_tablet_manager.cc:1916] T 8956ad28b35d45f28c683ac055bba441 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.361080 2888 raft_consensus.cc:493] T 53198283718d4b45b30df65b5bde96a6 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db42be7ad3c741a0aa427f27e245993b)
I20251025 07:52:47.361210 2888 raft_consensus.cc:515] T 53198283718d4b45b30df65b5bde96a6 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.361722 2888 leader_election.cc:290] T 53198283718d4b45b30df65b5bde96a6 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:47.353520 2169 ts_tablet_manager.cc:1950] T 5b6383c99e9e4c90a374da9ccd830a6f P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
W20251025 07:52:47.362507 712 leader_election.cc:343] T 53198283718d4b45b30df65b5bde96a6 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: 53198283718d4b45b30df65b5bde96a6
W20251025 07:52:47.362646 710 leader_election.cc:343] T 53198283718d4b45b30df65b5bde96a6 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 53198283718d4b45b30df65b5bde96a6
I20251025 07:52:47.362738 710 leader_election.cc:304] T 53198283718d4b45b30df65b5bde96a6 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.363076 2888 raft_consensus.cc:2749] T 53198283718d4b45b30df65b5bde96a6 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 07:52:47.368255 1167 tablet_bootstrap.cc:492] T 288e543bb279410c91c01dd1623e110f P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:47.368389 1167 ts_tablet_manager.cc:1403] T 288e543bb279410c91c01dd1623e110f P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.004s
I20251025 07:52:47.369258 2166 ts_tablet_manager.cc:1929] T 8956ad28b35d45f28c683ac055bba441 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.369410 2166 log.cc:1199] T 8956ad28b35d45f28c683ac055bba441 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/8956ad28b35d45f28c683ac055bba441
I20251025 07:52:47.369807 2166 ts_tablet_manager.cc:1950] T 8956ad28b35d45f28c683ac055bba441 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:47.370522 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 132) took 8629 ms. Trace:
I20251025 07:52:47.370643 2166 rpcz_store.cc:276] 1025 07:52:38.741081 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.741124 (+ 43us) service_pool.cc:225] Handling call
1025 07:52:47.359883 (+8618759us) ts_tablet_manager.cc:1151] Deleting tablet 8956ad28b35d45f28c683ac055bba441
1025 07:52:47.359894 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.365280 (+ 5386us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.369172 (+ 3892us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.370505 (+ 1333us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8618604}
I20251025 07:52:47.371150 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 8956ad28b35d45f28c683ac055bba441 (table default.Table_f84af906af804c7a8e98ad602783d747 [id=39a63c8186374189ad21fbf8492d5d2b]) successfully deleted
I20251025 07:52:47.371565 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 133) took 8628 ms. Trace:
I20251025 07:52:47.371668 2166 rpcz_store.cc:276] 1025 07:52:38.742994 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.743038 (+ 44us) service_pool.cc:225] Handling call
1025 07:52:47.371452 (+8628414us) ts_tablet_manager.cc:1151] Deleting tablet ffe7c8e1d6c44872b7b65e512764a56a
1025 07:52:47.371471 (+ 19us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.371552 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8628182}
W20251025 07:52:47.372107 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet ffe7c8e1d6c44872b7b65e512764a56a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ffe7c8e1d6c44872b7b65e512764a56a already in progress: creating tablet
I20251025 07:52:47.369181 1167 raft_consensus.cc:359] T 288e543bb279410c91c01dd1623e110f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.372378 1167 raft_consensus.cc:385] T 288e543bb279410c91c01dd1623e110f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.372476 1167 raft_consensus.cc:740] T 288e543bb279410c91c01dd1623e110f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.372802 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 175) took 5925 ms. Trace:
I20251025 07:52:47.372845 1167 consensus_queue.cc:260] T 288e543bb279410c91c01dd1623e110f P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.372651 2166 tablet_replica.cc:333] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:47.372947 2169 rpcz_store.cc:276] 1025 07:52:41.447219 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.447479 (+ 260us) service_pool.cc:225] Handling call
1025 07:52:47.288626 (+5841147us) ts_tablet_manager.cc:1151] Deleting tablet 5b6383c99e9e4c90a374da9ccd830a6f
1025 07:52:47.288636 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.330956 (+ 42320us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.334007 (+ 3051us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.372776 (+ 38769us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5841018}
I20251025 07:52:47.373258 2166 raft_consensus.cc:2243] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:47.373478 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 173) took 5926 ms. Trace:
I20251025 07:52:47.373562 2166 raft_consensus.cc:2272] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.373593 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 5b6383c99e9e4c90a374da9ccd830a6f (table default.Table_32f8a762ec25446dac28e2729d6879bf [id=a13b2a42afca460595cd872ab2cf9ea0]) successfully deleted
I20251025 07:52:47.373615 2169 rpcz_store.cc:276] 1025 07:52:41.447020 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.447501 (+ 481us) service_pool.cc:225] Handling call
1025 07:52:47.373341 (+5925840us) ts_tablet_manager.cc:1151] Deleting tablet 4f0c4f0e8f014e59be9e46d2afe0be9d
1025 07:52:47.373353 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.373444 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5925716}
W20251025 07:52:47.373963 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 4f0c4f0e8f014e59be9e46d2afe0be9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f0c4f0e8f014e59be9e46d2afe0be9d already in progress: creating tablet
I20251025 07:52:47.374053 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 171) took 5927 ms. Trace:
I20251025 07:52:47.374114 2169 rpcz_store.cc:276] 1025 07:52:41.446814 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.447669 (+ 855us) service_pool.cc:225] Handling call
1025 07:52:47.373962 (+5926293us) ts_tablet_manager.cc:1151] Deleting tablet db99a1e68daf4e948a3d66ad276cc5f9
1025 07:52:47.373970 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.374038 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5926132}
I20251025 07:52:47.374430 2166 ts_tablet_manager.cc:1916] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:47.374593 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet db99a1e68daf4e948a3d66ad276cc5f9 because the tablet was not found. No further retry: Not found: Tablet not found: db99a1e68daf4e948a3d66ad276cc5f9
I20251025 07:52:47.374665 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 166) took 5928 ms. Trace:
I20251025 07:52:47.374722 2169 rpcz_store.cc:276] 1025 07:52:41.446366 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.447637 (+ 1271us) service_pool.cc:225] Handling call
1025 07:52:47.374543 (+5926906us) ts_tablet_manager.cc:1151] Deleting tablet 716fb61312e94a4d9eb8ab8336ca140c
1025 07:52:47.374552 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.374650 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5926690}
W20251025 07:52:47.375206 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 716fb61312e94a4d9eb8ab8336ca140c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 716fb61312e94a4d9eb8ab8336ca140c already in progress: creating tablet
I20251025 07:52:47.375231 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 167) took 5928 ms. Trace:
I20251025 07:52:47.375289 2169 rpcz_store.cc:276] 1025 07:52:41.446462 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.447881 (+ 1419us) service_pool.cc:225] Handling call
1025 07:52:47.375150 (+5927269us) ts_tablet_manager.cc:1151] Deleting tablet bef0ef23d90d480d9c99aef21d7271fd
1025 07:52:47.375158 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.375216 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5927129}
I20251025 07:52:47.375699 2169 tablet_replica.cc:333] T 91728987303b41fd840eb56f3a311c5f P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
W20251025 07:52:47.375689 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet bef0ef23d90d480d9c99aef21d7271fd because the tablet was not found. No further retry: Not found: Tablet not found: bef0ef23d90d480d9c99aef21d7271fd
I20251025 07:52:47.375996 2169 raft_consensus.cc:2243] T 91728987303b41fd840eb56f3a311c5f P db42be7ad3c741a0aa427f27e245993b [term 2 LEADER]: Raft consensus shutting down.
I20251025 07:52:47.373932 1167 ts_tablet_manager.cc:1434] T 288e543bb279410c91c01dd1623e110f P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251025 07:52:47.377360 2169 pending_rounds.cc:70] T 91728987303b41fd840eb56f3a311c5f P db42be7ad3c741a0aa427f27e245993b: Trying to abort 1 pending ops.
I20251025 07:52:47.377367 1167 tablet_bootstrap.cc:492] T 95efd553493e4966bc48a01cac6e1e10 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:47.377466 2169 pending_rounds.cc:77] T 91728987303b41fd840eb56f3a311c5f P db42be7ad3c741a0aa427f27e245993b: Aborting op as it isn't in flight: id { term: 2 index: 1 } timestamp: 7214607427448963072 op_type: NO_OP noop_request { }
I20251025 07:52:47.377630 2169 raft_consensus.cc:2889] T 91728987303b41fd840eb56f3a311c5f P db42be7ad3c741a0aa427f27e245993b [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 07:52:47.377794 2169 raft_consensus.cc:2272] T 91728987303b41fd840eb56f3a311c5f P db42be7ad3c741a0aa427f27e245993b [term 2 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.378758 1167 tablet_bootstrap.cc:654] T 95efd553493e4966bc48a01cac6e1e10 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.379323 1166 tablet_bootstrap.cc:492] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:47.379459 1166 ts_tablet_manager.cc:1403] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.053s user 0.005s sys 0.001s
I20251025 07:52:47.379547 2169 ts_tablet_manager.cc:1916] T 91728987303b41fd840eb56f3a311c5f P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.380187 1166 raft_consensus.cc:359] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.380355 1166 raft_consensus.cc:385] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.380409 1166 raft_consensus.cc:740] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.380613 1166 consensus_queue.cc:260] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.381146 1166 ts_tablet_manager.cc:1434] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:47.381575 1166 tablet_bootstrap.cc:492] T e9e8fffb8e984463b5fa2c7dd9e93227 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:47.383014 2166 ts_tablet_manager.cc:1929] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.383133 1166 tablet_bootstrap.cc:654] T e9e8fffb8e984463b5fa2c7dd9e93227 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.384882 2166 log.cc:1199] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/db99a1e68daf4e948a3d66ad276cc5f9
I20251025 07:52:47.385401 2166 ts_tablet_manager.cc:1950] T db99a1e68daf4e948a3d66ad276cc5f9 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:47.386831 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 134) took 8619 ms. Trace:
I20251025 07:52:47.387390 2166 rpcz_store.cc:276] 1025 07:52:38.770655 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.770807 (+ 152us) service_pool.cc:225] Handling call
1025 07:52:47.372616 (+8601809us) ts_tablet_manager.cc:1151] Deleting tablet db99a1e68daf4e948a3d66ad276cc5f9
1025 07:52:47.372623 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.379967 (+ 7344us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.382933 (+ 2966us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.386813 (+ 3880us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":131,"tablet-delete.queue_time_us":8601594}
I20251025 07:52:47.388211 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet db99a1e68daf4e948a3d66ad276cc5f9 (table default.Table_e1d698cd782e4d58b29e9cc7d1f56703 [id=e8dcb9f9f6554f5ebcdfee63c6259f03]) successfully deleted
I20251025 07:52:47.388391 2169 ts_tablet_manager.cc:1929] T 91728987303b41fd840eb56f3a311c5f P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20251025 07:52:47.388592 2166 tablet_replica.cc:333] T d5e5e5bbcf9f43118eea502b69880af5 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:47.388978 2166 raft_consensus.cc:2243] T d5e5e5bbcf9f43118eea502b69880af5 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:47.389364 2166 raft_consensus.cc:2272] T d5e5e5bbcf9f43118eea502b69880af5 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.390027 2166 ts_tablet_manager.cc:1916] T d5e5e5bbcf9f43118eea502b69880af5 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.392803 2146 ts_tablet_manager.cc:1929] T 050a5d3b5f934dd19bf2e221f2fe0780 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.395555 1167 tablet_bootstrap.cc:492] T 95efd553493e4966bc48a01cac6e1e10 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:47.395684 1167 ts_tablet_manager.cc:1403] T 95efd553493e4966bc48a01cac6e1e10 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.005s
I20251025 07:52:47.396291 1167 raft_consensus.cc:359] T 95efd553493e4966bc48a01cac6e1e10 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.396447 1167 raft_consensus.cc:385] T 95efd553493e4966bc48a01cac6e1e10 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.396507 1167 raft_consensus.cc:740] T 95efd553493e4966bc48a01cac6e1e10 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.396723 1167 consensus_queue.cc:260] T 95efd553493e4966bc48a01cac6e1e10 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.397262 1167 ts_tablet_manager.cc:1434] T 95efd553493e4966bc48a01cac6e1e10 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:47.397513 1167 tablet_bootstrap.cc:492] T 2d24e3e7a9e4431db609331ad1fd57c9 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:47.398895 1167 tablet_bootstrap.cc:654] T 2d24e3e7a9e4431db609331ad1fd57c9 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.402657 2166 ts_tablet_manager.cc:1929] T d5e5e5bbcf9f43118eea502b69880af5 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.403616 2169 log.cc:1199] T 91728987303b41fd840eb56f3a311c5f P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/91728987303b41fd840eb56f3a311c5f
I20251025 07:52:47.404115 2169 ts_tablet_manager.cc:1950] T 91728987303b41fd840eb56f3a311c5f P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:47.404408 2166 log.cc:1199] T d5e5e5bbcf9f43118eea502b69880af5 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/d5e5e5bbcf9f43118eea502b69880af5
I20251025 07:52:47.405004 2166 ts_tablet_manager.cc:1950] T d5e5e5bbcf9f43118eea502b69880af5 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:47.404984 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 178) took 5953 ms. Trace:
I20251025 07:52:47.405066 2169 rpcz_store.cc:276] 1025 07:52:41.451827 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.451905 (+ 78us) service_pool.cc:225] Handling call
1025 07:52:47.375656 (+5923751us) ts_tablet_manager.cc:1151] Deleting tablet 91728987303b41fd840eb56f3a311c5f
1025 07:52:47.375666 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.384957 (+ 9291us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.388290 (+ 3333us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.404961 (+ 16671us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5923551}
I20251025 07:52:47.392957 1169 raft_consensus.cc:385] T 627a9cfd398346f2aa34c8932a11b5a8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.405422 1169 raft_consensus.cc:740] T 627a9cfd398346f2aa34c8932a11b5a8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.405769 1169 consensus_queue.cc:260] T 627a9cfd398346f2aa34c8932a11b5a8 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.406860 1169 ts_tablet_manager.cc:1434] T 627a9cfd398346f2aa34c8932a11b5a8 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.053s user 0.011s sys 0.003s
I20251025 07:52:47.406100 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 136) took 8630 ms. Trace:
I20251025 07:52:47.407156 2166 rpcz_store.cc:276] 1025 07:52:38.776070 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.776122 (+ 52us) service_pool.cc:225] Handling call
1025 07:52:47.388549 (+8612427us) ts_tablet_manager.cc:1151] Deleting tablet d5e5e5bbcf9f43118eea502b69880af5
1025 07:52:47.388558 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.399667 (+ 11109us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.402568 (+ 2901us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.406084 (+ 3516us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8612154}
I20251025 07:52:47.407248 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 91728987303b41fd840eb56f3a311c5f (table default.Table_b53575428f5b4c25a5c912428e5d4f2d [id=88b6e7cfe0c74d169585a1b064d58e4a]) successfully deleted
I20251025 07:52:47.408034 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet d5e5e5bbcf9f43118eea502b69880af5 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]) successfully deleted
I20251025 07:52:47.408372 2166 tablet_replica.cc:333] T 7e6544202d4f489d82f22d6073ef240d P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:47.408777 2166 raft_consensus.cc:2243] T 7e6544202d4f489d82f22d6073ef240d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:47.409590 2166 raft_consensus.cc:2272] T 7e6544202d4f489d82f22d6073ef240d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.410033 1169 tablet_bootstrap.cc:492] T 143aa3fce9ba4850b000a94054e5dc3a P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:47.409430 1167 tablet_bootstrap.cc:492] T 2d24e3e7a9e4431db609331ad1fd57c9 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:47.410243 1167 ts_tablet_manager.cc:1403] T 2d24e3e7a9e4431db609331ad1fd57c9 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.001s
I20251025 07:52:47.410840 1167 raft_consensus.cc:359] T 2d24e3e7a9e4431db609331ad1fd57c9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.410984 1167 raft_consensus.cc:385] T 2d24e3e7a9e4431db609331ad1fd57c9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.411026 1167 raft_consensus.cc:740] T 2d24e3e7a9e4431db609331ad1fd57c9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.411226 1167 consensus_queue.cc:260] T 2d24e3e7a9e4431db609331ad1fd57c9 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.411814 1167 ts_tablet_manager.cc:1434] T 2d24e3e7a9e4431db609331ad1fd57c9 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:47.412057 1167 tablet_bootstrap.cc:492] T 243b8828fc344bacab46d7bab6bd2dec P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:47.412977 1169 tablet_bootstrap.cc:654] T 143aa3fce9ba4850b000a94054e5dc3a P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.413501 1167 tablet_bootstrap.cc:654] T 243b8828fc344bacab46d7bab6bd2dec P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.413789 2169 tablet_replica.cc:333] T aaef24ccdfba45fe8204327723df834e P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:47.414016 2169 raft_consensus.cc:2243] T aaef24ccdfba45fe8204327723df834e P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:47.410104 2166 ts_tablet_manager.cc:1916] T 7e6544202d4f489d82f22d6073ef240d P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.414294 2169 raft_consensus.cc:2272] T aaef24ccdfba45fe8204327723df834e P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.415030 2169 ts_tablet_manager.cc:1916] T aaef24ccdfba45fe8204327723df834e P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.415557 2671 raft_consensus.cc:493] T 570201e7bf05406a97c81dde071d894b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:47.415685 2671 raft_consensus.cc:515] T 570201e7bf05406a97c81dde071d894b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.416181 2671 leader_election.cc:290] T 570201e7bf05406a97c81dde071d894b P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:47.420960 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "570201e7bf05406a97c81dde071d894b" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:47.421236 770 raft_consensus.cc:2393] T 570201e7bf05406a97c81dde071d894b P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:47.421341 2166 ts_tablet_manager.cc:1929] T 7e6544202d4f489d82f22d6073ef240d P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:47.421433 2166 log.cc:1199] T 7e6544202d4f489d82f22d6073ef240d P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/7e6544202d4f489d82f22d6073ef240d
I20251025 07:52:47.424893 1167 tablet_bootstrap.cc:492] T 243b8828fc344bacab46d7bab6bd2dec P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:47.425024 1167 ts_tablet_manager.cc:1403] T 243b8828fc344bacab46d7bab6bd2dec P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.001s
I20251025 07:52:47.425630 1167 raft_consensus.cc:359] T 243b8828fc344bacab46d7bab6bd2dec P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.425794 1167 raft_consensus.cc:385] T 243b8828fc344bacab46d7bab6bd2dec P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.425851 1167 raft_consensus.cc:740] T 243b8828fc344bacab46d7bab6bd2dec P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.426061 1167 consensus_queue.cc:260] T 243b8828fc344bacab46d7bab6bd2dec P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.426725 1167 ts_tablet_manager.cc:1434] T 243b8828fc344bacab46d7bab6bd2dec P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:47.426971 1167 tablet_bootstrap.cc:492] T 792f32cbd66449ca98813c669b2a261d P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:47.428227 1167 tablet_bootstrap.cc:654] T 792f32cbd66449ca98813c669b2a261d P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.429133 2166 ts_tablet_manager.cc:1950] T 7e6544202d4f489d82f22d6073ef240d P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:47.429876 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 135) took 8658 ms. Trace:
I20251025 07:52:47.430007 2166 rpcz_store.cc:276] 1025 07:52:38.771167 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.771360 (+ 193us) service_pool.cc:225] Handling call
1025 07:52:47.408326 (+8636966us) ts_tablet_manager.cc:1151] Deleting tablet 7e6544202d4f489d82f22d6073ef240d
1025 07:52:47.408336 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.417266 (+ 8930us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.421247 (+ 3981us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.429858 (+ 8611us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8631315}
I20251025 07:52:47.430617 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 7e6544202d4f489d82f22d6073ef240d (table default.Table_8da73d66704143de8a4b4480c28fd1d9 [id=93fcfeca6dbf48baa3691b7939d0e4ec]) successfully deleted
I20251025 07:52:47.431048 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 137) took 8653 ms. Trace:
I20251025 07:52:47.431162 2166 rpcz_store.cc:276] 1025 07:52:38.777501 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.777546 (+ 45us) service_pool.cc:225] Handling call
1025 07:52:47.430937 (+8653391us) ts_tablet_manager.cc:1151] Deleting tablet 5361f08f50b745eb98bf02e979d7c71e
1025 07:52:47.430946 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.431028 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8653188}
W20251025 07:52:47.431648 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 5361f08f50b745eb98bf02e979d7c71e because the tablet was not found. No further retry: Not found: Tablet not found: 5361f08f50b745eb98bf02e979d7c71e
I20251025 07:52:47.431988 2166 tablet_replica.cc:333] T 53198283718d4b45b30df65b5bde96a6 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:47.432215 2166 raft_consensus.cc:2243] T 53198283718d4b45b30df65b5bde96a6 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:47.432399 2166 raft_consensus.cc:2272] T 53198283718d4b45b30df65b5bde96a6 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.432961 2166 ts_tablet_manager.cc:1916] T 53198283718d4b45b30df65b5bde96a6 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.436851 2169 ts_tablet_manager.cc:1929] T aaef24ccdfba45fe8204327723df834e P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.438737 2169 log.cc:1199] T aaef24ccdfba45fe8204327723df834e P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/aaef24ccdfba45fe8204327723df834e
I20251025 07:52:47.439337 2169 ts_tablet_manager.cc:1950] T aaef24ccdfba45fe8204327723df834e P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:47.439558 1167 tablet_bootstrap.cc:492] T 792f32cbd66449ca98813c669b2a261d P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:47.439690 1167 ts_tablet_manager.cc:1403] T 792f32cbd66449ca98813c669b2a261d P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20251025 07:52:47.440318 1167 raft_consensus.cc:359] T 792f32cbd66449ca98813c669b2a261d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.440485 1167 raft_consensus.cc:385] T 792f32cbd66449ca98813c669b2a261d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.440534 1167 raft_consensus.cc:740] T 792f32cbd66449ca98813c669b2a261d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.440750 1167 consensus_queue.cc:260] T 792f32cbd66449ca98813c669b2a261d P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.441313 1167 ts_tablet_manager.cc:1434] T 792f32cbd66449ca98813c669b2a261d P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:47.441569 1167 tablet_bootstrap.cc:492] T a6ae1315b28940d49b667e3c80347021 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:47.442205 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 180) took 5990 ms. Trace:
I20251025 07:52:47.443133 1167 tablet_bootstrap.cc:654] T a6ae1315b28940d49b667e3c80347021 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.445716 2166 ts_tablet_manager.cc:1929] T 53198283718d4b45b30df65b5bde96a6 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.442855 1169 tablet_bootstrap.cc:492] T 143aa3fce9ba4850b000a94054e5dc3a P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:47.402173 1166 tablet_bootstrap.cc:492] T e9e8fffb8e984463b5fa2c7dd9e93227 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:47.446153 1166 ts_tablet_manager.cc:1403] T e9e8fffb8e984463b5fa2c7dd9e93227 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.065s user 0.003s sys 0.000s
I20251025 07:52:47.446101 1169 ts_tablet_manager.cc:1403] T 143aa3fce9ba4850b000a94054e5dc3a P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.005s
I20251025 07:52:47.446389 2166 log.cc:1199] T 53198283718d4b45b30df65b5bde96a6 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/53198283718d4b45b30df65b5bde96a6
I20251025 07:52:47.446805 2166 ts_tablet_manager.cc:1950] T 53198283718d4b45b30df65b5bde96a6 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:47.447000 1166 raft_consensus.cc:359] T e9e8fffb8e984463b5fa2c7dd9e93227 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.447018 1169 raft_consensus.cc:359] T 143aa3fce9ba4850b000a94054e5dc3a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.447166 1169 raft_consensus.cc:385] T 143aa3fce9ba4850b000a94054e5dc3a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.447165 1166 raft_consensus.cc:385] T e9e8fffb8e984463b5fa2c7dd9e93227 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.447225 1166 raft_consensus.cc:740] T e9e8fffb8e984463b5fa2c7dd9e93227 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.447250 1169 raft_consensus.cc:740] T 143aa3fce9ba4850b000a94054e5dc3a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.447417 1166 consensus_queue.cc:260] T e9e8fffb8e984463b5fa2c7dd9e93227 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.447608 1167 tablet_bootstrap.cc:492] T a6ae1315b28940d49b667e3c80347021 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:47.447515 1169 consensus_queue.cc:260] T 143aa3fce9ba4850b000a94054e5dc3a P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.447768 1167 ts_tablet_manager.cc:1403] T a6ae1315b28940d49b667e3c80347021 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20251025 07:52:47.447927 1166 ts_tablet_manager.cc:1434] T e9e8fffb8e984463b5fa2c7dd9e93227 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:47.448190 1166 tablet_bootstrap.cc:492] T fd265cdbf6a8440393a2b5d8240d1872 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:47.446038 2146 log.cc:1199] T 050a5d3b5f934dd19bf2e221f2fe0780 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/050a5d3b5f934dd19bf2e221f2fe0780
I20251025 07:52:47.448339 1169 ts_tablet_manager.cc:1434] T 143aa3fce9ba4850b000a94054e5dc3a P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:47.448370 1167 raft_consensus.cc:359] T a6ae1315b28940d49b667e3c80347021 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.448526 1167 raft_consensus.cc:385] T a6ae1315b28940d49b667e3c80347021 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.448580 1167 raft_consensus.cc:740] T a6ae1315b28940d49b667e3c80347021 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.448765 2146 ts_tablet_manager.cc:1950] T 050a5d3b5f934dd19bf2e221f2fe0780 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:47.448853 1169 tablet_bootstrap.cc:492] T 4f0c4f0e8f014e59be9e46d2afe0be9d P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:47.448791 1167 consensus_queue.cc:260] T a6ae1315b28940d49b667e3c80347021 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.449376 1167 ts_tablet_manager.cc:1434] T a6ae1315b28940d49b667e3c80347021 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:47.449640 1167 tablet_bootstrap.cc:492] T 062c0f3e51a3487880cea373f8cf1f54 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:47.450299 1169 tablet_bootstrap.cc:654] T 4f0c4f0e8f014e59be9e46d2afe0be9d P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.450394 2169 rpcz_store.cc:276] 1025 07:52:41.452036 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.452080 (+ 44us) service_pool.cc:225] Handling call
1025 07:52:47.407226 (+5955146us) ts_tablet_manager.cc:1151] Deleting tablet aaef24ccdfba45fe8204327723df834e
1025 07:52:47.413730 (+ 6504us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.423666 (+ 9936us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.436743 (+ 13077us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.442181 (+ 5438us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5954989}
I20251025 07:52:47.451501 2169 tablet_replica.cc:333] T 050a5d3b5f934dd19bf2e221f2fe0780 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:47.451478 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet aaef24ccdfba45fe8204327723df834e (table default.Table_bd5089a7892041649dd05b7fb0961255 [id=840b3ffcb44144548843457c48f46380]) successfully deleted
I20251025 07:52:47.451728 2169 raft_consensus.cc:2243] T 050a5d3b5f934dd19bf2e221f2fe0780 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:47.451932 2169 raft_consensus.cc:2272] T 050a5d3b5f934dd19bf2e221f2fe0780 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.452425 1167 tablet_bootstrap.cc:654] T 062c0f3e51a3487880cea373f8cf1f54 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.452605 2169 ts_tablet_manager.cc:1916] T 050a5d3b5f934dd19bf2e221f2fe0780 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.453971 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 138) took 8676 ms. Trace:
I20251025 07:52:47.454092 2166 rpcz_store.cc:276] 1025 07:52:38.777942 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.777987 (+ 45us) service_pool.cc:225] Handling call
1025 07:52:47.431945 (+8653958us) ts_tablet_manager.cc:1151] Deleting tablet 53198283718d4b45b30df65b5bde96a6
1025 07:52:47.431953 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.436602 (+ 4649us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.445622 (+ 9020us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.453953 (+ 8331us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8653755}
I20251025 07:52:47.454612 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 53198283718d4b45b30df65b5bde96a6 (table default.Table_77761ef857464942be15bf6e67127d6d [id=d9d0e8bb796a45a8b17336c04b12cd01]) successfully deleted
I20251025 07:52:47.454927 2166 tablet_replica.cc:333] T c88c46e022ff41a58b02538f2151e260 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:47.455132 2166 raft_consensus.cc:2243] T c88c46e022ff41a58b02538f2151e260 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:47.455471 2166 raft_consensus.cc:2272] T c88c46e022ff41a58b02538f2151e260 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.456097 2166 ts_tablet_manager.cc:1916] T c88c46e022ff41a58b02538f2151e260 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.416780 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "570201e7bf05406a97c81dde071d894b" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:47.449761 1166 tablet_bootstrap.cc:654] T fd265cdbf6a8440393a2b5d8240d1872 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.457201 983 leader_election.cc:304] T 570201e7bf05406a97c81dde071d894b P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:47.457602 2597 raft_consensus.cc:3060] T 570201e7bf05406a97c81dde071d894b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:47.405694 847 consensus_peers.cc:597] T f656945454fe4921ab819c772a089eb3 P 3aea2869ff814cc7b142012faa274382 -> Peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Couldn't send request to peer 23eeffd3de254ce18db5febfde971c6b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f656945454fe4921ab819c772a089eb3. This is attempt 1: this message will repeat every 5th retry.
W20251025 07:52:47.457801 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:46231, user_credentials={real_user=slave}} blocked reactor thread for 52147.6us
I20251025 07:52:47.460103 2597 raft_consensus.cc:2749] T 570201e7bf05406a97c81dde071d894b P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:47.461302 2169 ts_tablet_manager.cc:1929] T 050a5d3b5f934dd19bf2e221f2fe0780 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.462061 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 172) took 6045 ms. Trace:
I20251025 07:52:47.463912 2146 rpcz_store.cc:276] 1025 07:52:41.416623 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.416660 (+ 37us) service_pool.cc:225] Handling call
1025 07:52:47.356666 (+5940006us) ts_tablet_manager.cc:1151] Deleting tablet 050a5d3b5f934dd19bf2e221f2fe0780
1025 07:52:47.356677 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.389858 (+ 33181us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.392717 (+ 2859us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.462036 (+ 69319us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5939886}
I20251025 07:52:47.465911 1167 tablet_bootstrap.cc:492] T 062c0f3e51a3487880cea373f8cf1f54 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:47.466041 1167 ts_tablet_manager.cc:1403] T 062c0f3e51a3487880cea373f8cf1f54 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.002s
I20251025 07:52:47.466357 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 050a5d3b5f934dd19bf2e221f2fe0780 (table default.Table_3bbc24e4eb334b6487c9da0142b1fe03 [id=9f09387ac53b4027a147057debd10743]) successfully deleted
I20251025 07:52:47.466853 2146 tablet_replica.cc:333] T 04449cc97f404a19b6a1b20dea074f95 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:47.466836 1167 raft_consensus.cc:359] T 062c0f3e51a3487880cea373f8cf1f54 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.466992 1167 raft_consensus.cc:385] T 062c0f3e51a3487880cea373f8cf1f54 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.467036 1167 raft_consensus.cc:740] T 062c0f3e51a3487880cea373f8cf1f54 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.467229 1167 consensus_queue.cc:260] T 062c0f3e51a3487880cea373f8cf1f54 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.467841 1167 ts_tablet_manager.cc:1434] T 062c0f3e51a3487880cea373f8cf1f54 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:47.468073 1167 tablet_bootstrap.cc:492] T e429b94326cc4827a03968fd92de38e0 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:47.469380 2169 log.cc:1199] T 050a5d3b5f934dd19bf2e221f2fe0780 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/050a5d3b5f934dd19bf2e221f2fe0780
I20251025 07:52:47.469762 1167 tablet_bootstrap.cc:654] T e429b94326cc4827a03968fd92de38e0 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.470252 2169 ts_tablet_manager.cc:1950] T 050a5d3b5f934dd19bf2e221f2fe0780 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:47.471501 2166 ts_tablet_manager.cc:1929] T c88c46e022ff41a58b02538f2151e260 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.471616 2166 log.cc:1199] T c88c46e022ff41a58b02538f2151e260 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/c88c46e022ff41a58b02538f2151e260
I20251025 07:52:47.471941 2166 ts_tablet_manager.cc:1950] T c88c46e022ff41a58b02538f2151e260 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:47.472606 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 139) took 8694 ms. Trace:
I20251025 07:52:47.472695 2166 rpcz_store.cc:276] 1025 07:52:38.778372 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.778413 (+ 41us) service_pool.cc:225] Handling call
1025 07:52:47.454884 (+8676471us) ts_tablet_manager.cc:1151] Deleting tablet c88c46e022ff41a58b02538f2151e260
1025 07:52:47.454893 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.459304 (+ 4411us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.471395 (+ 12091us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.472588 (+ 1193us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8676305}
I20251025 07:52:47.473201 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet c88c46e022ff41a58b02538f2151e260 (table default.Table_f6e21e14e6fb4649af45c7d10a5f993e [id=c5eda563853a4e15b846d063608d42ed]) successfully deleted
I20251025 07:52:47.473654 2166 tablet_replica.cc:333] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:47.473868 2166 raft_consensus.cc:2243] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:47.474272 2166 raft_consensus.cc:2272] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.475157 2166 ts_tablet_manager.cc:1916] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.476161 1167 tablet_bootstrap.cc:492] T e429b94326cc4827a03968fd92de38e0 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:47.478200 2146 raft_consensus.cc:2243] T 04449cc97f404a19b6a1b20dea074f95 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:47.478327 1167 ts_tablet_manager.cc:1403] T e429b94326cc4827a03968fd92de38e0 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20251025 07:52:47.478987 1167 raft_consensus.cc:359] T e429b94326cc4827a03968fd92de38e0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:47.479168 1167 raft_consensus.cc:385] T e429b94326cc4827a03968fd92de38e0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.479215 1167 raft_consensus.cc:740] T e429b94326cc4827a03968fd92de38e0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.479461 1167 consensus_queue.cc:260] T e429b94326cc4827a03968fd92de38e0 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:47.480039 1167 ts_tablet_manager.cc:1434] T e429b94326cc4827a03968fd92de38e0 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:47.480185 2888 raft_consensus.cc:493] T 792f32cbd66449ca98813c669b2a261d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:47.480276 2888 raft_consensus.cc:515] T 792f32cbd66449ca98813c669b2a261d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.480826 2888 leader_election.cc:290] T 792f32cbd66449ca98813c669b2a261d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:47.481477 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "792f32cbd66449ca98813c669b2a261d" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:47.481572 2888 raft_consensus.cc:493] T d46673a283fe42b78ddeb99fbd2678eb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:47.481681 2888 raft_consensus.cc:515] T d46673a283fe42b78ddeb99fbd2678eb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.481846 1167 tablet_bootstrap.cc:492] T af743ee344e543a0aa6da3cd7a30e467 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
W20251025 07:52:47.482012 712 leader_election.cc:343] T 792f32cbd66449ca98813c669b2a261d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.482148 2888 leader_election.cc:290] T d46673a283fe42b78ddeb99fbd2678eb P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:47.482570 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d46673a283fe42b78ddeb99fbd2678eb" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:47.483004 712 leader_election.cc:343] T d46673a283fe42b78ddeb99fbd2678eb P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.483429 1167 tablet_bootstrap.cc:654] T af743ee344e543a0aa6da3cd7a30e467 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.486547 2146 raft_consensus.cc:2272] T 04449cc97f404a19b6a1b20dea074f95 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.487260 2146 ts_tablet_manager.cc:1916] T 04449cc97f404a19b6a1b20dea074f95 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.475805 1166 tablet_bootstrap.cc:492] T fd265cdbf6a8440393a2b5d8240d1872 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:47.481673 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "792f32cbd66449ca98813c669b2a261d" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:47.482800 905 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d46673a283fe42b78ddeb99fbd2678eb" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:47.490515 1166 ts_tablet_manager.cc:1403] T fd265cdbf6a8440393a2b5d8240d1872 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.042s user 0.009s sys 0.000s
W20251025 07:52:47.490829 710 leader_election.cc:343] T d46673a283fe42b78ddeb99fbd2678eb P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.490911 2166 ts_tablet_manager.cc:1929] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.490938 710 leader_election.cc:304] T d46673a283fe42b78ddeb99fbd2678eb P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.490991 2166 log.cc:1199] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/9d6a9937d10a4cc2813671dd1bfa1664
I20251025 07:52:47.491190 1166 raft_consensus.cc:359] T fd265cdbf6a8440393a2b5d8240d1872 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:47.491328 1166 raft_consensus.cc:385] T fd265cdbf6a8440393a2b5d8240d1872 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251025 07:52:47.491309 710 leader_election.cc:343] T 792f32cbd66449ca98813c669b2a261d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.491371 1166 raft_consensus.cc:740] T fd265cdbf6a8440393a2b5d8240d1872 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.491437 710 leader_election.cc:304] T 792f32cbd66449ca98813c669b2a261d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.491602 1166 consensus_queue.cc:260] T fd265cdbf6a8440393a2b5d8240d1872 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:47.491305 2166 ts_tablet_manager.cc:1950] T 9d6a9937d10a4cc2813671dd1bfa1664 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:47.491852 2663 raft_consensus.cc:2749] T d46673a283fe42b78ddeb99fbd2678eb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:47.491998 2663 raft_consensus.cc:2749] T 792f32cbd66449ca98813c669b2a261d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:47.492149 1166 ts_tablet_manager.cc:1434] T fd265cdbf6a8440393a2b5d8240d1872 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:47.492395 1166 tablet_bootstrap.cc:492] T dff7f86ae8924685b26011e504e2247b P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:47.492502 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 140) took 8713 ms. Trace:
I20251025 07:52:47.492587 2166 rpcz_store.cc:276] 1025 07:52:38.778834 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.778873 (+ 39us) service_pool.cc:225] Handling call
1025 07:52:47.473586 (+8694713us) ts_tablet_manager.cc:1151] Deleting tablet 9d6a9937d10a4cc2813671dd1bfa1664
1025 07:52:47.473598 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.485480 (+ 11882us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.489279 (+ 3799us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.492485 (+ 3206us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8694530}
I20251025 07:52:47.493045 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 9d6a9937d10a4cc2813671dd1bfa1664 (table default.Table_e7d32e91b9c64ce1b3fcff3d806e959f [id=bab869e5001c4d999daa22083fca128a]) successfully deleted
I20251025 07:52:47.494556 2166 tablet_replica.cc:333] T f3c87a83082b421cb4be546ac4ce7ad8 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:47.494969 2166 raft_consensus.cc:2243] T f3c87a83082b421cb4be546ac4ce7ad8 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:47.495376 2166 raft_consensus.cc:2272] T f3c87a83082b421cb4be546ac4ce7ad8 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.496091 2166 ts_tablet_manager.cc:1916] T f3c87a83082b421cb4be546ac4ce7ad8 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.499073 1166 tablet_bootstrap.cc:654] T dff7f86ae8924685b26011e504e2247b P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.507138 1029 tablet_service.cc:1505] Processing CreateTablet for tablet eef92eab7f3943809842696e918cc129 (DEFAULT_TABLE table=default.Table_735de8c152fe443991bb9516d9a56d8d [id=e66a9203695e46d4aa3027db9065011e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.509476 2166 ts_tablet_manager.cc:1929] T f3c87a83082b421cb4be546ac4ce7ad8 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.511193 2166 log.cc:1199] T f3c87a83082b421cb4be546ac4ce7ad8 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/f3c87a83082b421cb4be546ac4ce7ad8
I20251025 07:52:47.511639 878 tablet_service.cc:1505] Processing CreateTablet for tablet eef92eab7f3943809842696e918cc129 (DEFAULT_TABLE table=default.Table_735de8c152fe443991bb9516d9a56d8d [id=e66a9203695e46d4aa3027db9065011e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.512138 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eef92eab7f3943809842696e918cc129. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.511757 2166 ts_tablet_manager.cc:1950] T f3c87a83082b421cb4be546ac4ce7ad8 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:47.512513 743 tablet_service.cc:1505] Processing CreateTablet for tablet eef92eab7f3943809842696e918cc129 (DEFAULT_TABLE table=default.Table_735de8c152fe443991bb9516d9a56d8d [id=e66a9203695e46d4aa3027db9065011e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.512897 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eef92eab7f3943809842696e918cc129. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.512977 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 141) took 8733 ms. Trace:
I20251025 07:52:47.513053 2166 rpcz_store.cc:276] 1025 07:52:38.779277 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.779318 (+ 41us) service_pool.cc:225] Handling call
1025 07:52:47.494513 (+8715195us) ts_tablet_manager.cc:1151] Deleting tablet f3c87a83082b421cb4be546ac4ce7ad8
1025 07:52:47.494522 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.501374 (+ 6852us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.509370 (+ 7996us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.512957 (+ 3587us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8715023}
I20251025 07:52:47.513484 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet f3c87a83082b421cb4be546ac4ce7ad8 (table default.Table_fcddb31989ee4236a4981c12779b3f5d [id=f3794fa081364666b944c412415bdba0]) successfully deleted
I20251025 07:52:47.514873 894 tablet_service.cc:1505] Processing CreateTablet for tablet 0a65981d224c44f0b3fc7a33fc2e001f (DEFAULT_TABLE table=default.Table_5fd6059c44b34bc5a067c100d84f4c3d [id=569c2982e2264b139f3a6d6547227bb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.514861 1017 tablet_service.cc:1505] Processing CreateTablet for tablet 0a65981d224c44f0b3fc7a33fc2e001f (DEFAULT_TABLE table=default.Table_5fd6059c44b34bc5a067c100d84f4c3d [id=569c2982e2264b139f3a6d6547227bb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.515251 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a65981d224c44f0b3fc7a33fc2e001f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.515334 2166 tablet_replica.cc:333] T 062c0f3e51a3487880cea373f8cf1f54 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:47.515525 2166 raft_consensus.cc:2243] T 062c0f3e51a3487880cea373f8cf1f54 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:47.515676 2166 raft_consensus.cc:2272] T 062c0f3e51a3487880cea373f8cf1f54 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.516782 754 tablet_service.cc:1505] Processing CreateTablet for tablet 0a65981d224c44f0b3fc7a33fc2e001f (DEFAULT_TABLE table=default.Table_5fd6059c44b34bc5a067c100d84f4c3d [id=569c2982e2264b139f3a6d6547227bb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.516973 2166 ts_tablet_manager.cc:1916] T 062c0f3e51a3487880cea373f8cf1f54 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.517217 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a65981d224c44f0b3fc7a33fc2e001f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.519305 875 tablet_service.cc:1505] Processing CreateTablet for tablet c600345cbefa4621b833cc834e37ba85 (DEFAULT_TABLE table=default.Table_acb609759cce423fa0dd70c8d6be1373 [id=15a724cc575340db97a581caff5a0a65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.519747 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c600345cbefa4621b833cc834e37ba85. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.519780 749 tablet_service.cc:1505] Processing CreateTablet for tablet c600345cbefa4621b833cc834e37ba85 (DEFAULT_TABLE table=default.Table_acb609759cce423fa0dd70c8d6be1373 [id=15a724cc575340db97a581caff5a0a65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.520082 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c600345cbefa4621b833cc834e37ba85. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.523248 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a65981d224c44f0b3fc7a33fc2e001f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.523535 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 179) took 6071 ms. Trace:
I20251025 07:52:47.523630 2169 rpcz_store.cc:276] 1025 07:52:41.451947 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.452149 (+ 202us) service_pool.cc:225] Handling call
1025 07:52:47.451454 (+5999305us) ts_tablet_manager.cc:1151] Deleting tablet 050a5d3b5f934dd19bf2e221f2fe0780
1025 07:52:47.451464 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.458327 (+ 6863us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.461220 (+ 2893us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.523502 (+ 62282us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5999166}
I20251025 07:52:47.523993 2169 tablet_replica.cc:333] T 0f5f6ca6c3cd4d369690e425387a4344 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:47.524181 2169 raft_consensus.cc:2243] T 0f5f6ca6c3cd4d369690e425387a4344 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:47.524335 2169 raft_consensus.cc:2272] T 0f5f6ca6c3cd4d369690e425387a4344 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.524897 2169 ts_tablet_manager.cc:1916] T 0f5f6ca6c3cd4d369690e425387a4344 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.526347 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eef92eab7f3943809842696e918cc129. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.526671 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 050a5d3b5f934dd19bf2e221f2fe0780 (table default.Table_3bbc24e4eb334b6487c9da0142b1fe03 [id=9f09387ac53b4027a147057debd10743]) successfully deleted
I20251025 07:52:47.530423 1019 tablet_service.cc:1505] Processing CreateTablet for tablet c600345cbefa4621b833cc834e37ba85 (DEFAULT_TABLE table=default.Table_acb609759cce423fa0dd70c8d6be1373 [id=15a724cc575340db97a581caff5a0a65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.530742 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c600345cbefa4621b833cc834e37ba85. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.539134 2169 ts_tablet_manager.cc:1929] T 0f5f6ca6c3cd4d369690e425387a4344 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.539258 2169 log.cc:1199] T 0f5f6ca6c3cd4d369690e425387a4344 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/0f5f6ca6c3cd4d369690e425387a4344
I20251025 07:52:47.539651 2169 ts_tablet_manager.cc:1950] T 0f5f6ca6c3cd4d369690e425387a4344 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:47.540383 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 181) took 6088 ms. Trace:
I20251025 07:52:47.540478 2169 rpcz_store.cc:276] 1025 07:52:41.452367 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.452418 (+ 51us) service_pool.cc:225] Handling call
1025 07:52:47.523949 (+6071531us) ts_tablet_manager.cc:1151] Deleting tablet 0f5f6ca6c3cd4d369690e425387a4344
1025 07:52:47.523958 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.531293 (+ 7335us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.538973 (+ 7680us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.540363 (+ 1390us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6071360}
I20251025 07:52:47.540928 2169 tablet_replica.cc:333] T ce1ebb1ec14b4801930a72765bca1a7d P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:47.541114 2169 raft_consensus.cc:2243] T ce1ebb1ec14b4801930a72765bca1a7d P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:47.541898 2169 raft_consensus.cc:2272] T ce1ebb1ec14b4801930a72765bca1a7d P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.542755 2169 ts_tablet_manager.cc:1916] T ce1ebb1ec14b4801930a72765bca1a7d P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.543216 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 0f5f6ca6c3cd4d369690e425387a4344 (table default.Table_e317f8d728da4c76886f67d0995ab7f4 [id=4d2121dea4904850aecb65da5dd2d8f9]) successfully deleted
I20251025 07:52:47.546567 1166 tablet_bootstrap.cc:492] T dff7f86ae8924685b26011e504e2247b P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:47.546799 1166 ts_tablet_manager.cc:1403] T dff7f86ae8924685b26011e504e2247b P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.054s user 0.001s sys 0.004s
I20251025 07:52:47.549499 2166 ts_tablet_manager.cc:1929] T 062c0f3e51a3487880cea373f8cf1f54 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:47.549607 2166 log.cc:1199] T 062c0f3e51a3487880cea373f8cf1f54 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/062c0f3e51a3487880cea373f8cf1f54
I20251025 07:52:47.549881 2166 ts_tablet_manager.cc:1950] T 062c0f3e51a3487880cea373f8cf1f54 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:47.551970 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 142) took 8657 ms. Trace:
I20251025 07:52:47.552054 2166 rpcz_store.cc:276] 1025 07:52:38.894713 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:38.894764 (+ 51us) service_pool.cc:225] Handling call
1025 07:52:47.515293 (+8620529us) ts_tablet_manager.cc:1151] Deleting tablet 062c0f3e51a3487880cea373f8cf1f54
1025 07:52:47.515302 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.521939 (+ 6637us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.549393 (+ 27454us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.551951 (+ 2558us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8619865}
I20251025 07:52:47.552240 1167 tablet_bootstrap.cc:492] T af743ee344e543a0aa6da3cd7a30e467 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:47.552343 1167 ts_tablet_manager.cc:1403] T af743ee344e543a0aa6da3cd7a30e467 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.071s user 0.012s sys 0.007s
I20251025 07:52:47.552492 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 143) took 8398 ms. Trace:
I20251025 07:52:47.552554 2166 rpcz_store.cc:276] 1025 07:52:39.154314 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:39.154372 (+ 58us) service_pool.cc:225] Handling call
1025 07:52:47.552317 (+8397945us) ts_tablet_manager.cc:1151] Deleting tablet 3c4970c8a491460e8b4b1c29fde83e4c
1025 07:52:47.552326 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.552477 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8397714}
I20251025 07:52:47.552739 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 062c0f3e51a3487880cea373f8cf1f54 (table default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de]) successfully deleted
I20251025 07:52:47.552788 2166 tablet_replica.cc:333] T 36c8fcafd5464b4e999276a8062a812a P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:47.552960 2166 raft_consensus.cc:2243] T 36c8fcafd5464b4e999276a8062a812a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:47.553092 2166 raft_consensus.cc:2272] T 36c8fcafd5464b4e999276a8062a812a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.553036 1167 raft_consensus.cc:359] T af743ee344e543a0aa6da3cd7a30e467 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.553176 1167 raft_consensus.cc:385] T af743ee344e543a0aa6da3cd7a30e467 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.553221 1167 raft_consensus.cc:740] T af743ee344e543a0aa6da3cd7a30e467 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.553442 1167 consensus_queue.cc:260] T af743ee344e543a0aa6da3cd7a30e467 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:47.553915 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 3c4970c8a491460e8b4b1c29fde83e4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c4970c8a491460e8b4b1c29fde83e4c already in progress: creating tablet
I20251025 07:52:47.554013 1167 ts_tablet_manager.cc:1434] T af743ee344e543a0aa6da3cd7a30e467 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:47.554275 1167 tablet_bootstrap.cc:492] T 6ce62847e80f48e8807fdbbbdc50803c P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:47.554904 754 tablet_service.cc:1505] Processing CreateTablet for tablet b2ac640e69794a09811bd93aa02a22ea (DEFAULT_TABLE table=default.Table_58b693b694cf4930bedf9f4cc6c52c35 [id=00233e389d0a4f1482425635c1ef4a7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.555315 1017 tablet_service.cc:1505] Processing CreateTablet for tablet b2ac640e69794a09811bd93aa02a22ea (DEFAULT_TABLE table=default.Table_58b693b694cf4930bedf9f4cc6c52c35 [id=00233e389d0a4f1482425635c1ef4a7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.555804 1167 tablet_bootstrap.cc:654] T 6ce62847e80f48e8807fdbbbdc50803c P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.555811 1019 tablet_service.cc:1505] Processing CreateTablet for tablet 11bcdd48155247c5bb935afdeaca6b5d (DEFAULT_TABLE table=default.Table_f4321d2891254856a874d7c806c4ad60 [id=ac8cf5f06fb8420b937f28a0c67bd128]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.557353 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2ac640e69794a09811bd93aa02a22ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.557363 894 tablet_service.cc:1505] Processing CreateTablet for tablet b2ac640e69794a09811bd93aa02a22ea (DEFAULT_TABLE table=default.Table_58b693b694cf4930bedf9f4cc6c52c35 [id=00233e389d0a4f1482425635c1ef4a7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.557691 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2ac640e69794a09811bd93aa02a22ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.559818 2467 consensus_queue.cc:1048] T 2e2e6893a48449969b71ee1c08d1960e P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:47.561403 875 tablet_service.cc:1505] Processing CreateTablet for tablet 11bcdd48155247c5bb935afdeaca6b5d (DEFAULT_TABLE table=default.Table_f4321d2891254856a874d7c806c4ad60 [id=ac8cf5f06fb8420b937f28a0c67bd128]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.561470 743 tablet_service.cc:1505] Processing CreateTablet for tablet 11bcdd48155247c5bb935afdeaca6b5d (DEFAULT_TABLE table=default.Table_f4321d2891254856a874d7c806c4ad60 [id=ac8cf5f06fb8420b937f28a0c67bd128]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.561749 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11bcdd48155247c5bb935afdeaca6b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.561797 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11bcdd48155247c5bb935afdeaca6b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.563601 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 6ee5d9bb565145b8b4a61902bce967e5 (DEFAULT_TABLE table=default.Table_e84098955d80499cafb72da95cf65cb1 [id=a6a986d2bfb84ef5a163f1f37dfbf7ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.566845 2146 ts_tablet_manager.cc:1929] T 04449cc97f404a19b6a1b20dea074f95 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.566953 2146 log.cc:1199] T 04449cc97f404a19b6a1b20dea074f95 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/04449cc97f404a19b6a1b20dea074f95
I20251025 07:52:47.567317 2146 ts_tablet_manager.cc:1950] T 04449cc97f404a19b6a1b20dea074f95 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:47.568115 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 173) took 6151 ms. Trace:
I20251025 07:52:47.568207 2146 rpcz_store.cc:276] 1025 07:52:41.416705 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.416738 (+ 33us) service_pool.cc:225] Handling call
1025 07:52:47.466788 (+6050050us) ts_tablet_manager.cc:1151] Deleting tablet 04449cc97f404a19b6a1b20dea074f95
1025 07:52:47.466802 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.549978 (+ 83176us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.566767 (+ 16789us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.568095 (+ 1328us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6049934}
I20251025 07:52:47.568526 2146 tablet_replica.cc:333] T f43ef51ee45d47a99ec732b49d50d3b4 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:47.568718 2146 raft_consensus.cc:2243] T f43ef51ee45d47a99ec732b49d50d3b4 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:47.568704 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 04449cc97f404a19b6a1b20dea074f95 (table default.Table_b7da40b96d6c4359bed6f898e8cf6275 [id=e041946c093b4c789ce175c736cc230b]) successfully deleted
I20251025 07:52:47.568878 2146 raft_consensus.cc:2272] T f43ef51ee45d47a99ec732b49d50d3b4 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.569564 2146 ts_tablet_manager.cc:1916] T f43ef51ee45d47a99ec732b49d50d3b4 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.570312 2166 ts_tablet_manager.cc:1916] T 36c8fcafd5464b4e999276a8062a812a P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.573837 894 tablet_service.cc:1505] Processing CreateTablet for tablet 6ee5d9bb565145b8b4a61902bce967e5 (DEFAULT_TABLE table=default.Table_e84098955d80499cafb72da95cf65cb1 [id=a6a986d2bfb84ef5a163f1f37dfbf7ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.574173 1169 tablet_bootstrap.cc:492] T 4f0c4f0e8f014e59be9e46d2afe0be9d P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:47.574265 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ee5d9bb565145b8b4a61902bce967e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.574361 1169 ts_tablet_manager.cc:1403] T 4f0c4f0e8f014e59be9e46d2afe0be9d P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.126s user 0.011s sys 0.012s
I20251025 07:52:47.575007 1169 raft_consensus.cc:359] T 4f0c4f0e8f014e59be9e46d2afe0be9d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:47.575183 1169 raft_consensus.cc:385] T 4f0c4f0e8f014e59be9e46d2afe0be9d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.575249 1169 raft_consensus.cc:740] T 4f0c4f0e8f014e59be9e46d2afe0be9d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.575475 1169 consensus_queue.cc:260] T 4f0c4f0e8f014e59be9e46d2afe0be9d P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:47.576092 1169 ts_tablet_manager.cc:1434] T 4f0c4f0e8f014e59be9e46d2afe0be9d P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:47.576622 2166 ts_tablet_manager.cc:1929] T 36c8fcafd5464b4e999276a8062a812a P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.576731 2166 log.cc:1199] T 36c8fcafd5464b4e999276a8062a812a P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/36c8fcafd5464b4e999276a8062a812a
I20251025 07:52:47.577075 2166 ts_tablet_manager.cc:1950] T 36c8fcafd5464b4e999276a8062a812a P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:47.577761 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 144) took 8401 ms. Trace:
I20251025 07:52:47.577854 2166 rpcz_store.cc:276] 1025 07:52:39.176196 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:39.176256 (+ 60us) service_pool.cc:225] Handling call
1025 07:52:47.552740 (+8376484us) ts_tablet_manager.cc:1151] Deleting tablet 36c8fcafd5464b4e999276a8062a812a
1025 07:52:47.552749 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.573436 (+ 20687us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.576540 (+ 3104us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.577742 (+ 1202us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8376267}
I20251025 07:52:47.579023 1021 tablet_service.cc:1505] Processing CreateTablet for tablet ae835c2078624c42a57ca63b7951dc79 (DEFAULT_TABLE table=default.Table_2cd70245aa3d4809a0647e7fa499f62f [id=a003268ec3da425989fa4041699de4d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.579130 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 36c8fcafd5464b4e999276a8062a812a (table default.Table_19a3326e4da345c1bbeb2cf083d28e24 [id=11c088570f6940b1895d1691465c10b7]) successfully deleted
I20251025 07:52:47.579800 875 tablet_service.cc:1505] Processing CreateTablet for tablet ae835c2078624c42a57ca63b7951dc79 (DEFAULT_TABLE table=default.Table_2cd70245aa3d4809a0647e7fa499f62f [id=a003268ec3da425989fa4041699de4d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.579816 749 tablet_service.cc:1505] Processing CreateTablet for tablet 6ee5d9bb565145b8b4a61902bce967e5 (DEFAULT_TABLE table=default.Table_e84098955d80499cafb72da95cf65cb1 [id=a6a986d2bfb84ef5a163f1f37dfbf7ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.580188 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ee5d9bb565145b8b4a61902bce967e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.580196 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae835c2078624c42a57ca63b7951dc79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.580406 748 tablet_service.cc:1505] Processing CreateTablet for tablet ae835c2078624c42a57ca63b7951dc79 (DEFAULT_TABLE table=default.Table_2cd70245aa3d4809a0647e7fa499f62f [id=a003268ec3da425989fa4041699de4d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.580736 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae835c2078624c42a57ca63b7951dc79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.582283 2886 raft_consensus.cc:493] T a0ea1edeca7347fc844faa5e4be1ae94 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:47.582410 2886 raft_consensus.cc:515] T a0ea1edeca7347fc844faa5e4be1ae94 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.582985 2886 leader_election.cc:290] T a0ea1edeca7347fc844faa5e4be1ae94 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:47.583123 1095 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
I20251025 07:52:47.583645 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0ea1edeca7347fc844faa5e4be1ae94" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:47.584144 850 leader_election.cc:343] T a0ea1edeca7347fc844faa5e4be1ae94 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.584662 765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0ea1edeca7347fc844faa5e4be1ae94" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:47.584838 765 raft_consensus.cc:2468] T a0ea1edeca7347fc844faa5e4be1ae94 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:47.585246 847 leader_election.cc:304] T a0ea1edeca7347fc844faa5e4be1ae94 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.585671 2886 raft_consensus.cc:2804] T a0ea1edeca7347fc844faa5e4be1ae94 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:47.585764 2886 raft_consensus.cc:493] T a0ea1edeca7347fc844faa5e4be1ae94 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:47.585836 2886 raft_consensus.cc:3060] T a0ea1edeca7347fc844faa5e4be1ae94 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.589111 878 tablet_service.cc:1505] Processing CreateTablet for tablet 4bb779d6d7e549db89687a5c7f1a16d3 (DEFAULT_TABLE table=default.Table_967b302c337a43aeb6158503e776a618 [id=b59f9132bfc84c3d8ffbca46eadcc015]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.589466 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bb779d6d7e549db89687a5c7f1a16d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.589438 1018 tablet_service.cc:1505] Processing CreateTablet for tablet 4bb779d6d7e549db89687a5c7f1a16d3 (DEFAULT_TABLE table=default.Table_967b302c337a43aeb6158503e776a618 [id=b59f9132bfc84c3d8ffbca46eadcc015]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.590288 2886 raft_consensus.cc:515] T a0ea1edeca7347fc844faa5e4be1ae94 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.591030 748 tablet_service.cc:1505] Processing CreateTablet for tablet 4bb779d6d7e549db89687a5c7f1a16d3 (DEFAULT_TABLE table=default.Table_967b302c337a43aeb6158503e776a618 [id=b59f9132bfc84c3d8ffbca46eadcc015]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.591372 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bb779d6d7e549db89687a5c7f1a16d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.591523 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0ea1edeca7347fc844faa5e4be1ae94" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:47.592166 850 leader_election.cc:343] T a0ea1edeca7347fc844faa5e4be1ae94 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.593238 2886 leader_election.cc:290] T a0ea1edeca7347fc844faa5e4be1ae94 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:47.593298 765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0ea1edeca7347fc844faa5e4be1ae94" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:47.593441 765 raft_consensus.cc:3060] T a0ea1edeca7347fc844faa5e4be1ae94 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.596489 765 raft_consensus.cc:2468] T a0ea1edeca7347fc844faa5e4be1ae94 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:47.596868 847 leader_election.cc:304] T a0ea1edeca7347fc844faa5e4be1ae94 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.601943 1167 tablet_bootstrap.cc:492] T 6ce62847e80f48e8807fdbbbdc50803c P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:47.602113 1167 ts_tablet_manager.cc:1403] T 6ce62847e80f48e8807fdbbbdc50803c P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.048s user 0.005s sys 0.001s
I20251025 07:52:47.602772 2169 ts_tablet_manager.cc:1929] T ce1ebb1ec14b4801930a72765bca1a7d P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.603632 1167 raft_consensus.cc:359] T 6ce62847e80f48e8807fdbbbdc50803c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.603804 1167 raft_consensus.cc:385] T 6ce62847e80f48e8807fdbbbdc50803c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.603852 1167 raft_consensus.cc:740] T 6ce62847e80f48e8807fdbbbdc50803c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.604142 1167 consensus_queue.cc:260] T 6ce62847e80f48e8807fdbbbdc50803c P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.604779 1167 ts_tablet_manager.cc:1434] T 6ce62847e80f48e8807fdbbbdc50803c P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251025 07:52:47.605051 1167 tablet_bootstrap.cc:492] T 72618756f756484e895e885ef12545c8 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:47.605594 756 tablet_service.cc:1505] Processing CreateTablet for tablet fd07452249dd4ca2b64474391de1e298 (DEFAULT_TABLE table=default.Table_7dcc5d1b56a9449e8cd9f533a5bef482 [id=3d0981cd394141deb203b662b84dd66e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.605980 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd07452249dd4ca2b64474391de1e298. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.606321 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 147) took 7463 ms. Trace:
I20251025 07:52:47.606479 2166 rpcz_store.cc:276] 1025 07:52:40.143269 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:40.143325 (+ 56us) service_pool.cc:225] Handling call
1025 07:52:47.606182 (+7462857us) ts_tablet_manager.cc:1151] Deleting tablet ff3daed6c9af47e3976b82b4b9a41086
1025 07:52:47.606191 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.606305 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7462686}
W20251025 07:52:47.606987 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet ff3daed6c9af47e3976b82b4b9a41086 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff3daed6c9af47e3976b82b4b9a41086 already in progress: creating tablet
I20251025 07:52:47.607439 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 159) took 7196 ms. Trace:
I20251025 07:52:47.607560 2166 rpcz_store.cc:276] 1025 07:52:40.410600 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:40.410651 (+ 51us) service_pool.cc:225] Handling call
1025 07:52:47.607277 (+7196626us) ts_tablet_manager.cc:1151] Deleting tablet b02f5c0ab0b24288bb50a3ba0e6bd78a
1025 07:52:47.607287 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.607424 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7196432}
W20251025 07:52:47.608021 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet b02f5c0ab0b24288bb50a3ba0e6bd78a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b02f5c0ab0b24288bb50a3ba0e6bd78a already in progress: creating tablet
I20251025 07:52:47.608548 2166 tablet_replica.cc:333] T a77eb82f26ae43b48de1e85693372d34 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:47.608888 2166 raft_consensus.cc:2243] T a77eb82f26ae43b48de1e85693372d34 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:47.609181 2166 raft_consensus.cc:2272] T a77eb82f26ae43b48de1e85693372d34 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.610018 2166 ts_tablet_manager.cc:1916] T a77eb82f26ae43b48de1e85693372d34 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.610616 1167 tablet_bootstrap.cc:654] T 72618756f756484e895e885ef12545c8 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.611826 894 tablet_service.cc:1505] Processing CreateTablet for tablet fd07452249dd4ca2b64474391de1e298 (DEFAULT_TABLE table=default.Table_7dcc5d1b56a9449e8cd9f533a5bef482 [id=3d0981cd394141deb203b662b84dd66e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.612497 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd07452249dd4ca2b64474391de1e298. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.548789 1166 raft_consensus.cc:359] T dff7f86ae8924685b26011e504e2247b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.606703 1023 tablet_service.cc:1505] Processing CreateTablet for tablet fd07452249dd4ca2b64474391de1e298 (DEFAULT_TABLE table=default.Table_7dcc5d1b56a9449e8cd9f533a5bef482 [id=3d0981cd394141deb203b662b84dd66e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.615516 2169 log.cc:1199] T ce1ebb1ec14b4801930a72765bca1a7d P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/ce1ebb1ec14b4801930a72765bca1a7d
I20251025 07:52:47.614101 1166 raft_consensus.cc:385] T dff7f86ae8924685b26011e504e2247b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.610795 1169 tablet_bootstrap.cc:492] T 141e413d2faa4c67b4c822192e631417 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:47.615549 1010 tablet_service.cc:1505] Processing CreateTablet for tablet 2d4edb6b5e3a4c60909eeb4aed0fc358 (DEFAULT_TABLE table=default.Table_59197c6875cd424f89a3e4a6cba7629f [id=1e0ccaef5ba14c3c86b6a207722e06ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.607937 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2ac640e69794a09811bd93aa02a22ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.616267 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae835c2078624c42a57ca63b7951dc79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.616436 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd07452249dd4ca2b64474391de1e298. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.617705 1169 tablet_bootstrap.cc:654] T 141e413d2faa4c67b4c822192e631417 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.618347 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d4edb6b5e3a4c60909eeb4aed0fc358. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.619592 2169 ts_tablet_manager.cc:1950] T ce1ebb1ec14b4801930a72765bca1a7d P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:47.620273 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 177) took 6168 ms. Trace:
I20251025 07:52:47.620357 2169 rpcz_store.cc:276] 1025 07:52:41.451737 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.452510 (+ 773us) service_pool.cc:225] Handling call
1025 07:52:47.540883 (+6088373us) ts_tablet_manager.cc:1151] Deleting tablet ce1ebb1ec14b4801930a72765bca1a7d
1025 07:52:47.540893 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.557370 (+ 16477us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.602668 (+ 45298us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.620255 (+ 17587us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6088239}
I20251025 07:52:47.622051 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bb779d6d7e549db89687a5c7f1a16d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.622222 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ee5d9bb565145b8b4a61902bce967e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.622474 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet ce1ebb1ec14b4801930a72765bca1a7d (table default.Table_d5abcbf8c8c741a2b40880a3467f2bcd [id=eab2657210e74ee6ac6f59544c9ba12e]) successfully deleted
I20251025 07:52:47.623059 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11bcdd48155247c5bb935afdeaca6b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.631307 756 tablet_service.cc:1505] Processing CreateTablet for tablet 2d4edb6b5e3a4c60909eeb4aed0fc358 (DEFAULT_TABLE table=default.Table_59197c6875cd424f89a3e4a6cba7629f [id=1e0ccaef5ba14c3c86b6a207722e06ca]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:47.631568 983 consensus_peers.cc:597] T 04449cc97f404a19b6a1b20dea074f95 P db42be7ad3c741a0aa427f27e245993b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04449cc97f404a19b6a1b20dea074f95. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:47.631877 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d4edb6b5e3a4c60909eeb4aed0fc358. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.636376 1023 tablet_service.cc:1505] Processing CreateTablet for tablet e330e1b53bed4e7cace1c4fb7fa6b565 (DEFAULT_TABLE table=default.Table_a9026f5e095c4063b86d4c544c686e22 [id=7944225d3ce7439b881bfb9289d9f040]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.636961 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e330e1b53bed4e7cace1c4fb7fa6b565. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.616058 875 tablet_service.cc:1505] Processing CreateTablet for tablet 2d4edb6b5e3a4c60909eeb4aed0fc358 (DEFAULT_TABLE table=default.Table_59197c6875cd424f89a3e4a6cba7629f [id=1e0ccaef5ba14c3c86b6a207722e06ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.638315 1166 raft_consensus.cc:740] T dff7f86ae8924685b26011e504e2247b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.638676 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d4edb6b5e3a4c60909eeb4aed0fc358. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.640309 2888 raft_consensus.cc:493] T 075f0a151e024220b51ba7fcaecdcc3d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:47.640450 2888 raft_consensus.cc:515] T 075f0a151e024220b51ba7fcaecdcc3d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.641073 2888 leader_election.cc:290] T 075f0a151e024220b51ba7fcaecdcc3d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:47.636541 2169 tablet_replica.cc:333] T ebc824ec4c60411dab1a2ea4a385148e P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:47.641947 2169 raft_consensus.cc:2243] T ebc824ec4c60411dab1a2ea4a385148e P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:47.638590 1166 consensus_queue.cc:260] T dff7f86ae8924685b26011e504e2247b P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.643070 1166 ts_tablet_manager.cc:1434] T dff7f86ae8924685b26011e504e2247b P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.096s user 0.002s sys 0.000s
I20251025 07:52:47.641953 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "075f0a151e024220b51ba7fcaecdcc3d" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:47.643646 2169 raft_consensus.cc:2272] T ebc824ec4c60411dab1a2ea4a385148e P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:47.643918 712 leader_election.cc:343] T 075f0a151e024220b51ba7fcaecdcc3d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.646957 2169 ts_tablet_manager.cc:1916] T ebc824ec4c60411dab1a2ea4a385148e P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.650787 878 tablet_service.cc:1505] Processing CreateTablet for tablet f255cf2adf7c42c79903e28259ba9ad9 (DEFAULT_TABLE table=default.Table_913d48d5976244519b5f452c92b02b14 [id=816dfbbc853d4fc2be404f7865ce1293]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.651170 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f255cf2adf7c42c79903e28259ba9ad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.652429 2166 ts_tablet_manager.cc:1929] T a77eb82f26ae43b48de1e85693372d34 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:47.652546 2166 log.cc:1199] T a77eb82f26ae43b48de1e85693372d34 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/a77eb82f26ae43b48de1e85693372d34
I20251025 07:52:47.652832 2166 ts_tablet_manager.cc:1950] T a77eb82f26ae43b48de1e85693372d34 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:47.653537 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 160) took 7221 ms. Trace:
I20251025 07:52:47.654652 754 tablet_service.cc:1505] Processing CreateTablet for tablet e330e1b53bed4e7cace1c4fb7fa6b565 (DEFAULT_TABLE table=default.Table_a9026f5e095c4063b86d4c544c686e22 [id=7944225d3ce7439b881bfb9289d9f040]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.656533 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e330e1b53bed4e7cace1c4fb7fa6b565. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.658676 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5]). Expected version 1 got 0
I20251025 07:52:47.655112 748 tablet_service.cc:1505] Processing CreateTablet for tablet f255cf2adf7c42c79903e28259ba9ad9 (DEFAULT_TABLE table=default.Table_913d48d5976244519b5f452c92b02b14 [id=816dfbbc853d4fc2be404f7865ce1293]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.655539 2166 rpcz_store.cc:276] 1025 07:52:40.432521 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:40.432580 (+ 59us) service_pool.cc:225] Handling call
1025 07:52:47.608485 (+7175905us) ts_tablet_manager.cc:1151] Deleting tablet a77eb82f26ae43b48de1e85693372d34
1025 07:52:47.608493 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.635703 (+ 27210us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.652333 (+ 16630us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.653517 (+ 1184us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7175728}
I20251025 07:52:47.661690 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f255cf2adf7c42c79903e28259ba9ad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.662364 1018 tablet_service.cc:1505] Processing CreateTablet for tablet 8f05894c45154cd0b65f5b9bcef8cd4c (DEFAULT_TABLE table=default.Table_ac4beca0c3bd4cf29565faf5323d9ff7 [id=857bcd1dbe2e477bbc0d77f79a58f7f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.662717 1019 tablet_service.cc:1505] Processing CreateTablet for tablet f255cf2adf7c42c79903e28259ba9ad9 (DEFAULT_TABLE table=default.Table_913d48d5976244519b5f452c92b02b14 [id=816dfbbc853d4fc2be404f7865ce1293]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.662921 1023 tablet_service.cc:1505] Processing CreateTablet for tablet 978abb6c12ef495d97fc8b5c46eddd92 (DEFAULT_TABLE table=default.Table_f34197ad40904305af4d5568e865489b [id=528001b076f240e68880575225d86566]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.665258 1166 tablet_bootstrap.cc:492] T c187a40f8ff14c6a8c184fbeb030ad13 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:47.665508 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f05894c45154cd0b65f5b9bcef8cd4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.665834 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f255cf2adf7c42c79903e28259ba9ad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.666040 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 978abb6c12ef495d97fc8b5c46eddd92. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.667169 894 tablet_service.cc:1505] Processing CreateTablet for tablet e330e1b53bed4e7cace1c4fb7fa6b565 (DEFAULT_TABLE table=default.Table_a9026f5e095c4063b86d4c544c686e22 [id=7944225d3ce7439b881bfb9289d9f040]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.667593 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e330e1b53bed4e7cace1c4fb7fa6b565. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.667670 1166 tablet_bootstrap.cc:654] T c187a40f8ff14c6a8c184fbeb030ad13 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.663697 2166 tablet_replica.cc:333] T 6897405bbc1c47a4ab0aa3af80e60e3d P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:47.655757 743 tablet_service.cc:1505] Processing CreateTablet for tablet 8f05894c45154cd0b65f5b9bcef8cd4c (DEFAULT_TABLE table=default.Table_ac4beca0c3bd4cf29565faf5323d9ff7 [id=857bcd1dbe2e477bbc0d77f79a58f7f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.670933 1167 tablet_bootstrap.cc:492] T 72618756f756484e895e885ef12545c8 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:47.674819 2166 raft_consensus.cc:2243] T 6897405bbc1c47a4ab0aa3af80e60e3d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:47.674878 1167 ts_tablet_manager.cc:1403] T 72618756f756484e895e885ef12545c8 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.070s user 0.004s sys 0.002s
I20251025 07:52:47.675011 2166 raft_consensus.cc:2272] T 6897405bbc1c47a4ab0aa3af80e60e3d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.675168 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f05894c45154cd0b65f5b9bcef8cd4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.675537 2166 ts_tablet_manager.cc:1916] T 6897405bbc1c47a4ab0aa3af80e60e3d P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.675647 1167 raft_consensus.cc:359] T 72618756f756484e895e885ef12545c8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.675822 1167 raft_consensus.cc:385] T 72618756f756484e895e885ef12545c8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.675905 1167 raft_consensus.cc:740] T 72618756f756484e895e885ef12545c8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.676138 1167 consensus_queue.cc:260] T 72618756f756484e895e885ef12545c8 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.676836 1167 ts_tablet_manager.cc:1434] T 72618756f756484e895e885ef12545c8 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:47.677109 1167 tablet_bootstrap.cc:492] T e4da89f863cc43f3bf6df908bb9dc6bf P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:47.679606 1167 tablet_bootstrap.cc:654] T e4da89f863cc43f3bf6df908bb9dc6bf P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.681479 1169 tablet_bootstrap.cc:492] T 141e413d2faa4c67b4c822192e631417 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:47.681730 1169 ts_tablet_manager.cc:1403] T 141e413d2faa4c67b4c822192e631417 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.071s user 0.008s sys 0.000s
I20251025 07:52:47.682909 1169 raft_consensus.cc:359] T 141e413d2faa4c67b4c822192e631417 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.683224 1169 raft_consensus.cc:385] T 141e413d2faa4c67b4c822192e631417 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.683339 1169 raft_consensus.cc:740] T 141e413d2faa4c67b4c822192e631417 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.683708 1169 consensus_queue.cc:260] T 141e413d2faa4c67b4c822192e631417 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.686720 1018 tablet_service.cc:1505] Processing CreateTablet for tablet 3760348dbe284c41a01a4fe479a2103f (DEFAULT_TABLE table=default.Table_088aad2a46b041bfa29ed8e6929760b1 [id=834669c54dae449790818684fd8659bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.687161 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3760348dbe284c41a01a4fe479a2103f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.689039 2888 raft_consensus.cc:493] T 5dab439bc50d4e869548154e1d222ad6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:47.689198 2888 raft_consensus.cc:515] T 5dab439bc50d4e869548154e1d222ad6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.689976 2888 leader_election.cc:290] T 5dab439bc50d4e869548154e1d222ad6 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:47.690251 2886 raft_consensus.cc:2804] T a0ea1edeca7347fc844faa5e4be1ae94 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:47.690383 2886 raft_consensus.cc:697] T a0ea1edeca7347fc844faa5e4be1ae94 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:47.690642 2886 consensus_queue.cc:237] T a0ea1edeca7347fc844faa5e4be1ae94 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.691051 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5dab439bc50d4e869548154e1d222ad6" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:47.692130 883 tablet_service.cc:1505] Processing CreateTablet for tablet 3760348dbe284c41a01a4fe479a2103f (DEFAULT_TABLE table=default.Table_088aad2a46b041bfa29ed8e6929760b1 [id=834669c54dae449790818684fd8659bc]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:47.692247 712 leader_election.cc:343] T 5dab439bc50d4e869548154e1d222ad6 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.692471 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3760348dbe284c41a01a4fe479a2103f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.692778 1169 ts_tablet_manager.cc:1434] T 141e413d2faa4c67b4c822192e631417 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20251025 07:52:47.693076 1169 tablet_bootstrap.cc:492] T 7e6544202d4f489d82f22d6073ef240d P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:47.694489 2166 ts_tablet_manager.cc:1929] T 6897405bbc1c47a4ab0aa3af80e60e3d P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 07:52:47.694600 2166 log.cc:1199] T 6897405bbc1c47a4ab0aa3af80e60e3d P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/6897405bbc1c47a4ab0aa3af80e60e3d
I20251025 07:52:47.694859 1169 tablet_bootstrap.cc:654] T 7e6544202d4f489d82f22d6073ef240d P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.694938 2166 ts_tablet_manager.cc:1950] T 6897405bbc1c47a4ab0aa3af80e60e3d P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:47.661700 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet a77eb82f26ae43b48de1e85693372d34 (table default.Table_bcd5afa1072f47898d67504fc72b84c4 [id=3d6f0426d32f481ba8152eaacb298784]) successfully deleted
I20251025 07:52:47.695688 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3]). Expected version 1 got 0
I20251025 07:52:47.696019 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_fa83d0528b4b481989bfc0ca35c6de44 [id=bf77b5ff6f1048dd996d948437f9ff5e]). Expected version 3 got 0
I20251025 07:52:47.696458 749 tablet_service.cc:1505] Processing CreateTablet for tablet 978abb6c12ef495d97fc8b5c46eddd92 (DEFAULT_TABLE table=default.Table_f34197ad40904305af4d5568e865489b [id=528001b076f240e68880575225d86566]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.696463 745 tablet_service.cc:1505] Processing CreateTablet for tablet 3760348dbe284c41a01a4fe479a2103f (DEFAULT_TABLE table=default.Table_088aad2a46b041bfa29ed8e6929760b1 [id=834669c54dae449790818684fd8659bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.697438 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f]). Expected version 1 got 0
I20251025 07:52:47.697981 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7]). Expected version 1 got 0
I20251025 07:52:47.698395 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 872cb7512d104827b1f1f186d33a4a44 (table default.Table_d5c65dce8b1b4214ba519d964012bfc0 [id=d4be274b36a845a99a4b96b8846248fd]). Expected version 1 got 0
I20251025 07:52:47.698715 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 978abb6c12ef495d97fc8b5c46eddd92. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.698989 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3760348dbe284c41a01a4fe479a2103f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.698957 878 tablet_service.cc:1505] Processing CreateTablet for tablet 978abb6c12ef495d97fc8b5c46eddd92 (DEFAULT_TABLE table=default.Table_f34197ad40904305af4d5568e865489b [id=528001b076f240e68880575225d86566]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.699275 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 978abb6c12ef495d97fc8b5c46eddd92. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.699347 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 161) took 7216 ms. Trace:
I20251025 07:52:47.702255 2166 rpcz_store.cc:276] 1025 07:52:40.482957 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:40.483024 (+ 67us) service_pool.cc:225] Handling call
1025 07:52:47.661470 (+7178446us) ts_tablet_manager.cc:1151] Deleting tablet 6897405bbc1c47a4ab0aa3af80e60e3d
1025 07:52:47.661803 (+ 333us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.678989 (+ 17186us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.694376 (+ 15387us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.699329 (+ 4953us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7178256}
I20251025 07:52:47.702775 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 6897405bbc1c47a4ab0aa3af80e60e3d (table default.Table_92c0120dd29348618040150dedb14c43 [id=ea892976886644d8aa025ae2edef718d]) successfully deleted
W20251025 07:52:47.703497 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:46231, user_credentials={real_user=slave}} blocked reactor thread for 106654us
I20251025 07:52:47.703785 2166 tablet_replica.cc:333] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:47.704030 2166 raft_consensus.cc:2243] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:47.705618 875 tablet_service.cc:1505] Processing CreateTablet for tablet 8f05894c45154cd0b65f5b9bcef8cd4c (DEFAULT_TABLE table=default.Table_ac4beca0c3bd4cf29565faf5323d9ff7 [id=857bcd1dbe2e477bbc0d77f79a58f7f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:47.706023 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f05894c45154cd0b65f5b9bcef8cd4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:47.708181 905 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "075f0a151e024220b51ba7fcaecdcc3d" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:47.708822 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5dab439bc50d4e869548154e1d222ad6" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:47.709769 2146 ts_tablet_manager.cc:1929] T f43ef51ee45d47a99ec732b49d50d3b4 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.709883 2146 log.cc:1199] T f43ef51ee45d47a99ec732b49d50d3b4 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/f43ef51ee45d47a99ec732b49d50d3b4
I20251025 07:52:47.710343 2146 ts_tablet_manager.cc:1950] T f43ef51ee45d47a99ec732b49d50d3b4 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
W20251025 07:52:47.710654 710 leader_election.cc:343] T 075f0a151e024220b51ba7fcaecdcc3d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.710793 710 leader_election.cc:304] T 075f0a151e024220b51ba7fcaecdcc3d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.711166 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 174) took 6294 ms. Trace:
I20251025 07:52:47.711242 2146 rpcz_store.cc:276] 1025 07:52:41.416839 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.416882 (+ 43us) service_pool.cc:225] Handling call
1025 07:52:47.568476 (+6151594us) ts_tablet_manager.cc:1151] Deleting tablet f43ef51ee45d47a99ec732b49d50d3b4
1025 07:52:47.568487 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.628949 (+ 60462us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.709676 (+ 80727us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.711149 (+ 1473us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6151485}
I20251025 07:52:47.711469 2888 raft_consensus.cc:2749] T 075f0a151e024220b51ba7fcaecdcc3d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:47.711838 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet f43ef51ee45d47a99ec732b49d50d3b4 (table default.Table_0b0bd6b71b92454fb8cdf967ffd8349e [id=e9b21659de7d4276aa1f9effb7076eb4]) successfully deleted
I20251025 07:52:47.712165 2146 tablet_replica.cc:333] T 80ee75935d7e461d9d4066fac9819bef P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
W20251025 07:52:47.712203 710 leader_election.cc:343] T 5dab439bc50d4e869548154e1d222ad6 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.712294 710 leader_election.cc:304] T 5dab439bc50d4e869548154e1d222ad6 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.712347 2146 raft_consensus.cc:2243] T 80ee75935d7e461d9d4066fac9819bef P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:47.712486 2146 raft_consensus.cc:2272] T 80ee75935d7e461d9d4066fac9819bef P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.712764 2888 raft_consensus.cc:2749] T 5dab439bc50d4e869548154e1d222ad6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:47.713033 2146 ts_tablet_manager.cc:1916] T 80ee75935d7e461d9d4066fac9819bef P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.713075 2671 raft_consensus.cc:493] T 627a9cfd398346f2aa34c8932a11b5a8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:47.713207 2671 raft_consensus.cc:515] T 627a9cfd398346f2aa34c8932a11b5a8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.713791 2671 leader_election.cc:290] T 627a9cfd398346f2aa34c8932a11b5a8 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:47.714154 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627a9cfd398346f2aa34c8932a11b5a8" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:47.714516 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627a9cfd398346f2aa34c8932a11b5a8" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:47.715003 982 leader_election.cc:304] T 627a9cfd398346f2aa34c8932a11b5a8 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:47.715351 2671 raft_consensus.cc:2749] T 627a9cfd398346f2aa34c8932a11b5a8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:47.720144 2826 raft_consensus.cc:493] T 1b5ac7e845a24b2eb5b2429d47361232 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:47.720271 2826 raft_consensus.cc:515] T 1b5ac7e845a24b2eb5b2429d47361232 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.720829 2826 leader_election.cc:290] T 1b5ac7e845a24b2eb5b2429d47361232 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:47.721522 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b5ac7e845a24b2eb5b2429d47361232" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:47.722111 850 leader_election.cc:343] T 1b5ac7e845a24b2eb5b2429d47361232 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.723662 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b5ac7e845a24b2eb5b2429d47361232" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:47.723896 772 raft_consensus.cc:2468] T 1b5ac7e845a24b2eb5b2429d47361232 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:47.729770 2169 ts_tablet_manager.cc:1929] T ebc824ec4c60411dab1a2ea4a385148e P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.729983 2169 log.cc:1199] T ebc824ec4c60411dab1a2ea4a385148e P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/ebc824ec4c60411dab1a2ea4a385148e
I20251025 07:52:47.730499 2169 ts_tablet_manager.cc:1950] T ebc824ec4c60411dab1a2ea4a385148e P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:47.731120 1166 tablet_bootstrap.cc:492] T c187a40f8ff14c6a8c184fbeb030ad13 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:47.731258 1166 ts_tablet_manager.cc:1403] T c187a40f8ff14c6a8c184fbeb030ad13 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.066s user 0.002s sys 0.001s
I20251025 07:52:47.731941 1166 raft_consensus.cc:359] T c187a40f8ff14c6a8c184fbeb030ad13 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.732106 1166 raft_consensus.cc:385] T c187a40f8ff14c6a8c184fbeb030ad13 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.732156 1166 raft_consensus.cc:740] T c187a40f8ff14c6a8c184fbeb030ad13 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.732352 1167 tablet_bootstrap.cc:492] T e4da89f863cc43f3bf6df908bb9dc6bf P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:47.732367 1166 consensus_queue.cc:260] T c187a40f8ff14c6a8c184fbeb030ad13 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.732597 1167 ts_tablet_manager.cc:1403] T e4da89f863cc43f3bf6df908bb9dc6bf P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.056s user 0.000s sys 0.005s
I20251025 07:52:47.732824 1166 ts_tablet_manager.cc:1434] T c187a40f8ff14c6a8c184fbeb030ad13 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251025 07:52:47.733750 1167 raft_consensus.cc:359] T e4da89f863cc43f3bf6df908bb9dc6bf P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.733986 1167 raft_consensus.cc:385] T e4da89f863cc43f3bf6df908bb9dc6bf P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.734081 1167 raft_consensus.cc:740] T e4da89f863cc43f3bf6df908bb9dc6bf P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.734362 1167 consensus_queue.cc:260] T e4da89f863cc43f3bf6df908bb9dc6bf P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.734885 1167 ts_tablet_manager.cc:1434] T e4da89f863cc43f3bf6df908bb9dc6bf P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:47.734969 1166 tablet_bootstrap.cc:492] T fa826b85f0c94150a17d4e417dd3d62f P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:47.735258 1167 tablet_bootstrap.cc:492] T d0240d20f9904b4bbc8bf7f0f05e8850 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:47.736683 1167 tablet_bootstrap.cc:654] T d0240d20f9904b4bbc8bf7f0f05e8850 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.736956 1166 tablet_bootstrap.cc:654] T fa826b85f0c94150a17d4e417dd3d62f P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.737104 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 04d27d54fdb7483b869408af65ff85c9 (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]). Expected version 2 got 0
I20251025 07:52:47.738703 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet aa9d9ab8073945d28d52fb02213f201a (table default.Table_4791672658f043f6b97b9320a531ebb5 [id=dfd20cf9c3e541c4abe23d59e0aa7f2a]). Expected version 1 got 0
I20251025 07:52:47.739264 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet f70091024c034c118fb86578db19a31c (table default.Table_b7da40b96d6c4359bed6f898e8cf6275 [id=e041946c093b4c789ce175c736cc230b]). Expected version 2 got 0
I20251025 07:52:47.739607 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 5dab439bc50d4e869548154e1d222ad6 (table default.Table_fb9def2302784c999bf0f5707c60eb4c [id=45325c188eac4d34bb016d01de091f87]). Expected version 3 got 0
I20251025 07:52:47.740382 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 5fc092c2bb2d41ac8c5b4fc07460c8ec (table default.Table_cd20e202397645f8821dcde13b5c0ce3 [id=630c274339664b9db18ab4ef7c29c16f]). Expected version 2 got 0
I20251025 07:52:47.740649 847 leader_election.cc:304] T 1b5ac7e845a24b2eb5b2429d47361232 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.740878 2826 raft_consensus.cc:2804] T 1b5ac7e845a24b2eb5b2429d47361232 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:47.740948 2826 raft_consensus.cc:493] T 1b5ac7e845a24b2eb5b2429d47361232 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:47.741019 2826 raft_consensus.cc:3060] T 1b5ac7e845a24b2eb5b2429d47361232 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.743796 2826 raft_consensus.cc:515] T 1b5ac7e845a24b2eb5b2429d47361232 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.744647 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b5ac7e845a24b2eb5b2429d47361232" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:47.744251 2886 raft_consensus.cc:493] T 1068f566477e4c6db7dde566fff4c2fb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:47.745260 2886 raft_consensus.cc:515] T 1068f566477e4c6db7dde566fff4c2fb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.745736 2886 leader_election.cc:290] T 1068f566477e4c6db7dde566fff4c2fb P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
W20251025 07:52:47.745126 850 leader_election.cc:343] T 1b5ac7e845a24b2eb5b2429d47361232 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.746387 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1068f566477e4c6db7dde566fff4c2fb" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:47.746639 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 176) took 6295 ms. Trace:
W20251025 07:52:47.746795 850 leader_election.cc:343] T 1068f566477e4c6db7dde566fff4c2fb P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.744355 2826 leader_election.cc:290] T 1b5ac7e845a24b2eb5b2429d47361232 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:47.748863 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]). Expected version 3 got 0
I20251025 07:52:47.749044 2169 rpcz_store.cc:276] 1025 07:52:41.451648 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.452669 (+ 1021us) service_pool.cc:225] Handling call
1025 07:52:47.636491 (+6183822us) ts_tablet_manager.cc:1151] Deleting tablet ebc824ec4c60411dab1a2ea4a385148e
1025 07:52:47.636501 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.668776 (+ 32275us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.729644 (+ 60868us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.746616 (+ 16972us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6183669}
I20251025 07:52:47.749516 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 22f7db660616457b83b24e947a0b0bae (table default.Table_4a86191163f2408f9bcc795cd8d1bbf6 [id=bb316a7e000748ca902a9bfb494802d7]). Expected version 1 got 0
I20251025 07:52:47.749593 2826 raft_consensus.cc:493] T 2b594d4718344aafbdaec34040695589 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:47.749749 2826 raft_consensus.cc:515] T 2b594d4718344aafbdaec34040695589 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.750445 2826 leader_election.cc:290] T 2b594d4718344aafbdaec34040695589 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:47.750720 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b594d4718344aafbdaec34040695589" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:47.751150 850 leader_election.cc:343] T 2b594d4718344aafbdaec34040695589 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.744184 2166 raft_consensus.cc:2272] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.745298 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b5ac7e845a24b2eb5b2429d47361232" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:47.751544 1166 tablet_bootstrap.cc:492] T fa826b85f0c94150a17d4e417dd3d62f P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:47.751622 2169 tablet_replica.cc:333] T d9bfd7c0c3ad4e5b98b6708793d61d41 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:47.751648 772 raft_consensus.cc:3060] T 1b5ac7e845a24b2eb5b2429d47361232 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.751690 1166 ts_tablet_manager.cc:1403] T fa826b85f0c94150a17d4e417dd3d62f P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.001s
I20251025 07:52:47.751981 2166 ts_tablet_manager.cc:1916] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.752182 2169 raft_consensus.cc:2243] T d9bfd7c0c3ad4e5b98b6708793d61d41 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:47.752357 1166 raft_consensus.cc:359] T fa826b85f0c94150a17d4e417dd3d62f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.752501 1166 raft_consensus.cc:385] T fa826b85f0c94150a17d4e417dd3d62f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.752552 1166 raft_consensus.cc:740] T fa826b85f0c94150a17d4e417dd3d62f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.752648 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]). Expected version 1 got 0
I20251025 07:52:47.752775 1166 consensus_queue.cc:260] T fa826b85f0c94150a17d4e417dd3d62f P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.753073 2169 raft_consensus.cc:2272] T d9bfd7c0c3ad4e5b98b6708793d61d41 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:47.753227 442 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_fa83d0528b4b481989bfc0ca35c6de44 [id=bf77b5ff6f1048dd996d948437f9ff5e], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet d46673a283fe42b78ddeb99fbd2678eb consensus state has no leader
I20251025 07:52:47.753304 1166 ts_tablet_manager.cc:1434] T fa826b85f0c94150a17d4e417dd3d62f P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:47.753538 1166 tablet_bootstrap.cc:492] T c501ef5d70e742598c1a902efe47feb1 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:47.754194 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 0325d462441349ef81841f1b0cab2a20 (table default.Table_750ecb905839434fa9a1aa72f87394be [id=9ce3ba77c56a4ef09cb3b40ea0e488f6]). Expected version 1 got 0
I20251025 07:52:47.754871 1166 tablet_bootstrap.cc:654] T c501ef5d70e742598c1a902efe47feb1 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.755014 772 raft_consensus.cc:2468] T 1b5ac7e845a24b2eb5b2429d47361232 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:47.755993 1169 tablet_bootstrap.cc:492] T 7e6544202d4f489d82f22d6073ef240d P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:47.756165 1169 ts_tablet_manager.cc:1403] T 7e6544202d4f489d82f22d6073ef240d P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.000s
I20251025 07:52:47.757004 1169 raft_consensus.cc:359] T 7e6544202d4f489d82f22d6073ef240d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.757225 1169 raft_consensus.cc:385] T 7e6544202d4f489d82f22d6073ef240d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.757313 1169 raft_consensus.cc:740] T 7e6544202d4f489d82f22d6073ef240d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.757570 1169 consensus_queue.cc:260] T 7e6544202d4f489d82f22d6073ef240d P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.757954 2888 raft_consensus.cc:493] T 67cf96a7dd3f42e19b86746c87e689f6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:47.758095 2888 raft_consensus.cc:515] T 67cf96a7dd3f42e19b86746c87e689f6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.758352 2166 ts_tablet_manager.cc:1929] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.758421 1169 ts_tablet_manager.cc:1434] T 7e6544202d4f489d82f22d6073ef240d P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:47.758446 2166 log.cc:1199] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/fa998b9ca6724c4cb1eb68a3eecb0b11
I20251025 07:52:47.758683 1169 tablet_bootstrap.cc:492] T 682c1e22d7764dce862d970b33b88cb3 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:47.758728 2888 leader_election.cc:290] T 67cf96a7dd3f42e19b86746c87e689f6 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:47.758775 2166 ts_tablet_manager.cc:1950] T fa998b9ca6724c4cb1eb68a3eecb0b11 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:47.759503 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 162) took 6393 ms. Trace:
I20251025 07:52:47.759508 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67cf96a7dd3f42e19b86746c87e689f6" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:47.759590 2166 rpcz_store.cc:276] 1025 07:52:41.366524 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.366595 (+ 71us) service_pool.cc:225] Handling call
1025 07:52:47.703737 (+6337142us) ts_tablet_manager.cc:1151] Deleting tablet fa998b9ca6724c4cb1eb68a3eecb0b11
1025 07:52:47.703747 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.755220 (+ 51473us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.758268 (+ 3048us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.759487 (+ 1219us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6336963}
W20251025 07:52:47.759974 712 leader_election.cc:343] T 67cf96a7dd3f42e19b86746c87e689f6 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.760016 1169 tablet_bootstrap.cc:654] T 682c1e22d7764dce862d970b33b88cb3 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.763288 847 leader_election.cc:304] T 1b5ac7e845a24b2eb5b2429d47361232 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.763613 2166 tablet_replica.cc:333] T 5b6383c99e9e4c90a374da9ccd830a6f P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:47.763568 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67cf96a7dd3f42e19b86746c87e689f6" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:47.763695 2686 raft_consensus.cc:2804] T 1b5ac7e845a24b2eb5b2429d47361232 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:47.763804 2686 raft_consensus.cc:697] T 1b5ac7e845a24b2eb5b2429d47361232 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:47.763856 2166 raft_consensus.cc:2243] T 5b6383c99e9e4c90a374da9ccd830a6f P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:47.763991 2166 raft_consensus.cc:2272] T 5b6383c99e9e4c90a374da9ccd830a6f P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.764068 2686 consensus_queue.cc:237] T 1b5ac7e845a24b2eb5b2429d47361232 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.756984 2169 ts_tablet_manager.cc:1916] T d9bfd7c0c3ad4e5b98b6708793d61d41 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.766609 2166 ts_tablet_manager.cc:1916] T 5b6383c99e9e4c90a374da9ccd830a6f P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.769989 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1068f566477e4c6db7dde566fff4c2fb" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:47.770633 772 raft_consensus.cc:2468] T 1068f566477e4c6db7dde566fff4c2fb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:47.772377 2671 consensus_queue.cc:1048] T 2e2e6893a48449969b71ee1c08d1960e P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:47.772842 2166 ts_tablet_manager.cc:1929] T 5b6383c99e9e4c90a374da9ccd830a6f P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.772980 2166 log.cc:1199] T 5b6383c99e9e4c90a374da9ccd830a6f P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/5b6383c99e9e4c90a374da9ccd830a6f
I20251025 07:52:47.773402 2166 ts_tablet_manager.cc:1950] T 5b6383c99e9e4c90a374da9ccd830a6f P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:47.778956 1166 tablet_bootstrap.cc:492] T c501ef5d70e742598c1a902efe47feb1 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:47.760253 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet fa998b9ca6724c4cb1eb68a3eecb0b11 (table default.Table_4976d4f5d05540258d144751d129bdb8 [id=8cde66661cf749919839da894ed4dc05]) successfully deleted
I20251025 07:52:47.779172 1166 ts_tablet_manager.cc:1403] T c501ef5d70e742598c1a902efe47feb1 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20251025 07:52:47.779515 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 3a0f50c8000144a4acb5219484b01800 (table default.Table_c62e0dc1ae504f9bb912013506ff3031 [id=6f395892b09240fea02482b3199cfb5e]). Expected version 1 got 0
I20251025 07:52:47.779819 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525]). Expected version 1 got 0
I20251025 07:52:47.779964 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_67b99749f3bf43228b40d8b9807e69b4 [id=fc8180c09d7b4e74b2e161f7acb8fe0e]). Expected version 2 got 0
I20251025 07:52:47.779891 1166 raft_consensus.cc:359] T c501ef5d70e742598c1a902efe47feb1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.780046 1166 raft_consensus.cc:385] T c501ef5d70e742598c1a902efe47feb1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.780090 1166 raft_consensus.cc:740] T c501ef5d70e742598c1a902efe47feb1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.780156 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 8707b93d491e4e4386d148087a2f0870 (table default.Table_799dcb1cef2f4fe6b9b8f17f9f1026c5 [id=59840a567e5244e8a9024cbad42471e4]). Expected version 1 got 0
I20251025 07:52:47.780283 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236]). Expected version 1 got 0
I20251025 07:52:47.780308 1166 consensus_queue.cc:260] T c501ef5d70e742598c1a902efe47feb1 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.780752 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 755f0d6c87044e20a8020f8410940123 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 4 got 0
I20251025 07:52:47.780776 1166 ts_tablet_manager.cc:1434] T c501ef5d70e742598c1a902efe47feb1 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:47.781035 1166 tablet_bootstrap.cc:492] T 6a33cfb2f2de48d4a5e9e39730d4900e P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:47.782173 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 163) took 6293 ms. Trace:
I20251025 07:52:47.782449 1166 tablet_bootstrap.cc:654] T 6a33cfb2f2de48d4a5e9e39730d4900e P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.782285 2166 rpcz_store.cc:276] 1025 07:52:41.480766 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.480830 (+ 64us) service_pool.cc:225] Handling call
1025 07:52:47.763567 (+6282737us) ts_tablet_manager.cc:1151] Deleting tablet 5b6383c99e9e4c90a374da9ccd830a6f
1025 07:52:47.763577 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.769702 (+ 6125us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.772735 (+ 3033us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.774122 (+ 1387us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6282542}
I20251025 07:52:47.782831 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 5b6383c99e9e4c90a374da9ccd830a6f (table default.Table_32f8a762ec25446dac28e2729d6879bf [id=a13b2a42afca460595cd872ab2cf9ea0]) successfully deleted
I20251025 07:52:47.783262 2166 tablet_replica.cc:333] T 80fe3255ed614270bd8352a2853fbc43 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:47.783517 2166 raft_consensus.cc:2243] T 80fe3255ed614270bd8352a2853fbc43 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:47.783746 2166 raft_consensus.cc:2272] T 80fe3255ed614270bd8352a2853fbc43 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.784308 2166 ts_tablet_manager.cc:1916] T 80fe3255ed614270bd8352a2853fbc43 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:47.790684 710 leader_election.cc:343] T 67cf96a7dd3f42e19b86746c87e689f6 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.790822 710 leader_election.cc:304] T 67cf96a7dd3f42e19b86746c87e689f6 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.791175 2888 raft_consensus.cc:2749] T 67cf96a7dd3f42e19b86746c87e689f6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:47.795171 1166 tablet_bootstrap.cc:492] T 6a33cfb2f2de48d4a5e9e39730d4900e P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:47.795325 1166 ts_tablet_manager.cc:1403] T 6a33cfb2f2de48d4a5e9e39730d4900e P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20251025 07:52:47.795977 1166 raft_consensus.cc:359] T 6a33cfb2f2de48d4a5e9e39730d4900e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.796145 1166 raft_consensus.cc:385] T 6a33cfb2f2de48d4a5e9e39730d4900e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.796197 1166 raft_consensus.cc:740] T 6a33cfb2f2de48d4a5e9e39730d4900e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.796420 1166 consensus_queue.cc:260] T 6a33cfb2f2de48d4a5e9e39730d4900e P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.796573 2467 raft_consensus.cc:493] T 4f0c4f0e8f014e59be9e46d2afe0be9d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:47.796823 2467 raft_consensus.cc:515] T 4f0c4f0e8f014e59be9e46d2afe0be9d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:47.797230 2888 raft_consensus.cc:493] T e429b94326cc4827a03968fd92de38e0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:47.797400 1166 ts_tablet_manager.cc:1434] T 6a33cfb2f2de48d4a5e9e39730d4900e P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:47.797356 2888 raft_consensus.cc:515] T e429b94326cc4827a03968fd92de38e0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:47.797654 1166 tablet_bootstrap.cc:492] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:47.797901 2888 leader_election.cc:290] T e429b94326cc4827a03968fd92de38e0 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:47.798935 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e429b94326cc4827a03968fd92de38e0" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:47.799516 712 leader_election.cc:343] T e429b94326cc4827a03968fd92de38e0 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.800469 765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f0c4f0e8f014e59be9e46d2afe0be9d" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:47.800637 2467 leader_election.cc:290] T 4f0c4f0e8f014e59be9e46d2afe0be9d P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:47.800657 765 raft_consensus.cc:2468] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 0.
I20251025 07:52:47.801044 982 leader_election.cc:304] T 4f0c4f0e8f014e59be9e46d2afe0be9d P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:47.801585 2467 raft_consensus.cc:2804] T 4f0c4f0e8f014e59be9e46d2afe0be9d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election won for term 1
W20251025 07:52:47.801615 983 leader_election.cc:343] T 4f0c4f0e8f014e59be9e46d2afe0be9d P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 4f0c4f0e8f014e59be9e46d2afe0be9d
I20251025 07:52:47.801734 2467 raft_consensus.cc:493] T 4f0c4f0e8f014e59be9e46d2afe0be9d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:47.801898 2467 raft_consensus.cc:3060] T 4f0c4f0e8f014e59be9e46d2afe0be9d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.802229 1166 tablet_bootstrap.cc:654] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.805373 2467 raft_consensus.cc:515] T 4f0c4f0e8f014e59be9e46d2afe0be9d P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:47.806583 1169 tablet_bootstrap.cc:492] T 682c1e22d7764dce862d970b33b88cb3 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:47.806514 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f0c4f0e8f014e59be9e46d2afe0be9d" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:47.806702 772 raft_consensus.cc:3060] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.806807 1169 ts_tablet_manager.cc:1403] T 682c1e22d7764dce862d970b33b88cb3 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.048s user 0.001s sys 0.004s
I20251025 07:52:47.807598 1169 raft_consensus.cc:359] T 682c1e22d7764dce862d970b33b88cb3 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.807819 1169 raft_consensus.cc:385] T 682c1e22d7764dce862d970b33b88cb3 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.807922 1169 raft_consensus.cc:740] T 682c1e22d7764dce862d970b33b88cb3 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.807943 2888 raft_consensus.cc:493] T 288e543bb279410c91c01dd1623e110f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:47.808121 2166 ts_tablet_manager.cc:1929] T 80fe3255ed614270bd8352a2853fbc43 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.808064 2888 raft_consensus.cc:515] T 288e543bb279410c91c01dd1623e110f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.808213 2166 log.cc:1199] T 80fe3255ed614270bd8352a2853fbc43 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/80fe3255ed614270bd8352a2853fbc43
I20251025 07:52:47.808344 1169 consensus_queue.cc:260] T 682c1e22d7764dce862d970b33b88cb3 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.808531 2166 ts_tablet_manager.cc:1950] T 80fe3255ed614270bd8352a2853fbc43 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:47.808669 2888 leader_election.cc:290] T 288e543bb279410c91c01dd1623e110f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:47.809204 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 164) took 6328 ms. Trace:
I20251025 07:52:47.809283 2166 rpcz_store.cc:276] 1025 07:52:41.480894 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.480938 (+ 44us) service_pool.cc:225] Handling call
1025 07:52:47.783200 (+6302262us) ts_tablet_manager.cc:1151] Deleting tablet 80fe3255ed614270bd8352a2853fbc43
1025 07:52:47.783212 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.793690 (+ 10478us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.808012 (+ 14322us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.809186 (+ 1174us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6302084}
I20251025 07:52:47.809423 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "288e543bb279410c91c01dd1623e110f" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:47.809734 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 80fe3255ed614270bd8352a2853fbc43 (table default.Table_d2a7e16c6f0d4b2886c16bfe6b685c00 [id=64c18a10455c4694babcc5ec0f463b85]) successfully deleted
I20251025 07:52:47.810086 2166 tablet_replica.cc:333] T ebc824ec4c60411dab1a2ea4a385148e P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:47.810314 2166 raft_consensus.cc:2243] T ebc824ec4c60411dab1a2ea4a385148e P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
W20251025 07:52:47.810452 712 leader_election.cc:343] T 288e543bb279410c91c01dd1623e110f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.810528 2166 raft_consensus.cc:2272] T ebc824ec4c60411dab1a2ea4a385148e P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.811133 2166 ts_tablet_manager.cc:1916] T ebc824ec4c60411dab1a2ea4a385148e P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.812678 772 raft_consensus.cc:2468] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db42be7ad3c741a0aa427f27e245993b in term 1.
I20251025 07:52:47.812682 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet b56209e4934e4407a8b4518ca4af0d46 (table default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de]). Expected version 1 got 0
I20251025 07:52:47.813009 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 830fdbc6812e416690d0db42a73af503 (table default.Table_92824b3f2a774564b3aea04aaa25d204 [id=fb21a45b86ab446e8c3dae4512a9598f]). Expected version 1 got 0
I20251025 07:52:47.813145 982 leader_election.cc:304] T 4f0c4f0e8f014e59be9e46d2afe0be9d P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:47.813400 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c]). Expected version 1 got 0
I20251025 07:52:47.813913 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84]). Expected version 1 got 0
I20251025 07:52:47.815898 1166 tablet_bootstrap.cc:492] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:47.816042 1166 ts_tablet_manager.cc:1403] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20251025 07:52:47.816068 1167 tablet_bootstrap.cc:492] T d0240d20f9904b4bbc8bf7f0f05e8850 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:47.816277 1167 ts_tablet_manager.cc:1403] T d0240d20f9904b4bbc8bf7f0f05e8850 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.081s user 0.004s sys 0.002s
W20251025 07:52:47.816650 847 leader_election.cc:343] T 2b594d4718344aafbdaec34040695589 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: 2b594d4718344aafbdaec34040695589
I20251025 07:52:47.816815 1166 raft_consensus.cc:359] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:47.816897 847 leader_election.cc:304] T 2b594d4718344aafbdaec34040695589 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.816982 1166 raft_consensus.cc:385] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.817303 847 leader_election.cc:304] T 1068f566477e4c6db7dde566fff4c2fb P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.817404 2826 raft_consensus.cc:2749] T 2b594d4718344aafbdaec34040695589 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:47.817539 2886 raft_consensus.cc:2804] T 1068f566477e4c6db7dde566fff4c2fb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:47.817032 1166 raft_consensus.cc:740] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.817618 2886 raft_consensus.cc:493] T 1068f566477e4c6db7dde566fff4c2fb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:47.817699 2886 raft_consensus.cc:3060] T 1068f566477e4c6db7dde566fff4c2fb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.817595 1167 raft_consensus.cc:359] T d0240d20f9904b4bbc8bf7f0f05e8850 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.817862 1167 raft_consensus.cc:385] T d0240d20f9904b4bbc8bf7f0f05e8850 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.817804 1166 consensus_queue.cc:260] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:47.817924 1167 raft_consensus.cc:740] T d0240d20f9904b4bbc8bf7f0f05e8850 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.818100 1167 consensus_queue.cc:260] T d0240d20f9904b4bbc8bf7f0f05e8850 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.818329 1166 ts_tablet_manager.cc:1434] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:47.818536 1167 ts_tablet_manager.cc:1434] T d0240d20f9904b4bbc8bf7f0f05e8850 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:47.818590 1166 tablet_bootstrap.cc:492] T fce535a0c0bd45c6bf17f81add297f40 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:47.818807 1167 tablet_bootstrap.cc:492] T ed243540f2d746b980c25687fbca7414 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:47.820202 1166 tablet_bootstrap.cc:654] T fce535a0c0bd45c6bf17f81add297f40 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.820251 1167 tablet_bootstrap.cc:654] T ed243540f2d746b980c25687fbca7414 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.820544 2886 raft_consensus.cc:515] T 1068f566477e4c6db7dde566fff4c2fb P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.821926 2671 raft_consensus.cc:2804] T 4f0c4f0e8f014e59be9e46d2afe0be9d P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:47.822059 2671 raft_consensus.cc:697] T 4f0c4f0e8f014e59be9e46d2afe0be9d P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Becoming Leader. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Running, Role: LEADER
I20251025 07:52:47.822024 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1068f566477e4c6db7dde566fff4c2fb" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:47.822304 2671 consensus_queue.cc:237] T 4f0c4f0e8f014e59be9e46d2afe0be9d P db42be7ad3c741a0aa427f27e245993b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
W20251025 07:52:47.822546 850 leader_election.cc:343] T 1068f566477e4c6db7dde566fff4c2fb P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.822788 2886 leader_election.cc:290] T 1068f566477e4c6db7dde566fff4c2fb P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:47.826820 2166 ts_tablet_manager.cc:1929] T ebc824ec4c60411dab1a2ea4a385148e P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.826934 2166 log.cc:1199] T ebc824ec4c60411dab1a2ea4a385148e P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/ebc824ec4c60411dab1a2ea4a385148e
I20251025 07:52:47.827245 2166 ts_tablet_manager.cc:1950] T ebc824ec4c60411dab1a2ea4a385148e P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:47.827970 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 165) took 6346 ms. Trace:
I20251025 07:52:47.828063 2166 rpcz_store.cc:276] 1025 07:52:41.481275 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.481316 (+ 41us) service_pool.cc:225] Handling call
1025 07:52:47.810041 (+6328725us) ts_tablet_manager.cc:1151] Deleting tablet ebc824ec4c60411dab1a2ea4a385148e
1025 07:52:47.810050 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.819543 (+ 9493us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.826718 (+ 7175us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.827951 (+ 1233us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6328571}
I20251025 07:52:47.830762 1169 ts_tablet_manager.cc:1434] T 682c1e22d7764dce862d970b33b88cb3 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20251025 07:52:47.831246 2146 ts_tablet_manager.cc:1929] T 80ee75935d7e461d9d4066fac9819bef P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.833881 1169 tablet_bootstrap.cc:492] T 9a8e2658a81e436298fe2190726080d4 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
W20251025 07:52:47.831866 850 consensus_peers.cc:597] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5b6383c99e9e4c90a374da9ccd830a6f. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:47.834228 2146 log.cc:1199] T 80ee75935d7e461d9d4066fac9819bef P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/80ee75935d7e461d9d4066fac9819bef
I20251025 07:52:47.834770 2146 ts_tablet_manager.cc:1950] T 80ee75935d7e461d9d4066fac9819bef P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:47.835248 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_87d38832f85e4965bfab1a077990114f [id=ad812862698d421184ebf501ce79b7be]). Expected version 2 got 0
I20251025 07:52:47.835847 2166 tablet_replica.cc:333] T 91728987303b41fd840eb56f3a311c5f P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:47.836032 2166 raft_consensus.cc:2243] T 91728987303b41fd840eb56f3a311c5f P 23eeffd3de254ce18db5febfde971c6b [term 2 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:47.836165 2166 raft_consensus.cc:2272] T 91728987303b41fd840eb56f3a311c5f P 23eeffd3de254ce18db5febfde971c6b [term 2 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.836666 2166 ts_tablet_manager.cc:1916] T 91728987303b41fd840eb56f3a311c5f P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.844807 2467 leader_election.cc:290] T 4f0c4f0e8f014e59be9e46d2afe0be9d P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:47.845186 983 leader_election.cc:343] T 4f0c4f0e8f014e59be9e46d2afe0be9d P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 4f0c4f0e8f014e59be9e46d2afe0be9d
I20251025 07:52:47.850978 2166 ts_tablet_manager.cc:1929] T 91728987303b41fd840eb56f3a311c5f P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:47.851112 2166 log.cc:1199] T 91728987303b41fd840eb56f3a311c5f P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/91728987303b41fd840eb56f3a311c5f
I20251025 07:52:47.851389 2166 ts_tablet_manager.cc:1950] T 91728987303b41fd840eb56f3a311c5f P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:47.852007 2888 raft_consensus.cc:493] T 243b8828fc344bacab46d7bab6bd2dec P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:47.852151 2888 raft_consensus.cc:515] T 243b8828fc344bacab46d7bab6bd2dec P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.852793 2888 leader_election.cc:290] T 243b8828fc344bacab46d7bab6bd2dec P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:47.852157 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 166) took 6370 ms. Trace:
I20251025 07:52:47.854218 2166 rpcz_store.cc:276] 1025 07:52:41.481620 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.481796 (+ 176us) service_pool.cc:225] Handling call
1025 07:52:47.835798 (+6354002us) ts_tablet_manager.cc:1151] Deleting tablet 91728987303b41fd840eb56f3a311c5f
1025 07:52:47.835808 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.839802 (+ 3994us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.850863 (+ 11061us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.852138 (+ 1275us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6353867}
I20251025 07:52:47.854569 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "243b8828fc344bacab46d7bab6bd2dec" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:47.854799 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 91728987303b41fd840eb56f3a311c5f (table default.Table_b53575428f5b4c25a5c912428e5d4f2d [id=88b6e7cfe0c74d169585a1b064d58e4a]) successfully deleted
W20251025 07:52:47.855096 712 leader_election.cc:343] T 243b8828fc344bacab46d7bab6bd2dec P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.855232 2166 tablet_replica.cc:333] T 0f5f6ca6c3cd4d369690e425387a4344 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:47.855425 2166 raft_consensus.cc:2243] T 0f5f6ca6c3cd4d369690e425387a4344 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:47.855584 2166 raft_consensus.cc:2272] T 0f5f6ca6c3cd4d369690e425387a4344 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.835640 1166 tablet_bootstrap.cc:492] T fce535a0c0bd45c6bf17f81add297f40 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:47.856088 1166 ts_tablet_manager.cc:1403] T fce535a0c0bd45c6bf17f81add297f40 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.000s
I20251025 07:52:47.856181 2166 ts_tablet_manager.cc:1916] T 0f5f6ca6c3cd4d369690e425387a4344 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.856810 1166 raft_consensus.cc:359] T fce535a0c0bd45c6bf17f81add297f40 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.856838 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 163) took 6441 ms. Trace:
I20251025 07:52:47.856973 1166 raft_consensus.cc:385] T fce535a0c0bd45c6bf17f81add297f40 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251025 07:52:47.836513 847 consensus_peers.cc:597] T e54fe7cbcffa40c4b04144b47bee2b48 P 3aea2869ff814cc7b142012faa274382 -> Peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Couldn't send request to peer 23eeffd3de254ce18db5febfde971c6b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e54fe7cbcffa40c4b04144b47bee2b48. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:47.857065 2146 rpcz_store.cc:276] 1025 07:52:41.415695 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.416922 (+ 1227us) service_pool.cc:225] Handling call
1025 07:52:47.712118 (+6295196us) ts_tablet_manager.cc:1151] Deleting tablet 80ee75935d7e461d9d4066fac9819bef
1025 07:52:47.712128 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.766257 (+ 54129us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.831127 (+ 64870us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.856818 (+ 25691us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6295037}
I20251025 07:52:47.857787 905 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "288e543bb279410c91c01dd1623e110f" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:47.858315 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 80ee75935d7e461d9d4066fac9819bef (table default.Table_bb258691f8844d85ab37eb73e9a4c314 [id=c80bab0ed06a459eb78b1d94df13f337]) successfully deleted
I20251025 07:52:47.858620 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "243b8828fc344bacab46d7bab6bd2dec" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:47.858631 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1068f566477e4c6db7dde566fff4c2fb" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:47.858776 770 raft_consensus.cc:3060] T 1068f566477e4c6db7dde566fff4c2fb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.857090 1166 raft_consensus.cc:740] T fce535a0c0bd45c6bf17f81add297f40 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.858956 1166 consensus_queue.cc:260] T fce535a0c0bd45c6bf17f81add297f40 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.859419 1166 ts_tablet_manager.cc:1434] T fce535a0c0bd45c6bf17f81add297f40 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251025 07:52:47.859719 1166 tablet_bootstrap.cc:492] T e8402fd33ba842e981c6c68787c7c548 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:47.861016 1166 tablet_bootstrap.cc:654] T e8402fd33ba842e981c6c68787c7c548 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.861415 770 raft_consensus.cc:2468] T 1068f566477e4c6db7dde566fff4c2fb P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:47.858083 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e429b94326cc4827a03968fd92de38e0" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:47.861996 2888 raft_consensus.cc:493] T 1c40b2e06d894a06af9e58285892ea8d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:47.862110 2888 raft_consensus.cc:515] T 1c40b2e06d894a06af9e58285892ea8d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.862684 2888 leader_election.cc:290] T 1c40b2e06d894a06af9e58285892ea8d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:47.862833 2166 ts_tablet_manager.cc:1929] T 0f5f6ca6c3cd4d369690e425387a4344 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.862928 2166 log.cc:1199] T 0f5f6ca6c3cd4d369690e425387a4344 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/0f5f6ca6c3cd4d369690e425387a4344
I20251025 07:52:47.863242 2166 ts_tablet_manager.cc:1950] T 0f5f6ca6c3cd4d369690e425387a4344 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:47.863332 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c40b2e06d894a06af9e58285892ea8d" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:47.864034 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 168) took 6381 ms. Trace:
W20251025 07:52:47.864180 712 leader_election.cc:343] T 1c40b2e06d894a06af9e58285892ea8d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.864416 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c40b2e06d894a06af9e58285892ea8d" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:47.864553 710 leader_election.cc:343] T 288e543bb279410c91c01dd1623e110f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.864646 710 leader_election.cc:304] T 288e543bb279410c91c01dd1623e110f P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:47.864964 710 leader_election.cc:343] T 243b8828fc344bacab46d7bab6bd2dec P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.865039 710 leader_election.cc:304] T 243b8828fc344bacab46d7bab6bd2dec P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.865211 2663 raft_consensus.cc:2749] T 288e543bb279410c91c01dd1623e110f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:47.865257 2169 ts_tablet_manager.cc:1929] T d9bfd7c0c3ad4e5b98b6708793d61d41 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.865347 2169 log.cc:1199] T d9bfd7c0c3ad4e5b98b6708793d61d41 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/d9bfd7c0c3ad4e5b98b6708793d61d41
I20251025 07:52:47.865340 2663 raft_consensus.cc:2749] T 243b8828fc344bacab46d7bab6bd2dec P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:47.865370 710 leader_election.cc:343] T e429b94326cc4827a03968fd92de38e0 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.865458 710 leader_election.cc:304] T e429b94326cc4827a03968fd92de38e0 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.865628 2663 raft_consensus.cc:2749] T e429b94326cc4827a03968fd92de38e0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:47.865694 2169 ts_tablet_manager.cc:1950] T d9bfd7c0c3ad4e5b98b6708793d61d41 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:47.866459 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 36262ab8c6964dd684c01b4db6219a4f (table default.Table_76158f8cf82e4cdb9a90580e46aaea48 [id=f249f30497bc450c887d659721469de8]). Expected version 1 got 0
I20251025 07:52:47.866462 847 leader_election.cc:304] T 1068f566477e4c6db7dde566fff4c2fb P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.866824 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 288e543bb279410c91c01dd1623e110f (table default.Table_6e242ba55b194c23a6701f4b064fdbe5 [id=4c34b64b299f4162a53866cf01c5298f]). Expected version 1 got 0
I20251025 07:52:47.867062 2886 raft_consensus.cc:2804] T 1068f566477e4c6db7dde566fff4c2fb P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:47.867170 2886 raft_consensus.cc:697] T 1068f566477e4c6db7dde566fff4c2fb P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:47.867390 2886 consensus_queue.cc:237] T 1068f566477e4c6db7dde566fff4c2fb P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.867916 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3]). Expected version 2 got 0
I20251025 07:52:47.868638 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet bc2d16b33ca8426abbd7bd38339c9f1d (table default.Table_4ea985df4c5b415c9570c64506a98cc4 [id=98d2b7797e33485e8872054aa2eb8ec9]). Expected version 1 got 0
I20251025 07:52:47.868805 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e]). Expected version 1 got 0
I20251025 07:52:47.869063 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]). Expected version 1 got 0
I20251025 07:52:47.869273 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]). Expected version 1 got 0
I20251025 07:52:47.869554 451 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 292fe04055954afd8c08ac2636b68191 (table default.Table_9fa69b10b5e842639fc0b0a58cd9b6a5 [id=a253ce0253bd4da4814741c482193d8f]). Expected version 1 got 0
W20251025 07:52:47.869756 710 leader_election.cc:343] T 1c40b2e06d894a06af9e58285892ea8d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.869858 710 leader_election.cc:304] T 1c40b2e06d894a06af9e58285892ea8d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:47.870245 2888 raft_consensus.cc:2749] T 1c40b2e06d894a06af9e58285892ea8d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:47.871059 450 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet ab42909afc524f2fa3c1041495d13a59
I20251025 07:52:47.871649 450 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:47610 (request call id 101) took 3157 ms. Trace:
I20251025 07:52:47.871973 894 tablet_service.cc:1552] Processing DeleteTablet for tablet ab42909afc524f2fa3c1041495d13a59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:45 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:47.871762 450 rpcz_store.cc:276] 1025 07:52:44.713975 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.598233 (+884258us) service_pool.cc:225] Handling call
1025 07:52:45.598675 (+ 442us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:45.598858 (+ 183us) catalog_manager.cc:2815] Modifying in-memory table state
1025 07:52:45.598949 (+ 91us) catalog_manager.cc:2824] Locking tablets
1025 07:52:45.963526 (+364577us) catalog_manager.cc:2838] Removing table and tablets from system table
1025 07:52:46.125671 (+162145us) catalog_manager.cc:2856] Removing table from by-name map
1025 07:52:46.334880 (+209209us) catalog_manager.cc:2872] Committing in-memory state
1025 07:52:47.871005 (+1536125us) catalog_manager.cc:2876] Aborting table tasks
1025 07:52:47.871585 (+ 580us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:45.965938 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:45.966168 (+ 230us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:45.966174 (+ 6us) write_op.cc:435] Acquired schema lock
1025 07:52:45.966177 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:45.966333 (+ 156us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:45.966369 (+ 36us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:45.966374 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:45.966518 (+ 144us) tablet.cc:699] Row locks acquired
1025 07:52:45.966521 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:45.966590 (+ 69us) write_op.cc:270] Start()
1025 07:52:45.966643 (+ 53us) write_op.cc:276] Timestamp: P: 1761378765966575 usec, L: 0
1025 07:52:45.966646 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:45.966888 (+ 242us) log.cc:844] Serialized 756 byte log entry
1025 07:52:46.124240 (+157352us) op_driver.cc:464] REPLICATION: finished
1025 07:52:46.125207 (+ 967us) write_op.cc:301] APPLY: starting
1025 07:52:46.125217 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:46.125251 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:46.125274 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:46.125332 (+ 58us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:46.125360 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1025 07:52:46.125367 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:46.125420 (+ 53us) log.cc:844] Serialized 29 byte log entry
1025 07:52:46.125455 (+ 35us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:46.125469 (+ 14us) write_op.cc:454] Released schema lock
1025 07:52:46.125473 (+ 4us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:46.125531 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:46.125534 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":933,"apply.run_cpu_time_us":318,"apply.run_wall_time_us":317,"num_ops":2,"prepare.queue_time_us":1718,"prepare.run_cpu_time_us":761,"prepare.run_wall_time_us":1027,"replication_time_us":157573}]]}
I20251025 07:52:47.872269 754 tablet_service.cc:1552] Processing DeleteTablet for tablet ab42909afc524f2fa3c1041495d13a59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:45 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:47.874177 1166 tablet_bootstrap.cc:492] T e8402fd33ba842e981c6c68787c7c548 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:47.874398 1166 ts_tablet_manager.cc:1403] T e8402fd33ba842e981c6c68787c7c548 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.001s
I20251025 07:52:47.875299 1166 raft_consensus.cc:359] T e8402fd33ba842e981c6c68787c7c548 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:47.875511 1166 raft_consensus.cc:385] T e8402fd33ba842e981c6c68787c7c548 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:47.875622 1166 raft_consensus.cc:740] T e8402fd33ba842e981c6c68787c7c548 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:47.875895 1166 consensus_queue.cc:260] T e8402fd33ba842e981c6c68787c7c548 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:47.876647 1166 ts_tablet_manager.cc:1434] T e8402fd33ba842e981c6c68787c7c548 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:47.876968 1166 tablet_bootstrap.cc:492] T 645263a61b654f04985c40d04506c087 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:47.878400 1166 tablet_bootstrap.cc:654] T 645263a61b654f04985c40d04506c087 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.881966 454 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet 4a1ec8c2ed7f41abb588f592f62097d1
I20251025 07:52:47.870874 1169 tablet_bootstrap.cc:654] T 9a8e2658a81e436298fe2190726080d4 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.882668 454 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:47610 (request call id 105) took 2880 ms. Trace:
I20251025 07:52:47.882812 454 rpcz_store.cc:276] 1025 07:52:45.002650 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.705868 (+703218us) service_pool.cc:225] Handling call
1025 07:52:45.712894 (+ 7026us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:45.713041 (+ 147us) catalog_manager.cc:2815] Modifying in-memory table state
1025 07:52:45.713123 (+ 82us) catalog_manager.cc:2824] Locking tablets
1025 07:52:45.961859 (+248736us) catalog_manager.cc:2838] Removing table and tablets from system table
1025 07:52:46.125627 (+163768us) catalog_manager.cc:2856] Removing table from by-name map
1025 07:52:46.334946 (+209319us) catalog_manager.cc:2872] Committing in-memory state
1025 07:52:47.881877 (+1546931us) catalog_manager.cc:2876] Aborting table tasks
1025 07:52:47.882628 (+ 751us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:45.964576 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:45.964866 (+ 290us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:45.964872 (+ 6us) write_op.cc:435] Acquired schema lock
1025 07:52:45.964875 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:45.965134 (+ 259us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:45.965174 (+ 40us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:45.965179 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:45.965373 (+ 194us) tablet.cc:699] Row locks acquired
1025 07:52:45.965376 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:45.965436 (+ 60us) write_op.cc:270] Start()
1025 07:52:45.965467 (+ 31us) write_op.cc:276] Timestamp: P: 1761378765965395 usec, L: 0
1025 07:52:45.965470 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:45.965768 (+ 298us) log.cc:844] Serialized 725 byte log entry
1025 07:52:46.124166 (+158398us) op_driver.cc:464] REPLICATION: finished
1025 07:52:46.124628 (+ 462us) write_op.cc:301] APPLY: starting
1025 07:52:46.124648 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:46.124696 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:46.124734 (+ 38us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:46.124810 (+ 76us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:46.124842 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1025 07:52:46.124849 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:46.124939 (+ 90us) log.cc:844] Serialized 29 byte log entry
1025 07:52:46.125010 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:46.125035 (+ 25us) write_op.cc:454] Released schema lock
1025 07:52:46.125044 (+ 9us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:46.125126 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:46.125132 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":400,"apply.run_cpu_time_us":494,"apply.run_wall_time_us":493,"num_ops":2,"prepare.queue_time_us":1972,"prepare.run_cpu_time_us":964,"prepare.run_wall_time_us":1320,"replication_time_us":158663}]]}
W20251025 07:52:47.885890 453 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c501ef5d70e742598c1a902efe47feb1 consensus state has no leader
I20251025 07:52:47.886063 453 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:47610 (request call id 115) took 2147 ms. Trace:
I20251025 07:52:47.886197 453 rpcz_store.cc:276] 1025 07:52:45.738359 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:46.626524 (+888165us) service_pool.cc:225] Handling call
1025 07:52:46.627013 (+ 489us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:46.627232 (+ 219us) catalog_manager.cc:3708] Apply alter schema
1025 07:52:46.627387 (+ 155us) catalog_manager.cc:3740] Acquired catalog manager lock
1025 07:52:46.627442 (+ 55us) catalog_manager.cc:3898] Updating metadata on disk
1025 07:52:46.639004 (+ 11562us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:46.639030 (+ 26us) write_op.cc:454] Released schema lock
1025 07:52:46.639119 (+ 89us) catalog_manager.cc:3929] Committing alterations to in-memory state
1025 07:52:47.886028 (+1246909us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:46.628315 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:46.628410 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:46.628418 (+ 8us) write_op.cc:435] Acquired schema lock
1025 07:52:46.628422 (+ 4us) tablet.cc:660] Decoding operations
1025 07:52:46.628511 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:46.628522 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:46.628525 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:46.628584 (+ 59us) tablet.cc:699] Row locks acquired
1025 07:52:46.628587 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:46.628631 (+ 44us) write_op.cc:270] Start()
1025 07:52:46.628659 (+ 28us) write_op.cc:276] Timestamp: P: 1761378766628617 usec, L: 0
1025 07:52:46.628662 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:46.628805 (+ 143us) log.cc:844] Serialized 405 byte log entry
1025 07:52:46.631298 (+ 2493us) op_driver.cc:464] REPLICATION: finished
1025 07:52:46.631418 (+ 120us) write_op.cc:301] APPLY: starting
1025 07:52:46.631431 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:46.631464 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:46.631496 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:46.631543 (+ 47us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:46.631571 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:46.631593 (+ 22us) write_op.cc:312] APPLY: finished
1025 07:52:46.631688 (+ 95us) log.cc:844] Serialized 23 byte log entry
1025 07:52:46.631741 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:46.631773 (+ 32us) write_op.cc:454] Released schema lock
1025 07:52:46.631778 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":70,"apply.run_cpu_time_us":411,"apply.run_wall_time_us":411,"num_ops":1,"prepare.queue_time_us":414,"prepare.run_cpu_time_us":582,"prepare.run_wall_time_us":583,"raft.queue_time_us":54,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":175,"replication_time_us":2609}]]}
I20251025 07:52:47.871932 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet ab42909afc524f2fa3c1041495d13a59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:45 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:47.895320 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a1ec8c2ed7f41abb588f592f62097d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:45 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:47.898275 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a1ec8c2ed7f41abb588f592f62097d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:45 UTC) from {username='slave'} at 127.0.0.1:44572
W20251025 07:52:47.898746 448 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 783487b60c144f719ed2eb3c67b5ccab consensus state has no leader
I20251025 07:52:47.900167 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 182) took 6447 ms. Trace:
I20251025 07:52:47.900272 2169 rpcz_store.cc:276] 1025 07:52:41.452472 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.452864 (+ 392us) service_pool.cc:225] Handling call
1025 07:52:47.751564 (+6298700us) ts_tablet_manager.cc:1151] Deleting tablet d9bfd7c0c3ad4e5b98b6708793d61d41
1025 07:52:47.751574 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.809667 (+ 58093us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.865161 (+ 55494us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.900140 (+ 34979us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6298569}
W20251025 07:52:47.900631 983 consensus_peers.cc:597] T f43ef51ee45d47a99ec732b49d50d3b4 P db42be7ad3c741a0aa427f27e245993b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f43ef51ee45d47a99ec732b49d50d3b4. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:47.900743 2169 tablet_replica.cc:333] T 04449cc97f404a19b6a1b20dea074f95 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:47.901376 2169 raft_consensus.cc:2243] T 04449cc97f404a19b6a1b20dea074f95 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:47.902102 2169 raft_consensus.cc:2272] T 04449cc97f404a19b6a1b20dea074f95 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.903007 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet d9bfd7c0c3ad4e5b98b6708793d61d41 (table default.Table_0c74279815ab4a5186497b68bb932523 [id=5202bb531f384527a167bf9efe00d9c6]) successfully deleted
I20251025 07:52:47.904304 2169 ts_tablet_manager.cc:1916] T 04449cc97f404a19b6a1b20dea074f95 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.904453 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a1ec8c2ed7f41abb588f592f62097d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:45 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:47.864130 2166 rpcz_store.cc:276] 1025 07:52:41.482069 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.482108 (+ 39us) service_pool.cc:225] Handling call
1025 07:52:47.855183 (+6373075us) ts_tablet_manager.cc:1151] Deleting tablet 0f5f6ca6c3cd4d369690e425387a4344
1025 07:52:47.855193 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.859852 (+ 4659us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.862744 (+ 2892us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.864010 (+ 1266us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6372903}
I20251025 07:52:47.906675 2166 tablet_replica.cc:333] T aaef24ccdfba45fe8204327723df834e P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:47.906800 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 0f5f6ca6c3cd4d369690e425387a4344 (table default.Table_e317f8d728da4c76886f67d0995ab7f4 [id=4d2121dea4904850aecb65da5dd2d8f9]) successfully deleted
I20251025 07:52:47.906877 2166 raft_consensus.cc:2243] T aaef24ccdfba45fe8204327723df834e P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:47.907032 2166 raft_consensus.cc:2272] T aaef24ccdfba45fe8204327723df834e P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:47.907653 2166 ts_tablet_manager.cc:1916] T aaef24ccdfba45fe8204327723df834e P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:47.915634 2686 raft_consensus.cc:493] T c187a40f8ff14c6a8c184fbeb030ad13 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:47.915786 2686 raft_consensus.cc:515] T c187a40f8ff14c6a8c184fbeb030ad13 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.916405 2686 leader_election.cc:290] T c187a40f8ff14c6a8c184fbeb030ad13 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:47.917712 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c187a40f8ff14c6a8c184fbeb030ad13" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:47.917933 770 raft_consensus.cc:2468] T c187a40f8ff14c6a8c184fbeb030ad13 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:47.918396 847 leader_election.cc:304] T c187a40f8ff14c6a8c184fbeb030ad13 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:47.918692 2886 raft_consensus.cc:2804] T c187a40f8ff14c6a8c184fbeb030ad13 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:47.918785 2886 raft_consensus.cc:493] T c187a40f8ff14c6a8c184fbeb030ad13 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:47.918869 2886 raft_consensus.cc:3060] T c187a40f8ff14c6a8c184fbeb030ad13 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.922042 455 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:47610 (request call id 117) took 2176 ms. Trace:
I20251025 07:52:47.922106 448 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:47610 (request call id 114) took 2203 ms. Trace:
I20251025 07:52:47.922319 2886 raft_consensus.cc:515] T c187a40f8ff14c6a8c184fbeb030ad13 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.922219 455 rpcz_store.cc:276] 1025 07:52:45.745899 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:46.868624 (+1122725us) service_pool.cc:225] Handling call
1025 07:52:46.869184 (+ 560us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:46.869415 (+ 231us) catalog_manager.cc:3708] Apply alter schema
1025 07:52:46.869664 (+ 249us) catalog_manager.cc:3740] Acquired catalog manager lock
1025 07:52:46.869738 (+ 74us) catalog_manager.cc:3898] Updating metadata on disk
1025 07:52:46.879712 (+ 9974us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:46.879720 (+ 8us) write_op.cc:454] Released schema lock
1025 07:52:46.879805 (+ 85us) catalog_manager.cc:3929] Committing alterations to in-memory state
1025 07:52:47.921981 (+1042176us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:46.870243 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:46.870329 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:46.870333 (+ 4us) write_op.cc:435] Acquired schema lock
1025 07:52:46.870335 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:46.870423 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:46.870434 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:46.870438 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:46.870493 (+ 55us) tablet.cc:699] Row locks acquired
1025 07:52:46.870496 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:46.870535 (+ 39us) write_op.cc:270] Start()
1025 07:52:46.870562 (+ 27us) write_op.cc:276] Timestamp: P: 1761378766870522 usec, L: 0
1025 07:52:46.870565 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:46.870741 (+ 176us) log.cc:844] Serialized 405 byte log entry
1025 07:52:46.874802 (+ 4061us) op_driver.cc:464] REPLICATION: finished
1025 07:52:46.874896 (+ 94us) write_op.cc:301] APPLY: starting
1025 07:52:46.874910 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:46.874946 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:46.874974 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:46.875019 (+ 45us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:46.875049 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:46.875056 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:46.875123 (+ 67us) log.cc:844] Serialized 23 byte log entry
1025 07:52:46.875179 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:46.875196 (+ 17us) write_op.cc:454] Released schema lock
1025 07:52:46.875203 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":51,"apply.run_cpu_time_us":354,"apply.run_wall_time_us":353,"num_ops":1,"prepare.queue_time_us":44,"prepare.run_cpu_time_us":617,"prepare.run_wall_time_us":615,"raft.queue_time_us":430,"raft.run_cpu_time_us":416,"raft.run_wall_time_us":416,"replication_time_us":4210}]]}
I20251025 07:52:47.922286 448 rpcz_store.cc:276] 1025 07:52:45.719085 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:46.459649 (+740564us) service_pool.cc:225] Handling call
1025 07:52:46.614519 (+154870us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:46.614758 (+ 239us) catalog_manager.cc:3708] Apply alter schema
1025 07:52:46.614962 (+ 204us) catalog_manager.cc:3740] Acquired catalog manager lock
1025 07:52:46.615037 (+ 75us) catalog_manager.cc:3898] Updating metadata on disk
1025 07:52:46.627641 (+ 12604us) catalog_manager.cc:3929] Committing alterations to in-memory state
1025 07:52:47.922052 (+1294411us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:46.616537 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:46.616623 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:46.616628 (+ 5us) write_op.cc:435] Acquired schema lock
1025 07:52:46.616630 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:46.616704 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:46.616713 (+ 9us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:46.616717 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:46.616764 (+ 47us) tablet.cc:699] Row locks acquired
1025 07:52:46.616766 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:46.616792 (+ 26us) write_op.cc:270] Start()
1025 07:52:46.616816 (+ 24us) write_op.cc:276] Timestamp: P: 1761378766616780 usec, L: 0
1025 07:52:46.616819 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:46.616933 (+ 114us) log.cc:844] Serialized 405 byte log entry
1025 07:52:46.623346 (+ 6413us) op_driver.cc:464] REPLICATION: finished
1025 07:52:46.625481 (+ 2135us) write_op.cc:301] APPLY: starting
1025 07:52:46.625499 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:46.625537 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:46.625581 (+ 44us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:46.625630 (+ 49us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:46.625688 (+ 58us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:46.625699 (+ 11us) write_op.cc:312] APPLY: finished
1025 07:52:46.625772 (+ 73us) log.cc:844] Serialized 23 byte log entry
1025 07:52:46.625832 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:46.625862 (+ 30us) write_op.cc:454] Released schema lock
1025 07:52:46.625870 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:46.625955 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:46.625962 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":896,"child_traces":[["op",{"apply.queue_time_us":2058,"apply.run_cpu_time_us":464,"apply.run_wall_time_us":464,"mutex_wait_us":59,"num_ops":1,"prepare.queue_time_us":1096,"prepare.run_cpu_time_us":445,"prepare.run_wall_time_us":443,"replication_time_us":6499}]]}
I20251025 07:52:47.923184 455 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_617594400ddd4cf4a8743365e27af63f" } new_table_name: "default.Table_68b39cfaff194c3d8dbe9eaee7d8b44c" modify_external_catalogs: true
I20251025 07:52:47.923301 448 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_10e57b42f97544208675708b1e028846" } new_table_name: "default.Table_a692a40ea2ef44ffa5ab6ca9e29ef70a" modify_external_catalogs: true
I20251025 07:52:47.924319 2886 leader_election.cc:290] T c187a40f8ff14c6a8c184fbeb030ad13 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:47.924405 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c187a40f8ff14c6a8c184fbeb030ad13" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:47.924607 770 raft_consensus.cc:3060] T c187a40f8ff14c6a8c184fbeb030ad13 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:47.926885 2151 rpcz_store.cc:267] Call kudu.server.GenericService.CheckLeaks from 127.0.0.1:51274 (request call id 0) took 956 ms (client timeout 1000 ms). Trace:
W20251025 07:52:47.927016 2151 rpcz_store.cc:269] 1025 07:52:46.970369 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:46.970436 (+ 67us) service_pool.cc:225] Handling call
1025 07:52:47.926821 (+956385us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 07:52:47.927698 770 raft_consensus.cc:2468] T c187a40f8ff14c6a8c184fbeb030ad13 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:47.929597 847 leader_election.cc:304] T c187a40f8ff14c6a8c184fbeb030ad13 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:47.929852 2686 raft_consensus.cc:493] T c501ef5d70e742598c1a902efe47feb1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:47.929968 2686 raft_consensus.cc:515] T c501ef5d70e742598c1a902efe47feb1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.930590 2686 leader_election.cc:290] T c501ef5d70e742598c1a902efe47feb1 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:47.930744 2686 raft_consensus.cc:2804] T c187a40f8ff14c6a8c184fbeb030ad13 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:47.930850 2686 raft_consensus.cc:697] T c187a40f8ff14c6a8c184fbeb030ad13 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:47.931088 2686 consensus_queue.cc:237] T c187a40f8ff14c6a8c184fbeb030ad13 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.933269 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c501ef5d70e742598c1a902efe47feb1" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:47.933455 770 raft_consensus.cc:2468] T c501ef5d70e742598c1a902efe47feb1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:47.933820 847 leader_election.cc:304] T c501ef5d70e742598c1a902efe47feb1 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:47.934053 2886 raft_consensus.cc:2804] T c501ef5d70e742598c1a902efe47feb1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:47.934116 2886 raft_consensus.cc:493] T c501ef5d70e742598c1a902efe47feb1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:47.934202 2886 raft_consensus.cc:3060] T c501ef5d70e742598c1a902efe47feb1 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.934983 2698 consensus_queue.cc:1048] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251025 07:52:47.936483 847 consensus_peers.cc:597] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382 -> Peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Couldn't send request to peer 23eeffd3de254ce18db5febfde971c6b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f5f6ca6c3cd4d369690e425387a4344. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:47.939168 2886 raft_consensus.cc:515] T c501ef5d70e742598c1a902efe47feb1 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
W20251025 07:52:47.940883 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskRBHlYt/build/asan/bin/kudu pid 1830: Timed out: CheckLeaks RPC to 127.0.104.61:42203 timed out after 1.000s (SENT)
I20251025 07:52:47.941073 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRBHlYt/build/asan/bin/kudu with pid 1830
I20251025 07:52:47.944202 765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c501ef5d70e742598c1a902efe47feb1" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:47.944406 765 raft_consensus.cc:3060] T c501ef5d70e742598c1a902efe47feb1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.944504 2886 leader_election.cc:290] T c501ef5d70e742598c1a902efe47feb1 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:47.947811 765 raft_consensus.cc:2468] T c501ef5d70e742598c1a902efe47feb1 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:47.948426 847 leader_election.cc:304] T c501ef5d70e742598c1a902efe47feb1 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:47.948714 2826 raft_consensus.cc:2804] T c501ef5d70e742598c1a902efe47feb1 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:47.948833 2826 raft_consensus.cc:697] T c501ef5d70e742598c1a902efe47feb1 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:47.949128 2826 consensus_queue.cc:237] T c501ef5d70e742598c1a902efe47feb1 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:47.964309 2146 tablet_replica.cc:333] T dda0b26e4f1a44928bc4b84f333901c0 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:47.965876 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c187a40f8ff14c6a8c184fbeb030ad13" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:47.966164 1043 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c501ef5d70e742598c1a902efe47feb1" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:47.967163 1045 raft_consensus.cc:3060] T 36c414bc700341188b7d85c14173c3e0 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:47.970275 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c187a40f8ff14c6a8c184fbeb030ad13" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:47.970635 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c501ef5d70e742598c1a902efe47feb1" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:47.973917 2166 ts_tablet_manager.cc:1929] T aaef24ccdfba45fe8204327723df834e P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:47.974052 2166 log.cc:1199] T aaef24ccdfba45fe8204327723df834e P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/aaef24ccdfba45fe8204327723df834e
I20251025 07:52:47.974483 2166 ts_tablet_manager.cc:1950] T aaef24ccdfba45fe8204327723df834e P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:47.975355 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 169) took 6492 ms. Trace:
I20251025 07:52:47.975463 2166 rpcz_store.cc:276] 1025 07:52:41.482420 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.482458 (+ 38us) service_pool.cc:225] Handling call
1025 07:52:47.906621 (+6424163us) ts_tablet_manager.cc:1151] Deleting tablet aaef24ccdfba45fe8204327723df834e
1025 07:52:47.906632 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.959812 (+ 53180us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.973789 (+ 13977us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.975338 (+ 1549us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6424029}
I20251025 07:52:47.975975 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet aaef24ccdfba45fe8204327723df834e (table default.Table_bd5089a7892041649dd05b7fb0961255 [id=840b3ffcb44144548843457c48f46380]) successfully deleted
I20251025 07:52:47.978957 2169 ts_tablet_manager.cc:1929] T 04449cc97f404a19b6a1b20dea074f95 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251025 07:52:47.971380 850 consensus_peers.cc:597] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d9bfd7c0c3ad4e5b98b6708793d61d41. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:47.980152 2146 raft_consensus.cc:2243] T dda0b26e4f1a44928bc4b84f333901c0 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:47.980393 2146 pending_rounds.cc:70] T dda0b26e4f1a44928bc4b84f333901c0 P 3aea2869ff814cc7b142012faa274382: Trying to abort 1 pending ops.
I20251025 07:52:47.980455 2146 pending_rounds.cc:77] T dda0b26e4f1a44928bc4b84f333901c0 P 3aea2869ff814cc7b142012faa274382: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214607430492286976 op_type: NO_OP noop_request { }
I20251025 07:52:47.980545 2146 raft_consensus.cc:2889] T dda0b26e4f1a44928bc4b84f333901c0 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
W20251025 07:52:47.980494 850 leader_election.cc:343] T c187a40f8ff14c6a8c184fbeb030ad13 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.980623 2146 raft_consensus.cc:2272] T dda0b26e4f1a44928bc4b84f333901c0 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:47.980896 850 consensus_peers.cc:597] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251025 07:52:47.981072 850 leader_election.cc:343] T c501ef5d70e742598c1a902efe47feb1 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:47.981496 850 leader_election.cc:343] T c187a40f8ff14c6a8c184fbeb030ad13 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:47.981787 850 leader_election.cc:343] T c501ef5d70e742598c1a902efe47feb1 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:47.982247 2698 consensus_queue.cc:1048] T 4a1ec8c2ed7f41abb588f592f62097d1 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:47.982741 2698 consensus_queue.cc:1048] T c7691c7bdd89437c8db060254c0db25d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:47.983569 2146 ts_tablet_manager.cc:1916] T dda0b26e4f1a44928bc4b84f333901c0 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:47.984431 850 consensus_peers.cc:597] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f5f6ca6c3cd4d369690e425387a4344. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:47.985461 1167 tablet_bootstrap.cc:492] T ed243540f2d746b980c25687fbca7414 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:47.988792 2888 raft_consensus.cc:493] T 72618756f756484e895e885ef12545c8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:47.988940 2888 raft_consensus.cc:515] T 72618756f756484e895e885ef12545c8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:47.989579 2888 leader_election.cc:290] T 72618756f756484e895e885ef12545c8 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:47.990196 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72618756f756484e895e885ef12545c8" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:47.993247 1167 ts_tablet_manager.cc:1403] T ed243540f2d746b980c25687fbca7414 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.175s user 0.003s sys 0.000s
W20251025 07:52:47.994164 710 leader_election.cc:343] T 72618756f756484e895e885ef12545c8 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.009683 2698 consensus_queue.cc:1048] T 9a8e2658a81e436298fe2190726080d4 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:48.015339 1167 raft_consensus.cc:359] T ed243540f2d746b980c25687fbca7414 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.015707 1167 raft_consensus.cc:385] T ed243540f2d746b980c25687fbca7414 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.015815 1167 raft_consensus.cc:740] T ed243540f2d746b980c25687fbca7414 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.016125 1167 consensus_queue.cc:260] T ed243540f2d746b980c25687fbca7414 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.016875 1167 ts_tablet_manager.cc:1434] T ed243540f2d746b980c25687fbca7414 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20251025 07:52:48.017211 1167 tablet_bootstrap.cc:492] T df9db9dd31d840e0b1394db04d662a62 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:48.019168 1167 tablet_bootstrap.cc:654] T df9db9dd31d840e0b1394db04d662a62 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:47.978178 2686 raft_consensus.cc:493] T c11f674385184ae6b235f1db8bf1fb91 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:48.025588 2686 raft_consensus.cc:515] T c11f674385184ae6b235f1db8bf1fb91 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.029618 2686 leader_election.cc:290] T c11f674385184ae6b235f1db8bf1fb91 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:47.982583 2671 raft_consensus.cc:493] T 682c1e22d7764dce862d970b33b88cb3 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:48.030949 2671 raft_consensus.cc:515] T 682c1e22d7764dce862d970b33b88cb3 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.031561 2671 leader_election.cc:290] T 682c1e22d7764dce862d970b33b88cb3 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:47.990352 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72618756f756484e895e885ef12545c8" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:48.032717 1048 raft_consensus.cc:3060] T 682c1e22d7764dce862d970b33b88cb3 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.035259 894 tablet_service.cc:1505] Processing CreateTablet for tablet 695b150f4c4742c7a8a567cbda05196d (DEFAULT_TABLE table=default.Table_976b27c0f51e4b4b871ac24a5a320b08 [id=016f352b8b474dff9b3294e53393f7f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:48.037698 1018 tablet_service.cc:1505] Processing CreateTablet for tablet e23152fbd6c24756a66f9cc712bfaff1 (DEFAULT_TABLE table=default.Table_4d4a21cfe7034318b758a28cc3471403 [id=f1e9ac3816f346d5bbf32772b9f8aa5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:48.037756 875 tablet_service.cc:1505] Processing CreateTablet for tablet e23152fbd6c24756a66f9cc712bfaff1 (DEFAULT_TABLE table=default.Table_4d4a21cfe7034318b758a28cc3471403 [id=f1e9ac3816f346d5bbf32772b9f8aa5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:48.037856 1019 tablet_service.cc:1505] Processing CreateTablet for tablet 695b150f4c4742c7a8a567cbda05196d (DEFAULT_TABLE table=default.Table_976b27c0f51e4b4b871ac24a5a320b08 [id=016f352b8b474dff9b3294e53393f7f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:48.038096 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e23152fbd6c24756a66f9cc712bfaff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:48.038368 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 695b150f4c4742c7a8a567cbda05196d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:48.046367 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 695b150f4c4742c7a8a567cbda05196d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:48.046844 2979 tablet.cc:1722] T cd23b8094d7e472eba6bd170248858a9 P 3aea2869ff814cc7b142012faa274382: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251025 07:52:48.046937 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e23152fbd6c24756a66f9cc712bfaff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:48.049450 2698 consensus_queue.cc:1048] T 682c1e22d7764dce862d970b33b88cb3 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:48.050920 2698 consensus_queue.cc:1048] T c7691c7bdd89437c8db060254c0db25d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251025 07:52:48.054037 850 consensus_peers.cc:597] T 9a8e2658a81e436298fe2190726080d4 P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:48.055863 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "682c1e22d7764dce862d970b33b88cb3" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:48.056968 1169 tablet_bootstrap.cc:492] T 9a8e2658a81e436298fe2190726080d4 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:48.057166 1169 ts_tablet_manager.cc:1403] T 9a8e2658a81e436298fe2190726080d4 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.223s user 0.004s sys 0.013s
I20251025 07:52:48.057974 1169 raft_consensus.cc:359] T 9a8e2658a81e436298fe2190726080d4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.058248 1169 raft_consensus.cc:385] T 9a8e2658a81e436298fe2190726080d4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.058351 1169 raft_consensus.cc:740] T 9a8e2658a81e436298fe2190726080d4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.058533 2169 log.cc:1199] T 04449cc97f404a19b6a1b20dea074f95 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/04449cc97f404a19b6a1b20dea074f95
I20251025 07:52:48.058681 1169 consensus_queue.cc:260] T 9a8e2658a81e436298fe2190726080d4 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.059113 2169 ts_tablet_manager.cc:1950] T 04449cc97f404a19b6a1b20dea074f95 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:48.029239 1043 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c11f674385184ae6b235f1db8bf1fb91" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:48.060379 850 leader_election.cc:343] T c11f674385184ae6b235f1db8bf1fb91 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.064925 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 183) took 6611 ms. Trace:
I20251025 07:52:48.065055 2169 rpcz_store.cc:276] 1025 07:52:41.453176 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.453230 (+ 54us) service_pool.cc:225] Handling call
1025 07:52:47.900688 (+6447458us) ts_tablet_manager.cc:1151] Deleting tablet 04449cc97f404a19b6a1b20dea074f95
1025 07:52:47.900700 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:47.963193 (+ 62493us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:47.978853 (+ 15660us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.064908 (+ 86055us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6447325}
I20251025 07:52:48.065491 2169 tablet_replica.cc:333] T f43ef51ee45d47a99ec732b49d50d3b4 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:48.065488 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 04449cc97f404a19b6a1b20dea074f95 (table default.Table_b7da40b96d6c4359bed6f898e8cf6275 [id=e041946c093b4c789ce175c736cc230b]) successfully deleted
I20251025 07:52:48.076598 1166 tablet_bootstrap.cc:492] T 645263a61b654f04985c40d04506c087 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:48.076858 1166 ts_tablet_manager.cc:1403] T 645263a61b654f04985c40d04506c087 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.200s user 0.018s sys 0.009s
I20251025 07:52:48.077908 1166 raft_consensus.cc:359] T 645263a61b654f04985c40d04506c087 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.078233 1166 raft_consensus.cc:385] T 645263a61b654f04985c40d04506c087 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.078343 1166 raft_consensus.cc:740] T 645263a61b654f04985c40d04506c087 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.078732 1166 consensus_queue.cc:260] T 645263a61b654f04985c40d04506c087 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.079572 1166 ts_tablet_manager.cc:1434] T 645263a61b654f04985c40d04506c087 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251025 07:52:48.080341 1166 tablet_bootstrap.cc:492] T a7ab3150a35b4caca9119a1374d3f2f7 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:48.080822 2169 raft_consensus.cc:2243] T f43ef51ee45d47a99ec732b49d50d3b4 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:48.081264 2169 raft_consensus.cc:2272] T f43ef51ee45d47a99ec732b49d50d3b4 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.059763 1169 ts_tablet_manager.cc:1434] T 9a8e2658a81e436298fe2190726080d4 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:48.026538 2978 tablet.cc:1722] T cd23b8094d7e472eba6bd170248858a9 P db42be7ad3c741a0aa427f27e245993b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251025 07:52:48.082716 1169 tablet_bootstrap.cc:492] T 34c63b69904d499bb0d286941a8303f2 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:48.084527 1169 tablet_bootstrap.cc:654] T 34c63b69904d499bb0d286941a8303f2 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.082729 2169 ts_tablet_manager.cc:1916] T f43ef51ee45d47a99ec732b49d50d3b4 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.026942 2166 tablet_replica.cc:333] T ce1ebb1ec14b4801930a72765bca1a7d P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:48.029707 765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c11f674385184ae6b235f1db8bf1fb91" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:48.093468 765 raft_consensus.cc:2468] T c11f674385184ae6b235f1db8bf1fb91 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
W20251025 07:52:48.032371 712 leader_election.cc:343] T 72618756f756484e895e885ef12545c8 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.093699 712 leader_election.cc:304] T 72618756f756484e895e885ef12545c8 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:48.050331 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "682c1e22d7764dce862d970b33b88cb3" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:48.093942 772 raft_consensus.cc:2393] T 682c1e22d7764dce862d970b33b88cb3 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:48.050477 2888 raft_consensus.cc:493] T e47c4f741f354a19b49777bd06e54764 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 07:52:48.094018 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:45417, user_credentials={real_user=slave}} blocked reactor thread for 61665.2us
I20251025 07:52:48.094153 2888 raft_consensus.cc:515] T e47c4f741f354a19b49777bd06e54764 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:48.094754 2888 leader_election.cc:290] T e47c4f741f354a19b49777bd06e54764 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:48.094879 2888 raft_consensus.cc:2749] T 72618756f756484e895e885ef12545c8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:48.095302 982 leader_election.cc:304] T 682c1e22d7764dce862d970b33b88cb3 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:48.095297 754 tablet_service.cc:1505] Processing CreateTablet for tablet 695b150f4c4742c7a8a567cbda05196d (DEFAULT_TABLE table=default.Table_976b27c0f51e4b4b871ac24a5a320b08 [id=016f352b8b474dff9b3294e53393f7f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:48.095703 2671 raft_consensus.cc:2749] T 682c1e22d7764dce862d970b33b88cb3 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:48.095875 748 tablet_service.cc:1505] Processing CreateTablet for tablet e23152fbd6c24756a66f9cc712bfaff1 (DEFAULT_TABLE table=default.Table_4d4a21cfe7034318b758a28cc3471403 [id=f1e9ac3816f346d5bbf32772b9f8aa5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:48.097157 847 leader_election.cc:304] T c11f674385184ae6b235f1db8bf1fb91 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:48.097723 2886 raft_consensus.cc:2804] T c11f674385184ae6b235f1db8bf1fb91 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:48.097831 2886 raft_consensus.cc:493] T c11f674385184ae6b235f1db8bf1fb91 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:48.097891 2886 raft_consensus.cc:3060] T c11f674385184ae6b235f1db8bf1fb91 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.097925 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e47c4f741f354a19b49777bd06e54764" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:48.098335 712 leader_election.cc:343] T e47c4f741f354a19b49777bd06e54764 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.101850 2166 raft_consensus.cc:2243] T ce1ebb1ec14b4801930a72765bca1a7d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:48.102033 2166 raft_consensus.cc:2272] T ce1ebb1ec14b4801930a72765bca1a7d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.102650 2166 ts_tablet_manager.cc:1916] T ce1ebb1ec14b4801930a72765bca1a7d P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.103718 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 695b150f4c4742c7a8a567cbda05196d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:48.105551 2886 raft_consensus.cc:515] T c11f674385184ae6b235f1db8bf1fb91 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.106411 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e23152fbd6c24756a66f9cc712bfaff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:48.113524 2886 leader_election.cc:290] T c11f674385184ae6b235f1db8bf1fb91 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:48.127588 1169 tablet_bootstrap.cc:492] T 34c63b69904d499bb0d286941a8303f2 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:48.127842 1169 ts_tablet_manager.cc:1403] T 34c63b69904d499bb0d286941a8303f2 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.002s
I20251025 07:52:48.128626 1169 raft_consensus.cc:359] T 34c63b69904d499bb0d286941a8303f2 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.131098 765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c11f674385184ae6b235f1db8bf1fb91" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:48.131222 1169 raft_consensus.cc:385] T 34c63b69904d499bb0d286941a8303f2 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.131287 1169 raft_consensus.cc:740] T 34c63b69904d499bb0d286941a8303f2 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.131284 765 raft_consensus.cc:3060] T c11f674385184ae6b235f1db8bf1fb91 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.134331 1169 consensus_queue.cc:260] T 34c63b69904d499bb0d286941a8303f2 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.134845 1169 ts_tablet_manager.cc:1434] T 34c63b69904d499bb0d286941a8303f2 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251025 07:52:48.135111 1169 tablet_bootstrap.cc:492] T dda0b26e4f1a44928bc4b84f333901c0 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:48.135228 765 raft_consensus.cc:2468] T c11f674385184ae6b235f1db8bf1fb91 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:48.141822 2166 ts_tablet_manager.cc:1929] T ce1ebb1ec14b4801930a72765bca1a7d P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.142506 847 leader_election.cc:304] T c11f674385184ae6b235f1db8bf1fb91 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:48.142923 2826 consensus_queue.cc:1048] T 682c1e22d7764dce862d970b33b88cb3 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:48.143306 2826 raft_consensus.cc:2804] T c11f674385184ae6b235f1db8bf1fb91 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:48.143420 2826 raft_consensus.cc:697] T c11f674385184ae6b235f1db8bf1fb91 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:48.143671 2826 consensus_queue.cc:237] T c11f674385184ae6b235f1db8bf1fb91 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.152202 2990 consensus_queue.cc:1048] T 37c7938e8c5e41f08506aff6747801c9 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:48.149734 2985 tablet.cc:1722] T cd23b8094d7e472eba6bd170248858a9 P 23eeffd3de254ce18db5febfde971c6b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251025 07:52:48.153700 2990 consensus_queue.cc:1048] T 4a1ec8c2ed7f41abb588f592f62097d1 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:48.106603 2146 ts_tablet_manager.cc:1929] T dda0b26e4f1a44928bc4b84f333901c0 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.108163 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e47c4f741f354a19b49777bd06e54764" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:48.112044 2686 raft_consensus.cc:493] T 645263a61b654f04985c40d04506c087 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:48.154611 2686 raft_consensus.cc:515] T 645263a61b654f04985c40d04506c087 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
W20251025 07:52:48.155061 710 leader_election.cc:343] T e47c4f741f354a19b49777bd06e54764 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.155179 710 leader_election.cc:304] T e47c4f741f354a19b49777bd06e54764 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:48.155234 2686 leader_election.cc:290] T 645263a61b654f04985c40d04506c087 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:48.155383 2981 raft_consensus.cc:2749] T e47c4f741f354a19b49777bd06e54764 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:48.155506 2146 log.cc:1199] T dda0b26e4f1a44928bc4b84f333901c0 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/dda0b26e4f1a44928bc4b84f333901c0
I20251025 07:52:48.156059 765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "645263a61b654f04985c40d04506c087" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:48.156250 765 raft_consensus.cc:2468] T 645263a61b654f04985c40d04506c087 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:48.156767 847 leader_election.cc:304] T 645263a61b654f04985c40d04506c087 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:48.156989 2686 raft_consensus.cc:2804] T 645263a61b654f04985c40d04506c087 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:48.151877 1167 tablet_bootstrap.cc:492] T df9db9dd31d840e0b1394db04d662a62 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:48.157099 2686 raft_consensus.cc:493] T 645263a61b654f04985c40d04506c087 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:48.157181 2686 raft_consensus.cc:3060] T 645263a61b654f04985c40d04506c087 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.157305 1167 ts_tablet_manager.cc:1403] T df9db9dd31d840e0b1394db04d662a62 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.140s user 0.005s sys 0.001s
I20251025 07:52:48.158382 1167 raft_consensus.cc:359] T df9db9dd31d840e0b1394db04d662a62 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.158579 1167 raft_consensus.cc:385] T df9db9dd31d840e0b1394db04d662a62 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.159862 1167 raft_consensus.cc:740] T df9db9dd31d840e0b1394db04d662a62 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.160024 2686 raft_consensus.cc:515] T 645263a61b654f04985c40d04506c087 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.160377 1167 consensus_queue.cc:260] T df9db9dd31d840e0b1394db04d662a62 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.160709 2686 leader_election.cc:290] T 645263a61b654f04985c40d04506c087 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:48.161121 765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "645263a61b654f04985c40d04506c087" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:48.161293 765 raft_consensus.cc:3060] T 645263a61b654f04985c40d04506c087 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:48.162914 850 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:45417, user_credentials={real_user=slave}} blocked reactor thread for 56962.3us
I20251025 07:52:48.153759 2166 log.cc:1199] T ce1ebb1ec14b4801930a72765bca1a7d P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/ce1ebb1ec14b4801930a72765bca1a7d
I20251025 07:52:48.163478 2166 ts_tablet_manager.cc:1950] T ce1ebb1ec14b4801930a72765bca1a7d P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:48.163776 2671 raft_consensus.cc:493] T 143aa3fce9ba4850b000a94054e5dc3a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:48.163908 2671 raft_consensus.cc:515] T 143aa3fce9ba4850b000a94054e5dc3a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.164418 2671 leader_election.cc:290] T 143aa3fce9ba4850b000a94054e5dc3a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:48.164649 765 raft_consensus.cc:2468] T 645263a61b654f04985c40d04506c087 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:48.165261 765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143aa3fce9ba4850b000a94054e5dc3a" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:48.164997 847 leader_election.cc:304] T 645263a61b654f04985c40d04506c087 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:48.165300 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143aa3fce9ba4850b000a94054e5dc3a" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:48.166033 1167 ts_tablet_manager.cc:1434] T df9db9dd31d840e0b1394db04d662a62 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251025 07:52:48.166184 982 leader_election.cc:304] T 143aa3fce9ba4850b000a94054e5dc3a P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:48.166594 2671 raft_consensus.cc:2749] T 143aa3fce9ba4850b000a94054e5dc3a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:48.166879 2886 raft_consensus.cc:2804] T 645263a61b654f04985c40d04506c087 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:48.167021 2886 raft_consensus.cc:697] T 645263a61b654f04985c40d04506c087 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:48.167063 1167 tablet_bootstrap.cc:492] T df0d86ad301245c1a6bcb44805936d25 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:48.167255 2886 consensus_queue.cc:237] T 645263a61b654f04985c40d04506c087 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.169185 1167 tablet_bootstrap.cc:654] T df0d86ad301245c1a6bcb44805936d25 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.169126 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c11f674385184ae6b235f1db8bf1fb91" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:48.170429 847 consensus_peers.cc:597] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382 -> Peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Couldn't send request to peer 23eeffd3de254ce18db5febfde971c6b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5b6383c99e9e4c90a374da9ccd830a6f. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:48.171116 2169 ts_tablet_manager.cc:1929] T f43ef51ee45d47a99ec732b49d50d3b4 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.175434 1166 tablet_bootstrap.cc:654] T a7ab3150a35b4caca9119a1374d3f2f7 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.175971 2146 ts_tablet_manager.cc:1950] T dda0b26e4f1a44928bc4b84f333901c0 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:48.176301 2169 log.cc:1199] T f43ef51ee45d47a99ec732b49d50d3b4 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/f43ef51ee45d47a99ec732b49d50d3b4
I20251025 07:52:48.174556 1042 raft_consensus.cc:3060] T 34c63b69904d499bb0d286941a8303f2 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.176810 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 166) took 6760 ms. Trace:
I20251025 07:52:48.176905 2146 rpcz_store.cc:276] 1025 07:52:41.416025 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.417123 (+ 1098us) service_pool.cc:225] Handling call
1025 07:52:47.964248 (+6547125us) ts_tablet_manager.cc:1151] Deleting tablet dda0b26e4f1a44928bc4b84f333901c0
1025 07:52:47.964261 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.001506 (+ 37245us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.106493 (+104987us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.176791 (+ 70298us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6440380}
I20251025 07:52:48.177237 2146 tablet_replica.cc:333] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:48.177436 2146 raft_consensus.cc:2243] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:48.177410 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet dda0b26e4f1a44928bc4b84f333901c0 (table default.Table_c1bf6bb7c62c4648853b9a5659c73da0 [id=d0b9f20f28d340f4944e85ee0f563fab]) successfully deleted
I20251025 07:52:48.177572 2146 pending_rounds.cc:70] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382: Trying to abort 1 pending ops.
I20251025 07:52:48.177613 2146 pending_rounds.cc:77] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214607421787213824 op_type: NO_OP noop_request { }
I20251025 07:52:48.177718 2146 raft_consensus.cc:2889] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251025 07:52:48.177798 2146 raft_consensus.cc:2272] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.178311 2146 ts_tablet_manager.cc:1916] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.179222 2169 ts_tablet_manager.cc:1950] T f43ef51ee45d47a99ec732b49d50d3b4 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:48.179703 1169 tablet_bootstrap.cc:654] T dda0b26e4f1a44928bc4b84f333901c0 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.183702 1166 tablet_bootstrap.cc:492] T a7ab3150a35b4caca9119a1374d3f2f7 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:48.183833 1166 ts_tablet_manager.cc:1403] T a7ab3150a35b4caca9119a1374d3f2f7 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.104s user 0.002s sys 0.001s
I20251025 07:52:48.184695 1166 raft_consensus.cc:359] T a7ab3150a35b4caca9119a1374d3f2f7 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.184878 1166 raft_consensus.cc:385] T a7ab3150a35b4caca9119a1374d3f2f7 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.184931 1166 raft_consensus.cc:740] T a7ab3150a35b4caca9119a1374d3f2f7 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.185108 1166 consensus_queue.cc:260] T a7ab3150a35b4caca9119a1374d3f2f7 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.185444 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 184) took 6725 ms. Trace:
I20251025 07:52:48.185566 1166 ts_tablet_manager.cc:1434] T a7ab3150a35b4caca9119a1374d3f2f7 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:48.185523 2169 rpcz_store.cc:276] 1025 07:52:41.459773 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.459822 (+ 49us) service_pool.cc:225] Handling call
1025 07:52:48.065436 (+6605614us) ts_tablet_manager.cc:1151] Deleting tablet f43ef51ee45d47a99ec732b49d50d3b4
1025 07:52:48.065445 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.088513 (+ 23068us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.170990 (+ 82477us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.185424 (+ 14434us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6605458}
I20251025 07:52:48.185808 1166 tablet_bootstrap.cc:492] T d77704b4fe6c4456a2ba40ffec0c60fc P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:48.186334 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "645263a61b654f04985c40d04506c087" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:48.186636 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet f43ef51ee45d47a99ec732b49d50d3b4 (table default.Table_0b0bd6b71b92454fb8cdf967ffd8349e [id=e9b21659de7d4276aa1f9effb7076eb4]) successfully deleted
I20251025 07:52:48.186686 1043 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "645263a61b654f04985c40d04506c087" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:48.186834 1045 raft_consensus.cc:3060] T 627a9cfd398346f2aa34c8932a11b5a8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.188051 1166 tablet_bootstrap.cc:654] T d77704b4fe6c4456a2ba40ffec0c60fc P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.171139 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 167) took 6689 ms. Trace:
I20251025 07:52:48.188333 2166 rpcz_store.cc:276] 1025 07:52:41.481756 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.484440 (+ 2684us) service_pool.cc:225] Handling call
1025 07:52:48.026890 (+6542450us) ts_tablet_manager.cc:1151] Deleting tablet ce1ebb1ec14b4801930a72765bca1a7d
1025 07:52:48.026901 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.110867 (+ 83966us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.141705 (+ 30838us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.171119 (+ 29414us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3245,"tablet-delete.queue_time_us":6542236}
I20251025 07:52:48.188812 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet ce1ebb1ec14b4801930a72765bca1a7d (table default.Table_d5abcbf8c8c741a2b40880a3467f2bcd [id=eab2657210e74ee6ac6f59544c9ba12e]) successfully deleted
I20251025 07:52:48.189240 2166 tablet_replica.cc:333] T 050a5d3b5f934dd19bf2e221f2fe0780 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:48.189483 2166 raft_consensus.cc:2243] T 050a5d3b5f934dd19bf2e221f2fe0780 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:48.189687 2166 raft_consensus.cc:2272] T 050a5d3b5f934dd19bf2e221f2fe0780 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.190232 2166 ts_tablet_manager.cc:1916] T 050a5d3b5f934dd19bf2e221f2fe0780 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.192003 2826 raft_consensus.cc:493] T fa826b85f0c94150a17d4e417dd3d62f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:48.192157 2826 raft_consensus.cc:515] T fa826b85f0c94150a17d4e417dd3d62f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.193275 2826 leader_election.cc:290] T fa826b85f0c94150a17d4e417dd3d62f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:48.193421 2826 raft_consensus.cc:493] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:48.193356 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa826b85f0c94150a17d4e417dd3d62f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:48.193553 2826 raft_consensus.cc:515] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.193722 772 raft_consensus.cc:2468] T fa826b85f0c94150a17d4e417dd3d62f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:48.194156 847 leader_election.cc:304] T fa826b85f0c94150a17d4e417dd3d62f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:48.194721 2826 leader_election.cc:290] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:48.194777 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aeba18be0cb4eb9949aeaf6ad73a812" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:48.194970 772 raft_consensus.cc:2468] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:48.195056 2990 raft_consensus.cc:2804] T fa826b85f0c94150a17d4e417dd3d62f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:48.195149 2990 raft_consensus.cc:493] T fa826b85f0c94150a17d4e417dd3d62f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:48.195211 2990 raft_consensus.cc:3060] T fa826b85f0c94150a17d4e417dd3d62f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.195411 847 leader_election.cc:304] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
W20251025 07:52:48.195600 850 leader_election.cc:343] T c11f674385184ae6b235f1db8bf1fb91 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.196719 2991 raft_consensus.cc:2804] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:48.196789 2991 raft_consensus.cc:493] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:48.196846 2991 raft_consensus.cc:3060] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.198073 2981 raft_consensus.cc:493] T 6ce62847e80f48e8807fdbbbdc50803c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:48.198215 2981 raft_consensus.cc:515] T 6ce62847e80f48e8807fdbbbdc50803c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.198768 2981 leader_election.cc:290] T 6ce62847e80f48e8807fdbbbdc50803c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:48.199405 2991 raft_consensus.cc:515] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.199901 2991 leader_election.cc:290] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:48.200407 2991 consensus_queue.cc:1048] T 34c63b69904d499bb0d286941a8303f2 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:48.197858 2990 raft_consensus.cc:515] T fa826b85f0c94150a17d4e417dd3d62f P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.201301 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ce62847e80f48e8807fdbbbdc50803c" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:48.201550 2990 leader_election.cc:290] T fa826b85f0c94150a17d4e417dd3d62f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:48.201857 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aeba18be0cb4eb9949aeaf6ad73a812" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
W20251025 07:52:48.201870 710 leader_election.cc:343] T 6ce62847e80f48e8807fdbbbdc50803c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.202034 772 raft_consensus.cc:3060] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.202608 771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa826b85f0c94150a17d4e417dd3d62f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:48.202740 771 raft_consensus.cc:3060] T fa826b85f0c94150a17d4e417dd3d62f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.202952 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ce62847e80f48e8807fdbbbdc50803c" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:48.203359 2146 ts_tablet_manager.cc:1929] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.204982 772 raft_consensus.cc:2468] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:48.205207 771 raft_consensus.cc:2468] T fa826b85f0c94150a17d4e417dd3d62f P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
W20251025 07:52:48.205485 712 leader_election.cc:343] T 6ce62847e80f48e8807fdbbbdc50803c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.205585 712 leader_election.cc:304] T 6ce62847e80f48e8807fdbbbdc50803c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:48.205966 2981 raft_consensus.cc:2749] T 6ce62847e80f48e8807fdbbbdc50803c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:48.205410 847 leader_election.cc:304] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:48.206398 847 leader_election.cc:304] T fa826b85f0c94150a17d4e417dd3d62f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:48.206517 2991 raft_consensus.cc:2804] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:48.206684 2990 raft_consensus.cc:2804] T fa826b85f0c94150a17d4e417dd3d62f P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:48.206851 2990 raft_consensus.cc:697] T fa826b85f0c94150a17d4e417dd3d62f P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
W20251025 07:52:48.206703 850 leader_election.cc:343] T 645263a61b654f04985c40d04506c087 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.207123 2990 consensus_queue.cc:237] T fa826b85f0c94150a17d4e417dd3d62f P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.208029 2991 raft_consensus.cc:697] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
W20251025 07:52:48.208349 850 consensus_peers.cc:597] T 716fb61312e94a4d9eb8ab8336ca140c P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:48.208308 2991 consensus_queue.cc:237] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.209008 2146 log.cc:1199] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/91728987303b41fd840eb56f3a311c5f
I20251025 07:52:48.209424 2146 ts_tablet_manager.cc:1950] T 91728987303b41fd840eb56f3a311c5f P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:48.209859 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 185) took 6749 ms. Trace:
I20251025 07:52:48.209962 2169 rpcz_store.cc:276] 1025 07:52:41.460176 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.460227 (+ 51us) service_pool.cc:225] Handling call
1025 07:52:48.209708 (+6749481us) ts_tablet_manager.cc:1151] Deleting tablet d37758f606344127b3db74c87babe69c
1025 07:52:48.209720 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.209838 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6749343}
I20251025 07:52:48.210381 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 170) took 6793 ms. Trace:
W20251025 07:52:48.210597 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet d37758f606344127b3db74c87babe69c because the tablet was not found. No further retry: Not found: Tablet not found: d37758f606344127b3db74c87babe69c
I20251025 07:52:48.210480 2146 rpcz_store.cc:276] 1025 07:52:41.416453 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.417288 (+ 835us) service_pool.cc:225] Handling call
1025 07:52:48.177181 (+6759893us) ts_tablet_manager.cc:1151] Deleting tablet 91728987303b41fd840eb56f3a311c5f
1025 07:52:48.177190 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.181703 (+ 4513us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.203245 (+ 21542us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.210359 (+ 7114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6759719}
I20251025 07:52:48.210911 2146 tablet_replica.cc:333] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:48.211042 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 91728987303b41fd840eb56f3a311c5f (table default.Table_b53575428f5b4c25a5c912428e5d4f2d [id=88b6e7cfe0c74d169585a1b064d58e4a]) successfully deleted
I20251025 07:52:48.211108 2146 raft_consensus.cc:2243] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:48.211424 2146 raft_consensus.cc:2272] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.211566 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 186) took 6751 ms. Trace:
I20251025 07:52:48.211654 2169 rpcz_store.cc:276] 1025 07:52:41.460508 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.460547 (+ 39us) service_pool.cc:225] Handling call
1025 07:52:48.211384 (+6750837us) ts_tablet_manager.cc:1151] Deleting tablet 3b940fa5ae26427dafbcfd26372754b3
1025 07:52:48.211394 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.211545 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6750710}
W20251025 07:52:48.212107 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 3b940fa5ae26427dafbcfd26372754b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b940fa5ae26427dafbcfd26372754b3 already in progress: creating tablet
I20251025 07:52:48.212440 2169 tablet_replica.cc:333] T b3a0af893a8947848e12093d0a7f0229 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:48.212649 2169 raft_consensus.cc:2243] T b3a0af893a8947848e12093d0a7f0229 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:48.212988 2169 raft_consensus.cc:2272] T b3a0af893a8947848e12093d0a7f0229 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.213565 2169 ts_tablet_manager.cc:1916] T b3a0af893a8947848e12093d0a7f0229 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:48.215155 850 leader_election.cc:343] T 645263a61b654f04985c40d04506c087 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.216287 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aeba18be0cb4eb9949aeaf6ad73a812" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:48.216737 2989 consensus_queue.cc:1048] T 37c7938e8c5e41f08506aff6747801c9 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:48.217535 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa826b85f0c94150a17d4e417dd3d62f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:48.218210 2989 consensus_queue.cc:1048] T 570201e7bf05406a97c81dde071d894b P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:48.220299 2146 ts_tablet_manager.cc:1916] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.231237 2981 raft_consensus.cc:493] T d0240d20f9904b4bbc8bf7f0f05e8850 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:48.231432 2981 raft_consensus.cc:515] T d0240d20f9904b4bbc8bf7f0f05e8850 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.232151 2981 leader_election.cc:290] T d0240d20f9904b4bbc8bf7f0f05e8850 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:48.232425 1167 tablet_bootstrap.cc:492] T df0d86ad301245c1a6bcb44805936d25 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:48.232591 1167 ts_tablet_manager.cc:1403] T df0d86ad301245c1a6bcb44805936d25 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.066s user 0.003s sys 0.001s
I20251025 07:52:48.232868 2146 ts_tablet_manager.cc:1929] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.233006 2146 log.cc:1199] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/5b6383c99e9e4c90a374da9ccd830a6f
I20251025 07:52:48.233294 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0240d20f9904b4bbc8bf7f0f05e8850" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:48.233892 710 leader_election.cc:343] T d0240d20f9904b4bbc8bf7f0f05e8850 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.234103 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0240d20f9904b4bbc8bf7f0f05e8850" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:48.234745 712 leader_election.cc:343] T d0240d20f9904b4bbc8bf7f0f05e8850 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.234861 712 leader_election.cc:304] T d0240d20f9904b4bbc8bf7f0f05e8850 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:48.235338 2981 raft_consensus.cc:2749] T d0240d20f9904b4bbc8bf7f0f05e8850 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:48.233899 1167 raft_consensus.cc:359] T df0d86ad301245c1a6bcb44805936d25 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.235913 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aeba18be0cb4eb9949aeaf6ad73a812" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:48.236042 1167 raft_consensus.cc:385] T df0d86ad301245c1a6bcb44805936d25 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.236238 1167 raft_consensus.cc:740] T df0d86ad301245c1a6bcb44805936d25 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.236791 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 783487b60c144f719ed2eb3c67b5ccab (table default.Table_09095c0829ed4d1cb8b763fc695e7fa8 [id=e372ca4384644779a6b37f389f549dec]). Expected version 1 got 0
I20251025 07:52:48.236642 1167 consensus_queue.cc:260] T df0d86ad301245c1a6bcb44805936d25 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.237229 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa826b85f0c94150a17d4e417dd3d62f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:48.237318 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f]). Expected version 1 got 0
I20251025 07:52:48.233366 2146 ts_tablet_manager.cc:1950] T 5b6383c99e9e4c90a374da9ccd830a6f P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:48.237875 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7]). Expected version 1 got 0
I20251025 07:52:48.237972 1167 ts_tablet_manager.cc:1434] T df0d86ad301245c1a6bcb44805936d25 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251025 07:52:48.238154 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 04d27d54fdb7483b869408af65ff85c9 (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]). Expected version 2 got 0
I20251025 07:52:48.238741 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 872cb7512d104827b1f1f186d33a4a44 (table default.Table_d5c65dce8b1b4214ba519d964012bfc0 [id=d4be274b36a845a99a4b96b8846248fd]). Expected version 1 got 0
I20251025 07:52:48.238785 1167 tablet_bootstrap.cc:492] T 2e098c9c4613459295b6043048cba587 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:48.239722 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet aa9d9ab8073945d28d52fb02213f201a (table default.Table_4791672658f043f6b97b9320a531ebb5 [id=dfd20cf9c3e541c4abe23d59e0aa7f2a]). Expected version 1 got 0
I20251025 07:52:48.241346 1167 tablet_bootstrap.cc:654] T 2e098c9c4613459295b6043048cba587 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.241770 1166 tablet_bootstrap.cc:492] T d77704b4fe6c4456a2ba40ffec0c60fc P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:48.241943 1166 ts_tablet_manager.cc:1403] T d77704b4fe6c4456a2ba40ffec0c60fc P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.056s user 0.005s sys 0.000s
W20251025 07:52:48.242377 850 leader_election.cc:343] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:48.242697 850 leader_election.cc:343] T fa826b85f0c94150a17d4e417dd3d62f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.242730 2988 raft_consensus.cc:493] T e9e8fffb8e984463b5fa2c7dd9e93227 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:48.242823 2988 raft_consensus.cc:515] T e9e8fffb8e984463b5fa2c7dd9e93227 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
W20251025 07:52:48.243026 850 leader_election.cc:343] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:48.243270 850 leader_election.cc:343] T fa826b85f0c94150a17d4e417dd3d62f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.243333 2988 leader_election.cc:290] T e9e8fffb8e984463b5fa2c7dd9e93227 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:48.244490 765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9e8fffb8e984463b5fa2c7dd9e93227" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:48.244668 765 raft_consensus.cc:2468] T e9e8fffb8e984463b5fa2c7dd9e93227 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:48.248399 847 leader_election.cc:304] T e9e8fffb8e984463b5fa2c7dd9e93227 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:48.248764 2826 consensus_queue.cc:1048] T 34c63b69904d499bb0d286941a8303f2 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:48.248687 2991 raft_consensus.cc:2804] T e9e8fffb8e984463b5fa2c7dd9e93227 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:48.251972 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9e8fffb8e984463b5fa2c7dd9e93227" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:48.252417 2991 raft_consensus.cc:493] T e9e8fffb8e984463b5fa2c7dd9e93227 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:48.252518 2991 raft_consensus.cc:3060] T e9e8fffb8e984463b5fa2c7dd9e93227 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.249598 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 168) took 6833 ms. Trace:
I20251025 07:52:48.254360 2146 rpcz_store.cc:276] 1025 07:52:41.416266 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.417491 (+ 1225us) service_pool.cc:225] Handling call
1025 07:52:48.210865 (+6793374us) ts_tablet_manager.cc:1151] Deleting tablet 5b6383c99e9e4c90a374da9ccd830a6f
1025 07:52:48.210874 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.223988 (+ 13114us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.232771 (+ 8783us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.249581 (+ 16810us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6302,"tablet-delete.queue_time_us":6793246}
I20251025 07:52:48.254905 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 175) took 6836 ms. Trace:
I20251025 07:52:48.255009 2146 rpcz_store.cc:276] 1025 07:52:41.418878 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.418926 (+ 48us) service_pool.cc:225] Handling call
1025 07:52:48.254757 (+6835831us) ts_tablet_manager.cc:1151] Deleting tablet d37758f606344127b3db74c87babe69c
1025 07:52:48.254773 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.254885 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6835676}
I20251025 07:52:48.255402 2146 tablet_replica.cc:333] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:48.255551 2166 ts_tablet_manager.cc:1929] T 050a5d3b5f934dd19bf2e221f2fe0780 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.255679 2166 log.cc:1199] T 050a5d3b5f934dd19bf2e221f2fe0780 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/050a5d3b5f934dd19bf2e221f2fe0780
I20251025 07:52:48.255661 2146 raft_consensus.cc:2243] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:48.256027 2166 ts_tablet_manager.cc:1950] T 050a5d3b5f934dd19bf2e221f2fe0780 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:48.256054 2146 raft_consensus.cc:2272] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.256875 2146 ts_tablet_manager.cc:1916] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.257843 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 5b6383c99e9e4c90a374da9ccd830a6f (table default.Table_32f8a762ec25446dac28e2729d6879bf [id=a13b2a42afca460595cd872ab2cf9ea0]) successfully deleted
W20251025 07:52:48.258252 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet d37758f606344127b3db74c87babe69c because the tablet was not found. No further retry: Not found: Tablet not found: d37758f606344127b3db74c87babe69c
I20251025 07:52:48.260164 1169 tablet_bootstrap.cc:492] T dda0b26e4f1a44928bc4b84f333901c0 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:48.260309 1169 ts_tablet_manager.cc:1403] T dda0b26e4f1a44928bc4b84f333901c0 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.125s user 0.001s sys 0.004s
I20251025 07:52:48.261086 1169 raft_consensus.cc:359] T dda0b26e4f1a44928bc4b84f333901c0 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.261262 1169 raft_consensus.cc:385] T dda0b26e4f1a44928bc4b84f333901c0 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.261318 1169 raft_consensus.cc:740] T dda0b26e4f1a44928bc4b84f333901c0 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.261521 1169 consensus_queue.cc:260] T dda0b26e4f1a44928bc4b84f333901c0 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.262039 1169 ts_tablet_manager.cc:1434] T dda0b26e4f1a44928bc4b84f333901c0 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 07:52:48.263814 1169 tablet_bootstrap.cc:492] T a0ea1edeca7347fc844faa5e4be1ae94 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:48.266494 1167 tablet_bootstrap.cc:492] T 2e098c9c4613459295b6043048cba587 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:48.266696 1167 ts_tablet_manager.cc:1403] T 2e098c9c4613459295b6043048cba587 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.002s
I20251025 07:52:48.267607 1167 raft_consensus.cc:359] T 2e098c9c4613459295b6043048cba587 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:48.267841 1167 raft_consensus.cc:385] T 2e098c9c4613459295b6043048cba587 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.267907 1167 raft_consensus.cc:740] T 2e098c9c4613459295b6043048cba587 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.269717 1166 raft_consensus.cc:359] T d77704b4fe6c4456a2ba40ffec0c60fc P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.269928 1166 raft_consensus.cc:385] T d77704b4fe6c4456a2ba40ffec0c60fc P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.270051 1166 raft_consensus.cc:740] T d77704b4fe6c4456a2ba40ffec0c60fc P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.270861 1166 consensus_queue.cc:260] T d77704b4fe6c4456a2ba40ffec0c60fc P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.271449 1166 ts_tablet_manager.cc:1434] T d77704b4fe6c4456a2ba40ffec0c60fc P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.026s user 0.001s sys 0.000s
I20251025 07:52:48.271744 1166 tablet_bootstrap.cc:492] T 4f7c15529be54a11961f385a953b0b03 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:48.274750 1043 raft_consensus.cc:3060] T 143aa3fce9ba4850b000a94054e5dc3a P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.276551 2991 raft_consensus.cc:515] T e9e8fffb8e984463b5fa2c7dd9e93227 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.278249 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]). Expected version 1 got 0
I20251025 07:52:48.278342 2990 raft_consensus.cc:493] T 6a33cfb2f2de48d4a5e9e39730d4900e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:48.278587 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]). Expected version 1 got 0
I20251025 07:52:48.278502 2990 raft_consensus.cc:515] T 6a33cfb2f2de48d4a5e9e39730d4900e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.278824 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_fa83d0528b4b481989bfc0ca35c6de44 [id=bf77b5ff6f1048dd996d948437f9ff5e]). Expected version 3 got 0
I20251025 07:52:48.279110 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet cd23b8094d7e472eba6bd170248858a9 (table default.Table_cea16af97a2242968d72900d3a84a86d [id=d82b3e1317434cf08a0d51e8a9427224]). Expected version 4 got 3
I20251025 07:52:48.279323 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet bc2d16b33ca8426abbd7bd38339c9f1d (table default.Table_4ea985df4c5b415c9570c64506a98cc4 [id=98d2b7797e33485e8872054aa2eb8ec9]). Expected version 1 got 0
I20251025 07:52:48.279316 2990 leader_election.cc:290] T 6a33cfb2f2de48d4a5e9e39730d4900e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:48.279449 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e]). Expected version 1 got 0
I20251025 07:52:48.279847 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet f70091024c034c118fb86578db19a31c (table default.Table_b7da40b96d6c4359bed6f898e8cf6275 [id=e041946c093b4c789ce175c736cc230b]). Expected version 2 got 0
I20251025 07:52:48.280157 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 5dab439bc50d4e869548154e1d222ad6 (table default.Table_fb9def2302784c999bf0f5707c60eb4c [id=45325c188eac4d34bb016d01de091f87]). Expected version 3 got 0
I20251025 07:52:48.280499 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]). Expected version 3 got 0
I20251025 07:52:48.268146 1167 consensus_queue.cc:260] T 2e098c9c4613459295b6043048cba587 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:48.281689 1166 tablet_bootstrap.cc:654] T 4f7c15529be54a11961f385a953b0b03 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.282049 1167 ts_tablet_manager.cc:1434] T 2e098c9c4613459295b6043048cba587 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20251025 07:52:48.282343 1167 tablet_bootstrap.cc:492] T 4b278ce7768349e49dcd1131e404ab34 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
W20251025 07:52:48.282864 850 leader_election.cc:343] T e9e8fffb8e984463b5fa2c7dd9e93227 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.278167 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9e8fffb8e984463b5fa2c7dd9e93227" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:48.283833 1167 tablet_bootstrap.cc:654] T 4b278ce7768349e49dcd1131e404ab34 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.280838 771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a33cfb2f2de48d4a5e9e39730d4900e" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:48.284025 771 raft_consensus.cc:2468] T 6a33cfb2f2de48d4a5e9e39730d4900e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:48.283845 772 raft_consensus.cc:3060] T e9e8fffb8e984463b5fa2c7dd9e93227 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.284482 847 leader_election.cc:304] T 6a33cfb2f2de48d4a5e9e39730d4900e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:48.284739 2990 raft_consensus.cc:2804] T 6a33cfb2f2de48d4a5e9e39730d4900e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:48.284832 2990 raft_consensus.cc:493] T 6a33cfb2f2de48d4a5e9e39730d4900e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:48.281452 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 170) took 6792 ms. Trace:
I20251025 07:52:48.284901 2990 raft_consensus.cc:3060] T 6a33cfb2f2de48d4a5e9e39730d4900e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.284941 2166 rpcz_store.cc:276] 1025 07:52:41.488882 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.488944 (+ 62us) service_pool.cc:225] Handling call
1025 07:52:48.189194 (+6700250us) ts_tablet_manager.cc:1151] Deleting tablet 050a5d3b5f934dd19bf2e221f2fe0780
1025 07:52:48.189204 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.228274 (+ 39070us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.255438 (+ 27164us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.281434 (+ 25996us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6700088}
I20251025 07:52:48.285293 2166 tablet_replica.cc:333] T f43ef51ee45d47a99ec732b49d50d3b4 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:48.285508 2166 raft_consensus.cc:2243] T f43ef51ee45d47a99ec732b49d50d3b4 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:48.285689 2166 raft_consensus.cc:2272] T f43ef51ee45d47a99ec732b49d50d3b4 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.286082 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 050a5d3b5f934dd19bf2e221f2fe0780 (table default.Table_3bbc24e4eb334b6487c9da0142b1fe03 [id=9f09387ac53b4027a147057debd10743]) successfully deleted
I20251025 07:52:48.286237 2166 ts_tablet_manager.cc:1916] T f43ef51ee45d47a99ec732b49d50d3b4 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.287047 772 raft_consensus.cc:2468] T e9e8fffb8e984463b5fa2c7dd9e93227 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:48.288385 847 leader_election.cc:304] T e9e8fffb8e984463b5fa2c7dd9e93227 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:48.287638 2990 raft_consensus.cc:515] T 6a33cfb2f2de48d4a5e9e39730d4900e P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.288951 2886 raft_consensus.cc:2804] T e9e8fffb8e984463b5fa2c7dd9e93227 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:48.289093 2886 raft_consensus.cc:697] T e9e8fffb8e984463b5fa2c7dd9e93227 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:48.289366 2886 consensus_queue.cc:237] T e9e8fffb8e984463b5fa2c7dd9e93227 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.289768 2990 leader_election.cc:290] T 6a33cfb2f2de48d4a5e9e39730d4900e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:48.292018 771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a33cfb2f2de48d4a5e9e39730d4900e" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:48.292524 771 raft_consensus.cc:3060] T 6a33cfb2f2de48d4a5e9e39730d4900e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.293725 2981 raft_consensus.cc:493] T a6ae1315b28940d49b667e3c80347021 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:48.293846 2981 raft_consensus.cc:515] T a6ae1315b28940d49b667e3c80347021 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.266089 1169 tablet_bootstrap.cc:654] T a0ea1edeca7347fc844faa5e4be1ae94 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.294412 2981 leader_election.cc:290] T a6ae1315b28940d49b667e3c80347021 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:48.295703 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6ae1315b28940d49b667e3c80347021" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:48.296212 712 leader_election.cc:343] T a6ae1315b28940d49b667e3c80347021 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.298365 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6ae1315b28940d49b667e3c80347021" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:48.298784 771 raft_consensus.cc:2468] T 6a33cfb2f2de48d4a5e9e39730d4900e P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
W20251025 07:52:48.298877 710 leader_election.cc:343] T a6ae1315b28940d49b667e3c80347021 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.299795 710 leader_election.cc:304] T a6ae1315b28940d49b667e3c80347021 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:48.300086 2981 raft_consensus.cc:2749] T a6ae1315b28940d49b667e3c80347021 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:48.300784 847 leader_election.cc:304] T 6a33cfb2f2de48d4a5e9e39730d4900e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:48.301168 2698 raft_consensus.cc:2804] T 6a33cfb2f2de48d4a5e9e39730d4900e P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:48.301242 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9e8fffb8e984463b5fa2c7dd9e93227" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:48.301332 2698 raft_consensus.cc:697] T 6a33cfb2f2de48d4a5e9e39730d4900e P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:48.301674 2698 consensus_queue.cc:237] T 6a33cfb2f2de48d4a5e9e39730d4900e P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
W20251025 07:52:48.303032 850 leader_election.cc:343] T e9e8fffb8e984463b5fa2c7dd9e93227 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.303220 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a33cfb2f2de48d4a5e9e39730d4900e" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:48.302435 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a33cfb2f2de48d4a5e9e39730d4900e" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:48.304972 2991 leader_election.cc:290] T e9e8fffb8e984463b5fa2c7dd9e93227 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
W20251025 07:52:48.305708 850 consensus_peers.cc:597] T aaef24ccdfba45fe8204327723df834e P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aaef24ccdfba45fe8204327723df834e. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:48.305716 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_d11bba62f65740319701b4337a5c255a [id=ad812862698d421184ebf501ce79b7be]). Expected version 3 got 0
I20251025 07:52:48.306061 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]). Expected version 1 got 0
W20251025 07:52:48.306399 850 leader_election.cc:343] T 6a33cfb2f2de48d4a5e9e39730d4900e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.306660 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 22f7db660616457b83b24e947a0b0bae (table default.Table_4a86191163f2408f9bcc795cd8d1bbf6 [id=bb316a7e000748ca902a9bfb494802d7]). Expected version 1 got 0
I20251025 07:52:48.306740 2981 raft_consensus.cc:493] T df0d86ad301245c1a6bcb44805936d25 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:48.306869 2981 raft_consensus.cc:515] T df0d86ad301245c1a6bcb44805936d25 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.307101 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 5fc092c2bb2d41ac8c5b4fc07460c8ec (table default.Table_cd20e202397645f8821dcde13b5c0ce3 [id=630c274339664b9db18ab4ef7c29c16f]). Expected version 2 got 0
I20251025 07:52:48.307472 2981 leader_election.cc:290] T df0d86ad301245c1a6bcb44805936d25 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:48.307682 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 0325d462441349ef81841f1b0cab2a20 (table default.Table_750ecb905839434fa9a1aa72f87394be [id=9ce3ba77c56a4ef09cb3b40ea0e488f6]). Expected version 1 got 0
I20251025 07:52:48.307914 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 3a0f50c8000144a4acb5219484b01800 (table default.Table_c62e0dc1ae504f9bb912013506ff3031 [id=6f395892b09240fea02482b3199cfb5e]). Expected version 1 got 0
I20251025 07:52:48.308147 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525]). Expected version 1 got 0
I20251025 07:52:48.308502 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df0d86ad301245c1a6bcb44805936d25" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:48.308533 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df0d86ad301245c1a6bcb44805936d25" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:48.308930 712 leader_election.cc:343] T df0d86ad301245c1a6bcb44805936d25 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:48.309204 710 leader_election.cc:343] T df0d86ad301245c1a6bcb44805936d25 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.309294 710 leader_election.cc:304] T df0d86ad301245c1a6bcb44805936d25 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:48.309763 2981 raft_consensus.cc:2749] T df0d86ad301245c1a6bcb44805936d25 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:48.312927 850 leader_election.cc:343] T 6a33cfb2f2de48d4a5e9e39730d4900e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.250099 2169 ts_tablet_manager.cc:1929] T b3a0af893a8947848e12093d0a7f0229 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.315124 2169 log.cc:1199] T b3a0af893a8947848e12093d0a7f0229 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/b3a0af893a8947848e12093d0a7f0229
I20251025 07:52:48.315207 1166 tablet_bootstrap.cc:492] T 4f7c15529be54a11961f385a953b0b03 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:48.315436 1166 ts_tablet_manager.cc:1403] T 4f7c15529be54a11961f385a953b0b03 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.000s
I20251025 07:52:48.315821 2169 ts_tablet_manager.cc:1950] T b3a0af893a8947848e12093d0a7f0229 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:48.316740 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 187) took 6855 ms. Trace:
I20251025 07:52:48.316880 2169 rpcz_store.cc:276] 1025 07:52:41.460810 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.460853 (+ 43us) service_pool.cc:225] Handling call
1025 07:52:48.212400 (+6751547us) ts_tablet_manager.cc:1151] Deleting tablet b3a0af893a8947848e12093d0a7f0229
1025 07:52:48.212408 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.246985 (+ 34577us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.250001 (+ 3016us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.316718 (+ 66717us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6751421}
I20251025 07:52:48.317919 2169 tablet_replica.cc:333] T f661d5909657455581dcb5e6eadd3c0b P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:48.318279 2169 raft_consensus.cc:2243] T f661d5909657455581dcb5e6eadd3c0b P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:48.318497 2169 raft_consensus.cc:2272] T f661d5909657455581dcb5e6eadd3c0b P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.319103 2169 ts_tablet_manager.cc:1916] T f661d5909657455581dcb5e6eadd3c0b P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.319687 2146 ts_tablet_manager.cc:1929] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.319835 2146 log.cc:1199] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/0f5f6ca6c3cd4d369690e425387a4344
I20251025 07:52:48.320209 2146 ts_tablet_manager.cc:1950] T 0f5f6ca6c3cd4d369690e425387a4344 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:48.320602 1169 tablet_bootstrap.cc:492] T a0ea1edeca7347fc844faa5e4be1ae94 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:48.320792 1169 ts_tablet_manager.cc:1403] T a0ea1edeca7347fc844faa5e4be1ae94 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.057s user 0.004s sys 0.000s
I20251025 07:52:48.324224 2981 raft_consensus.cc:493] T 2d24e3e7a9e4431db609331ad1fd57c9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:48.324374 2981 raft_consensus.cc:515] T 2d24e3e7a9e4431db609331ad1fd57c9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.324644 2886 consensus_queue.cc:1048] T 570201e7bf05406a97c81dde071d894b P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:48.324944 2981 leader_election.cc:290] T 2d24e3e7a9e4431db609331ad1fd57c9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:48.325780 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d24e3e7a9e4431db609331ad1fd57c9" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:48.326193 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 177) took 6906 ms. Trace:
I20251025 07:52:48.326324 2146 rpcz_store.cc:276] 1025 07:52:41.419357 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.419393 (+ 36us) service_pool.cc:225] Handling call
1025 07:52:48.255334 (+6835941us) ts_tablet_manager.cc:1151] Deleting tablet 0f5f6ca6c3cd4d369690e425387a4344
1025 07:52:48.255347 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.261318 (+ 5971us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.319560 (+ 58242us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.326106 (+ 6546us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4703,"tablet-delete.queue_time_us":6835724}
W20251025 07:52:48.326588 710 leader_election.cc:343] T 2d24e3e7a9e4431db609331ad1fd57c9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.327270 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 176) took 6908 ms. Trace:
I20251025 07:52:48.327356 1166 raft_consensus.cc:359] T 4f7c15529be54a11961f385a953b0b03 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.327558 1166 raft_consensus.cc:385] T 4f7c15529be54a11961f385a953b0b03 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.327420 2146 rpcz_store.cc:276] 1025 07:52:41.419262 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.419596 (+ 334us) service_pool.cc:225] Handling call
1025 07:52:48.327008 (+6907412us) ts_tablet_manager.cc:1151] Deleting tablet 3b940fa5ae26427dafbcfd26372754b3
1025 07:52:48.327097 (+ 89us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.327211 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6907278}
I20251025 07:52:48.328294 2146 tablet_replica.cc:333] T b3a0af893a8947848e12093d0a7f0229 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:48.328513 2146 raft_consensus.cc:2243] T b3a0af893a8947848e12093d0a7f0229 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:48.328702 2146 raft_consensus.cc:2272] T b3a0af893a8947848e12093d0a7f0229 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.330636 2146 ts_tablet_manager.cc:1916] T b3a0af893a8947848e12093d0a7f0229 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.317696 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet b3a0af893a8947848e12093d0a7f0229 (table default.Table_0f2c837d88c043fda9d4eb94706a5398 [id=c837adb3bd5a4436a4cd8ea6f7be6b9e]) successfully deleted
I20251025 07:52:48.326834 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 0f5f6ca6c3cd4d369690e425387a4344 (table default.Table_e317f8d728da4c76886f67d0995ab7f4 [id=4d2121dea4904850aecb65da5dd2d8f9]) successfully deleted
W20251025 07:52:48.334519 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 3b940fa5ae26427dafbcfd26372754b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b940fa5ae26427dafbcfd26372754b3 already in progress: creating tablet
I20251025 07:52:48.334517 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_67b99749f3bf43228b40d8b9807e69b4 [id=fc8180c09d7b4e74b2e161f7acb8fe0e]). Expected version 2 got 0
I20251025 07:52:48.335238 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5]). Expected version 1 got 0
I20251025 07:52:48.335604 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c]). Expected version 1 got 0
I20251025 07:52:48.336022 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3]). Expected version 1 got 0
I20251025 07:52:48.336285 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 755f0d6c87044e20a8020f8410940123 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 4 got 0
I20251025 07:52:48.336539 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 830fdbc6812e416690d0db42a73af503 (table default.Table_92824b3f2a774564b3aea04aaa25d204 [id=fb21a45b86ab446e8c3dae4512a9598f]). Expected version 1 got 0
I20251025 07:52:48.337146 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236]). Expected version 1 got 0
I20251025 07:52:48.342846 2146 ts_tablet_manager.cc:1929] T b3a0af893a8947848e12093d0a7f0229 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.343031 2146 log.cc:1199] T b3a0af893a8947848e12093d0a7f0229 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/b3a0af893a8947848e12093d0a7f0229
I20251025 07:52:48.343458 2146 ts_tablet_manager.cc:1950] T b3a0af893a8947848e12093d0a7f0229 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:48.344053 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d24e3e7a9e4431db609331ad1fd57c9" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:48.344223 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 182) took 6922 ms. Trace:
I20251025 07:52:48.344417 2146 rpcz_store.cc:276] 1025 07:52:41.421717 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.421757 (+ 40us) service_pool.cc:225] Handling call
1025 07:52:48.327991 (+6906234us) ts_tablet_manager.cc:1151] Deleting tablet b3a0af893a8947848e12093d0a7f0229
1025 07:52:48.328253 (+ 262us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.339815 (+ 11562us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.342749 (+ 2934us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.344202 (+ 1453us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6906085}
W20251025 07:52:48.344658 712 leader_election.cc:343] T 2d24e3e7a9e4431db609331ad1fd57c9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.344813 712 leader_election.cc:304] T 2d24e3e7a9e4431db609331ad1fd57c9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:48.345265 2146 tablet_replica.cc:333] T f97acc9a638b4d5a968dcc3412a384bc P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:48.345270 2981 raft_consensus.cc:2749] T 2d24e3e7a9e4431db609331ad1fd57c9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:48.345481 2146 raft_consensus.cc:2243] T f97acc9a638b4d5a968dcc3412a384bc P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:48.344967 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet b3a0af893a8947848e12093d0a7f0229 (table default.Table_0f2c837d88c043fda9d4eb94706a5398 [id=c837adb3bd5a4436a4cd8ea6f7be6b9e]) successfully deleted
I20251025 07:52:48.345793 2146 raft_consensus.cc:2272] T f97acc9a638b4d5a968dcc3412a384bc P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.347198 2146 ts_tablet_manager.cc:1916] T f97acc9a638b4d5a968dcc3412a384bc P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.347726 1169 raft_consensus.cc:359] T a0ea1edeca7347fc844faa5e4be1ae94 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.347903 1169 raft_consensus.cc:385] T a0ea1edeca7347fc844faa5e4be1ae94 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.347954 1169 raft_consensus.cc:740] T a0ea1edeca7347fc844faa5e4be1ae94 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.348137 1169 consensus_queue.cc:260] T a0ea1edeca7347fc844faa5e4be1ae94 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.348518 1166 raft_consensus.cc:740] T 4f7c15529be54a11961f385a953b0b03 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.348681 1169 ts_tablet_manager.cc:1434] T a0ea1edeca7347fc844faa5e4be1ae94 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20251025 07:52:48.348773 1166 consensus_queue.cc:260] T 4f7c15529be54a11961f385a953b0b03 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.348970 1169 tablet_bootstrap.cc:492] T 716fb61312e94a4d9eb8ab8336ca140c P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:48.349314 1166 ts_tablet_manager.cc:1434] T 4f7c15529be54a11961f385a953b0b03 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.034s user 0.021s sys 0.000s
I20251025 07:52:48.350217 1167 tablet_bootstrap.cc:492] T 4b278ce7768349e49dcd1131e404ab34 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:48.350369 1167 ts_tablet_manager.cc:1403] T 4b278ce7768349e49dcd1131e404ab34 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.068s user 0.003s sys 0.000s
I20251025 07:52:48.350639 1169 tablet_bootstrap.cc:654] T 716fb61312e94a4d9eb8ab8336ca140c P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.351007 1167 raft_consensus.cc:359] T 4b278ce7768349e49dcd1131e404ab34 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.351155 1167 raft_consensus.cc:385] T 4b278ce7768349e49dcd1131e404ab34 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.351200 1167 raft_consensus.cc:740] T 4b278ce7768349e49dcd1131e404ab34 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.351392 1167 consensus_queue.cc:260] T 4b278ce7768349e49dcd1131e404ab34 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.360898 2991 raft_consensus.cc:493] T e8402fd33ba842e981c6c68787c7c548 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:48.361117 2991 raft_consensus.cc:515] T e8402fd33ba842e981c6c68787c7c548 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:48.361917 2991 leader_election.cc:290] T e8402fd33ba842e981c6c68787c7c548 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:48.362635 771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8402fd33ba842e981c6c68787c7c548" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:48.362906 771 raft_consensus.cc:2468] T e8402fd33ba842e981c6c68787c7c548 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:48.363916 1167 ts_tablet_manager.cc:1434] T 4b278ce7768349e49dcd1131e404ab34 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20251025 07:52:48.364025 847 leader_election.cc:304] T e8402fd33ba842e981c6c68787c7c548 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:48.364225 1167 tablet_bootstrap.cc:492] T 4f7c15529be54a11961f385a953b0b03 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:48.364477 2991 raft_consensus.cc:2804] T e8402fd33ba842e981c6c68787c7c548 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:48.364576 2991 raft_consensus.cc:493] T e8402fd33ba842e981c6c68787c7c548 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:48.364670 2991 raft_consensus.cc:3060] T e8402fd33ba842e981c6c68787c7c548 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.364863 1166 tablet_bootstrap.cc:492] T 5dab439bc50d4e869548154e1d222ad6 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:48.365880 1167 tablet_bootstrap.cc:654] T 4f7c15529be54a11961f385a953b0b03 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.366331 1166 tablet_bootstrap.cc:654] T 5dab439bc50d4e869548154e1d222ad6 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.367106 2166 ts_tablet_manager.cc:1929] T f43ef51ee45d47a99ec732b49d50d3b4 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.367213 2166 log.cc:1199] T f43ef51ee45d47a99ec732b49d50d3b4 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/f43ef51ee45d47a99ec732b49d50d3b4
I20251025 07:52:48.367534 2166 ts_tablet_manager.cc:1950] T f43ef51ee45d47a99ec732b49d50d3b4 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:48.368883 2991 raft_consensus.cc:515] T e8402fd33ba842e981c6c68787c7c548 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:48.369146 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 172) took 6880 ms. Trace:
I20251025 07:52:48.369239 2166 rpcz_store.cc:276] 1025 07:52:41.489099 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.489570 (+ 471us) service_pool.cc:225] Handling call
1025 07:52:48.285247 (+6795677us) ts_tablet_manager.cc:1151] Deleting tablet f43ef51ee45d47a99ec732b49d50d3b4
1025 07:52:48.285257 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.289303 (+ 4046us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.366993 (+ 77690us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.369126 (+ 2133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6795522}
I20251025 07:52:48.370177 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet f43ef51ee45d47a99ec732b49d50d3b4 (table default.Table_0b0bd6b71b92454fb8cdf967ffd8349e [id=e9b21659de7d4276aa1f9effb7076eb4]) successfully deleted
I20251025 07:52:48.370415 771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8402fd33ba842e981c6c68787c7c548" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:48.370635 771 raft_consensus.cc:3060] T e8402fd33ba842e981c6c68787c7c548 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.371423 2146 ts_tablet_manager.cc:1929] T f97acc9a638b4d5a968dcc3412a384bc P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.371531 2146 log.cc:1199] T f97acc9a638b4d5a968dcc3412a384bc P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/f97acc9a638b4d5a968dcc3412a384bc
I20251025 07:52:48.371850 2146 ts_tablet_manager.cc:1950] T f97acc9a638b4d5a968dcc3412a384bc P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:48.373163 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 180) took 6951 ms. Trace:
I20251025 07:52:48.373238 2146 rpcz_store.cc:276] 1025 07:52:41.421521 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.421923 (+ 402us) service_pool.cc:225] Handling call
1025 07:52:48.345218 (+6923295us) ts_tablet_manager.cc:1151] Deleting tablet f97acc9a638b4d5a968dcc3412a384bc
1025 07:52:48.345228 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.368286 (+ 23058us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.371332 (+ 3046us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.373145 (+ 1813us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6923152}
I20251025 07:52:48.373667 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet f97acc9a638b4d5a968dcc3412a384bc (table default.Table_e50bf47b78d94bbc811e6f3fb3551edb [id=0827603890c8466082bc6c306c9c66af]) successfully deleted
W20251025 07:52:48.374379 439 connection.cc:537] client connection to 127.0.104.61:42203 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 07:52:48.374511 439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 9 similar messages]
I20251025 07:52:48.374580 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 181) took 6952 ms. Trace:
I20251025 07:52:48.374603 771 raft_consensus.cc:2468] T e8402fd33ba842e981c6c68787c7c548 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:48.374642 2146 rpcz_store.cc:276] 1025 07:52:41.421625 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.422086 (+ 461us) service_pool.cc:225] Handling call
1025 07:52:48.374467 (+6952381us) ts_tablet_manager.cc:1151] Deleting tablet 410ef150a7014c6c867a021b544a2f4e
1025 07:52:48.374478 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.374564 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6952226}
W20251025 07:52:48.375145 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 410ef150a7014c6c867a021b544a2f4e because the tablet was not found. No further retry: Not found: Tablet not found: 410ef150a7014c6c867a021b544a2f4e
I20251025 07:52:48.375223 847 leader_election.cc:304] T e8402fd33ba842e981c6c68787c7c548 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:48.375707 2886 raft_consensus.cc:2804] T e8402fd33ba842e981c6c68787c7c548 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:48.375784 2991 leader_election.cc:290] T e8402fd33ba842e981c6c68787c7c548 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:48.375862 2886 raft_consensus.cc:697] T e8402fd33ba842e981c6c68787c7c548 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:48.376098 2886 consensus_queue.cc:237] T e8402fd33ba842e981c6c68787c7c548 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:48.376217 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8402fd33ba842e981c6c68787c7c548" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:48.376353 2146 tablet_replica.cc:333] T aaef24ccdfba45fe8204327723df834e P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:48.376690 2146 raft_consensus.cc:2243] T aaef24ccdfba45fe8204327723df834e P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:48.377238 2146 raft_consensus.cc:2272] T aaef24ccdfba45fe8204327723df834e P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:48.377743 850 leader_election.cc:343] T e8402fd33ba842e981c6c68787c7c548 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.378525 2146 ts_tablet_manager.cc:1916] T aaef24ccdfba45fe8204327723df834e P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.381783 2597 raft_consensus.cc:493] T 141e413d2faa4c67b4c822192e631417 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:48.381922 2597 raft_consensus.cc:515] T 141e413d2faa4c67b4c822192e631417 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.382560 2597 leader_election.cc:290] T 141e413d2faa4c67b4c822192e631417 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:48.383318 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "141e413d2faa4c67b4c822192e631417" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:48.383541 909 raft_consensus.cc:2393] T 141e413d2faa4c67b4c822192e631417 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:48.384131 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 8707b93d491e4e4386d148087a2f0870 (table default.Table_799dcb1cef2f4fe6b9b8f17f9f1026c5 [id=59840a567e5244e8a9024cbad42471e4]). Expected version 1 got 0
I20251025 07:52:48.385640 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet b56209e4934e4407a8b4518ca4af0d46 (table default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de]). Expected version 1 got 0
I20251025 07:52:48.370644 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 173) took 6880 ms. Trace:
I20251025 07:52:48.386620 2166 rpcz_store.cc:276] 1025 07:52:41.489838 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.489967 (+ 129us) service_pool.cc:225] Handling call
1025 07:52:48.370539 (+6880572us) ts_tablet_manager.cc:1151] Deleting tablet d37758f606344127b3db74c87babe69c
1025 07:52:48.370550 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.370627 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6880440}
I20251025 07:52:48.389164 2146 ts_tablet_manager.cc:1929] T aaef24ccdfba45fe8204327723df834e P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.389304 2146 log.cc:1199] T aaef24ccdfba45fe8204327723df834e P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/aaef24ccdfba45fe8204327723df834e
I20251025 07:52:48.389294 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8402fd33ba842e981c6c68787c7c548" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:48.389725 2146 ts_tablet_manager.cc:1950] T aaef24ccdfba45fe8204327723df834e P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
W20251025 07:52:48.389837 850 leader_election.cc:343] T e8402fd33ba842e981c6c68787c7c548 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.390731 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 179) took 6969 ms. Trace:
I20251025 07:52:48.390861 2146 rpcz_store.cc:276] 1025 07:52:41.421444 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.422262 (+ 818us) service_pool.cc:225] Handling call
1025 07:52:48.376309 (+6954047us) ts_tablet_manager.cc:1151] Deleting tablet aaef24ccdfba45fe8204327723df834e
1025 07:52:48.376317 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.385000 (+ 8683us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.389047 (+ 4047us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.390709 (+ 1662us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6953925}
I20251025 07:52:48.391352 2698 raft_consensus.cc:493] T fd265cdbf6a8440393a2b5d8240d1872 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:48.391494 2698 raft_consensus.cc:515] T fd265cdbf6a8440393a2b5d8240d1872 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:48.391641 2146 tablet_replica.cc:333] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:48.392051 2698 leader_election.cc:290] T fd265cdbf6a8440393a2b5d8240d1872 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:48.392104 2146 raft_consensus.cc:2243] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:48.392783 2146 raft_consensus.cc:2272] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.388583 2981 raft_consensus.cc:493] T 95efd553493e4966bc48a01cac6e1e10 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:48.393079 771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "141e413d2faa4c67b4c822192e631417" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:48.393266 771 raft_consensus.cc:2393] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 23eeffd3de254ce18db5febfde971c6b in this term.
I20251025 07:52:48.393198 2981 raft_consensus.cc:515] T 95efd553493e4966bc48a01cac6e1e10 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.393242 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd265cdbf6a8440393a2b5d8240d1872" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:48.393414 2146 ts_tablet_manager.cc:1916] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:48.393688 850 leader_election.cc:343] T fd265cdbf6a8440393a2b5d8240d1872 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.393770 2981 leader_election.cc:290] T 95efd553493e4966bc48a01cac6e1e10 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:48.392398 2888 raft_consensus.cc:493] T af743ee344e543a0aa6da3cd7a30e467 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:48.393926 2888 raft_consensus.cc:515] T af743ee344e543a0aa6da3cd7a30e467 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.394372 2888 leader_election.cc:290] T af743ee344e543a0aa6da3cd7a30e467 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:48.392431 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd265cdbf6a8440393a2b5d8240d1872" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:48.395123 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95efd553493e4966bc48a01cac6e1e10" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:48.395247 772 raft_consensus.cc:2468] T fd265cdbf6a8440393a2b5d8240d1872 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:48.395278 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af743ee344e543a0aa6da3cd7a30e467" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:48.395506 982 connection.cc:537] client connection to 127.0.104.61:42203 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 07:52:48.395803 709 connection.cc:537] client connection to 127.0.104.61:42203 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251025 07:52:48.396157 847 connection.cc:537] client connection to 127.0.104.61:42203 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251025 07:52:48.396613 982 leader_election.cc:304] T 141e413d2faa4c67b4c822192e631417 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:48.396659 1043 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af743ee344e543a0aa6da3cd7a30e467" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:48.397410 847 leader_election.cc:304] T fd265cdbf6a8440393a2b5d8240d1872 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:48.397914 2698 raft_consensus.cc:2804] T fd265cdbf6a8440393a2b5d8240d1872 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:48.397987 2698 raft_consensus.cc:493] T fd265cdbf6a8440393a2b5d8240d1872 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:48.398041 2698 raft_consensus.cc:3060] T fd265cdbf6a8440393a2b5d8240d1872 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:48.395815 710 leader_election.cc:343] T 95efd553493e4966bc48a01cac6e1e10 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:48.398422 710 leader_election.cc:343] T af743ee344e543a0aa6da3cd7a30e467 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.400719 2698 raft_consensus.cc:515] T fd265cdbf6a8440393a2b5d8240d1872 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:48.401968 1043 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd265cdbf6a8440393a2b5d8240d1872" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:48.402087 712 leader_election.cc:343] T af743ee344e543a0aa6da3cd7a30e467 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.402227 712 leader_election.cc:304] T af743ee344e543a0aa6da3cd7a30e467 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:48.402684 2888 raft_consensus.cc:2749] T af743ee344e543a0aa6da3cd7a30e467 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:48.402933 2597 raft_consensus.cc:3060] T 141e413d2faa4c67b4c822192e631417 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:48.406510 850 leader_election.cc:343] T fd265cdbf6a8440393a2b5d8240d1872 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.407198 2698 leader_election.cc:290] T fd265cdbf6a8440393a2b5d8240d1872 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:48.407909 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd265cdbf6a8440393a2b5d8240d1872" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:48.408125 772 raft_consensus.cc:3060] T fd265cdbf6a8440393a2b5d8240d1872 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.408471 1167 tablet_bootstrap.cc:492] T 4f7c15529be54a11961f385a953b0b03 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:48.408624 1167 ts_tablet_manager.cc:1403] T 4f7c15529be54a11961f385a953b0b03 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.000s
I20251025 07:52:48.409580 1167 raft_consensus.cc:359] T 4f7c15529be54a11961f385a953b0b03 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.409799 1167 raft_consensus.cc:385] T 4f7c15529be54a11961f385a953b0b03 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.409870 1167 raft_consensus.cc:740] T 4f7c15529be54a11961f385a953b0b03 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.412027 772 raft_consensus.cc:2468] T fd265cdbf6a8440393a2b5d8240d1872 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:48.412453 847 leader_election.cc:304] T fd265cdbf6a8440393a2b5d8240d1872 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:48.412837 2597 raft_consensus.cc:2749] T 141e413d2faa4c67b4c822192e631417 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:48.412915 2886 raft_consensus.cc:2804] T fd265cdbf6a8440393a2b5d8240d1872 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:48.413039 2886 raft_consensus.cc:697] T fd265cdbf6a8440393a2b5d8240d1872 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:48.414462 2886 consensus_queue.cc:237] T fd265cdbf6a8440393a2b5d8240d1872 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:48.416452 2698 raft_consensus.cc:493] T fce535a0c0bd45c6bf17f81add297f40 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:48.416656 2698 raft_consensus.cc:515] T fce535a0c0bd45c6bf17f81add297f40 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.417747 1043 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fce535a0c0bd45c6bf17f81add297f40" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:48.418438 850 leader_election.cc:343] T fce535a0c0bd45c6bf17f81add297f40 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.418617 2698 leader_election.cc:290] T fce535a0c0bd45c6bf17f81add297f40 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:48.420070 2990 consensus_queue.cc:1048] T 716fb61312e94a4d9eb8ab8336ca140c P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:48.420197 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fce535a0c0bd45c6bf17f81add297f40" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:48.420374 772 raft_consensus.cc:2468] T fce535a0c0bd45c6bf17f81add297f40 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:48.420889 847 leader_election.cc:304] T fce535a0c0bd45c6bf17f81add297f40 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:48.421126 2991 raft_consensus.cc:2804] T fce535a0c0bd45c6bf17f81add297f40 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:48.421195 2991 raft_consensus.cc:493] T fce535a0c0bd45c6bf17f81add297f40 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:48.421252 2991 raft_consensus.cc:3060] T fce535a0c0bd45c6bf17f81add297f40 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.410102 1167 consensus_queue.cc:260] T 4f7c15529be54a11961f385a953b0b03 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.423282 1167 ts_tablet_manager.cc:1434] T 4f7c15529be54a11961f385a953b0b03 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.015s user 0.003s sys 0.000s
I20251025 07:52:48.423659 1167 tablet_bootstrap.cc:492] T 22fc90c704f64d2f9f4b8c54dee45576 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:48.425799 2991 raft_consensus.cc:515] T fce535a0c0bd45c6bf17f81add297f40 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.425899 1167 tablet_bootstrap.cc:654] T 22fc90c704f64d2f9f4b8c54dee45576 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.426879 2991 leader_election.cc:290] T fce535a0c0bd45c6bf17f81add297f40 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:48.426985 2989 consensus_queue.cc:1048] T 9f039d9107ae493486d799ab6a1d731d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:48.421034 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95efd553493e4966bc48a01cac6e1e10" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:48.427412 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fce535a0c0bd45c6bf17f81add297f40" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:48.427548 768 raft_consensus.cc:3060] T fce535a0c0bd45c6bf17f81add297f40 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.427649 2888 raft_consensus.cc:493] T e4da89f863cc43f3bf6df908bb9dc6bf P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 07:52:48.427716 712 leader_election.cc:343] T 95efd553493e4966bc48a01cac6e1e10 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.427820 712 leader_election.cc:304] T 95efd553493e4966bc48a01cac6e1e10 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:48.427802 2888 raft_consensus.cc:515] T e4da89f863cc43f3bf6df908bb9dc6bf P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.428227 2981 raft_consensus.cc:2749] T 95efd553493e4966bc48a01cac6e1e10 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:48.428489 2888 leader_election.cc:290] T e4da89f863cc43f3bf6df908bb9dc6bf P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:48.429181 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4da89f863cc43f3bf6df908bb9dc6bf" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:48.429502 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4da89f863cc43f3bf6df908bb9dc6bf" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:48.429575 712 leader_election.cc:343] T e4da89f863cc43f3bf6df908bb9dc6bf P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:48.429986 710 leader_election.cc:343] T e4da89f863cc43f3bf6df908bb9dc6bf P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.430075 710 leader_election.cc:304] T e4da89f863cc43f3bf6df908bb9dc6bf P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:48.430444 2888 raft_consensus.cc:2749] T e4da89f863cc43f3bf6df908bb9dc6bf P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:48.430871 768 raft_consensus.cc:2468] T fce535a0c0bd45c6bf17f81add297f40 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:48.426702 1043 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fce535a0c0bd45c6bf17f81add297f40" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:48.426498 2169 ts_tablet_manager.cc:1929] T f661d5909657455581dcb5e6eadd3c0b P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.431424 2169 log.cc:1199] T f661d5909657455581dcb5e6eadd3c0b P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/f661d5909657455581dcb5e6eadd3c0b
W20251025 07:52:48.431460 850 leader_election.cc:343] T fce535a0c0bd45c6bf17f81add297f40 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.431743 2597 raft_consensus.cc:493] T 7e6544202d4f489d82f22d6073ef240d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:48.431828 2169 ts_tablet_manager.cc:1950] T f661d5909657455581dcb5e6eadd3c0b P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:48.431939 2597 raft_consensus.cc:515] T 7e6544202d4f489d82f22d6073ef240d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.432227 847 leader_election.cc:304] T fce535a0c0bd45c6bf17f81add297f40 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:48.432595 2989 raft_consensus.cc:2804] T fce535a0c0bd45c6bf17f81add297f40 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:48.432593 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 188) took 6971 ms. Trace:
I20251025 07:52:48.432713 2989 raft_consensus.cc:697] T fce535a0c0bd45c6bf17f81add297f40 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:48.432670 2169 rpcz_store.cc:276] 1025 07:52:41.461128 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.461169 (+ 41us) service_pool.cc:225] Handling call
1025 07:52:48.317872 (+6856703us) ts_tablet_manager.cc:1151] Deleting tablet f661d5909657455581dcb5e6eadd3c0b
1025 07:52:48.317881 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.356590 (+ 38709us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.422059 (+ 65469us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.432574 (+ 10515us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6856574}
I20251025 07:52:48.432945 2989 consensus_queue.cc:237] T fce535a0c0bd45c6bf17f81add297f40 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.433099 2169 tablet_replica.cc:333] T 88eb51447e0f4080baa49dd902deb24f P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:48.433308 2169 raft_consensus.cc:2243] T 88eb51447e0f4080baa49dd902deb24f P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
W20251025 07:52:48.433454 983 leader_election.cc:343] T 7e6544202d4f489d82f22d6073ef240d P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 7e6544202d4f489d82f22d6073ef240d
I20251025 07:52:48.433485 2169 raft_consensus.cc:2272] T 88eb51447e0f4080baa49dd902deb24f P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.433835 2597 leader_election.cc:290] T 7e6544202d4f489d82f22d6073ef240d P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:48.391439 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet aaef24ccdfba45fe8204327723df834e (table default.Table_bd5089a7892041649dd05b7fb0961255 [id=840b3ffcb44144548843457c48f46380]) successfully deleted
I20251025 07:52:48.435086 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84]). Expected version 1 got 0
I20251025 07:52:48.433096 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet f661d5909657455581dcb5e6eadd3c0b (table default.Table_b8a12e9a7a974006904fcef12bb8225a [id=9bfd1a38f56e4b70850139909aff091f]) successfully deleted
W20251025 07:52:48.430941 439 consensus_peers.cc:597] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 -> Peer 8603bddd8a084ff7a73a25641768dfa9 (127.0.104.61:42203): Couldn't send request to peer 8603bddd8a084ff7a73a25641768dfa9. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.61:42203: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:48.442049 2146 ts_tablet_manager.cc:1929] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.442296 2146 log.cc:1199] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/d9bfd7c0c3ad4e5b98b6708793d61d41
I20251025 07:52:48.434123 2169 ts_tablet_manager.cc:1916] T 88eb51447e0f4080baa49dd902deb24f P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.443650 1166 tablet_bootstrap.cc:492] T 5dab439bc50d4e869548154e1d222ad6 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:48.445362 1166 ts_tablet_manager.cc:1403] T 5dab439bc50d4e869548154e1d222ad6 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.081s user 0.008s sys 0.003s
I20251025 07:52:48.445534 2146 ts_tablet_manager.cc:1950] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:48.446254 1166 raft_consensus.cc:359] T 5dab439bc50d4e869548154e1d222ad6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.446501 1166 raft_consensus.cc:385] T 5dab439bc50d4e869548154e1d222ad6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.446604 1166 raft_consensus.cc:740] T 5dab439bc50d4e869548154e1d222ad6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.446858 1166 consensus_queue.cc:260] T 5dab439bc50d4e869548154e1d222ad6 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.441526 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3]). Expected version 2 got 0
I20251025 07:52:48.447432 1166 ts_tablet_manager.cc:1434] T 5dab439bc50d4e869548154e1d222ad6 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:48.447678 1166 tablet_bootstrap.cc:492] T 437da5ded9b648abb668a8cb4896452c P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:48.447839 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 178) took 7026 ms. Trace:
I20251025 07:52:48.447917 2146 rpcz_store.cc:276] 1025 07:52:41.421356 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.422442 (+ 1086us) service_pool.cc:225] Handling call
1025 07:52:48.391217 (+6968775us) ts_tablet_manager.cc:1151] Deleting tablet d9bfd7c0c3ad4e5b98b6708793d61d41
1025 07:52:48.391230 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.430334 (+ 39104us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.439812 (+ 9478us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.447821 (+ 8009us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1537,"tablet-delete.queue_time_us":6968631}
I20251025 07:52:48.448225 2146 tablet_replica.cc:333] T f656945454fe4921ab819c772a089eb3 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:48.448397 2146 raft_consensus.cc:2243] T f656945454fe4921ab819c772a089eb3 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:48.448462 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet d9bfd7c0c3ad4e5b98b6708793d61d41 (table default.Table_0c74279815ab4a5186497b68bb932523 [id=5202bb531f384527a167bf9efe00d9c6]) successfully deleted
I20251025 07:52:48.448705 2146 raft_consensus.cc:2272] T f656945454fe4921ab819c772a089eb3 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.449064 1166 tablet_bootstrap.cc:654] T 437da5ded9b648abb668a8cb4896452c P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.449215 2146 ts_tablet_manager.cc:1916] T f656945454fe4921ab819c772a089eb3 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.449564 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 36262ab8c6964dd684c01b4db6219a4f (table default.Table_76158f8cf82e4cdb9a90580e46aaea48 [id=f249f30497bc450c887d659721469de8]). Expected version 1 got 0
I20251025 07:52:48.451442 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 292fe04055954afd8c08ac2636b68191 (table default.Table_9fa69b10b5e842639fc0b0a58cd9b6a5 [id=a253ce0253bd4da4814741c482193d8f]). Expected version 1 got 0
W20251025 07:52:48.452077 982 leader_election.cc:343] T 7e6544202d4f489d82f22d6073ef240d P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: 7e6544202d4f489d82f22d6073ef240d
I20251025 07:52:48.452132 450 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 288e543bb279410c91c01dd1623e110f (table default.Table_6e242ba55b194c23a6701f4b064fdbe5 [id=4c34b64b299f4162a53866cf01c5298f]). Expected version 1 got 0
I20251025 07:52:48.452280 982 leader_election.cc:304] T 7e6544202d4f489d82f22d6073ef240d P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:48.452564 2597 raft_consensus.cc:2749] T 7e6544202d4f489d82f22d6073ef240d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:48.455618 2146 ts_tablet_manager.cc:1929] T f656945454fe4921ab819c772a089eb3 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.455776 2146 log.cc:1199] T f656945454fe4921ab819c772a089eb3 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/f656945454fe4921ab819c772a089eb3
I20251025 07:52:48.456225 2146 ts_tablet_manager.cc:1950] T f656945454fe4921ab819c772a089eb3 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:48.456952 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 187) took 7033 ms. Trace:
I20251025 07:52:48.457098 2146 rpcz_store.cc:276] 1025 07:52:41.423144 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.423423 (+ 279us) service_pool.cc:225] Handling call
1025 07:52:48.448180 (+7024757us) ts_tablet_manager.cc:1151] Deleting tablet f656945454fe4921ab819c772a089eb3
1025 07:52:48.448189 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.452561 (+ 4372us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.455540 (+ 2979us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.456935 (+ 1395us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7024605}
I20251025 07:52:48.457635 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet f656945454fe4921ab819c772a089eb3 (table default.Table_7ce7627f71134d5c989f9ba5c6766a2a [id=b5bf17121a4e4e3a8f21dccf1dba4a99]) successfully deleted
I20251025 07:52:48.460693 2989 raft_consensus.cc:493] T dff7f86ae8924685b26011e504e2247b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:48.460860 2989 raft_consensus.cc:515] T dff7f86ae8924685b26011e504e2247b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.461875 2989 leader_election.cc:290] T dff7f86ae8924685b26011e504e2247b P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:48.461856 1043 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dff7f86ae8924685b26011e504e2247b" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:48.462654 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dff7f86ae8924685b26011e504e2247b" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:48.462847 768 raft_consensus.cc:2468] T dff7f86ae8924685b26011e504e2247b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
W20251025 07:52:48.462342 850 leader_election.cc:343] T dff7f86ae8924685b26011e504e2247b P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.467344 2169 ts_tablet_manager.cc:1929] T 88eb51447e0f4080baa49dd902deb24f P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.470979 2169 log.cc:1199] T 88eb51447e0f4080baa49dd902deb24f P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/88eb51447e0f4080baa49dd902deb24f
I20251025 07:52:48.472644 2169 ts_tablet_manager.cc:1950] T 88eb51447e0f4080baa49dd902deb24f P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:48.472896 2146 tablet_replica.cc:333] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:48.473165 2146 raft_consensus.cc:2243] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:48.473539 2146 raft_consensus.cc:2272] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.474515 2146 ts_tablet_manager.cc:1916] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.475205 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 189) took 7013 ms. Trace:
I20251025 07:52:48.475435 847 leader_election.cc:304] T dff7f86ae8924685b26011e504e2247b P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:48.475775 2169 rpcz_store.cc:276] 1025 07:52:41.461450 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.461488 (+ 38us) service_pool.cc:225] Handling call
1025 07:52:48.433039 (+6971551us) ts_tablet_manager.cc:1151] Deleting tablet 88eb51447e0f4080baa49dd902deb24f
1025 07:52:48.433050 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.452117 (+ 19067us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.467225 (+ 15108us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.475185 (+ 7960us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6971427}
I20251025 07:52:48.476732 2990 raft_consensus.cc:2804] T dff7f86ae8924685b26011e504e2247b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:48.477104 2990 raft_consensus.cc:493] T dff7f86ae8924685b26011e504e2247b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:48.477164 2990 raft_consensus.cc:3060] T dff7f86ae8924685b26011e504e2247b P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.479880 2990 raft_consensus.cc:515] T dff7f86ae8924685b26011e504e2247b P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.480866 2990 leader_election.cc:290] T dff7f86ae8924685b26011e504e2247b P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:48.482964 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dff7f86ae8924685b26011e504e2247b" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:48.483167 768 raft_consensus.cc:3060] T dff7f86ae8924685b26011e504e2247b P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.486511 768 raft_consensus.cc:2468] T dff7f86ae8924685b26011e504e2247b P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:48.488591 847 leader_election.cc:304] T dff7f86ae8924685b26011e504e2247b P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:48.490522 2991 raft_consensus.cc:2804] T dff7f86ae8924685b26011e504e2247b P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:48.490661 2991 raft_consensus.cc:697] T dff7f86ae8924685b26011e504e2247b P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:48.490923 2991 consensus_queue.cc:237] T dff7f86ae8924685b26011e504e2247b P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
W20251025 07:52:48.493160 850 consensus_peers.cc:597] T 9f039d9107ae493486d799ab6a1d731d P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:48.495911 1167 tablet_bootstrap.cc:492] T 22fc90c704f64d2f9f4b8c54dee45576 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:48.496172 1167 ts_tablet_manager.cc:1403] T 22fc90c704f64d2f9f4b8c54dee45576 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.073s user 0.003s sys 0.002s
I20251025 07:52:48.497040 1167 raft_consensus.cc:359] T 22fc90c704f64d2f9f4b8c54dee45576 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:48.497316 1167 raft_consensus.cc:385] T 22fc90c704f64d2f9f4b8c54dee45576 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.497434 1167 raft_consensus.cc:740] T 22fc90c704f64d2f9f4b8c54dee45576 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.497756 1167 consensus_queue.cc:260] T 22fc90c704f64d2f9f4b8c54dee45576 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:48.498940 1167 ts_tablet_manager.cc:1434] T 22fc90c704f64d2f9f4b8c54dee45576 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251025 07:52:48.499621 1167 tablet_bootstrap.cc:492] T 48b73e0c0dce4be686d05f991a21bd19 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:48.503275 1167 tablet_bootstrap.cc:654] T 48b73e0c0dce4be686d05f991a21bd19 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.480857 1043 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dff7f86ae8924685b26011e504e2247b" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:48.505461 850 leader_election.cc:343] T dff7f86ae8924685b26011e504e2247b P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.506016 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 88eb51447e0f4080baa49dd902deb24f (table default.Table_b210463b66db4877b9a791f7cd741492 [id=438936db6ab24eb09a48b31ca013031b]) successfully deleted
I20251025 07:52:48.509673 2146 ts_tablet_manager.cc:1929] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.509786 2146 log.cc:1199] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/48ff4ce7846b4e56a37eee9bcb17bd5e
I20251025 07:52:48.510179 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 190) took 7043 ms. Trace:
I20251025 07:52:48.510603 1043 raft_consensus.cc:3060] T a0ea1edeca7347fc844faa5e4be1ae94 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.510319 2169 rpcz_store.cc:276] 1025 07:52:41.461739 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.461777 (+ 38us) service_pool.cc:225] Handling call
1025 07:52:48.505424 (+7043647us) ts_tablet_manager.cc:1151] Deleting tablet 2f26667f053a4f16b5da330c0b5e40f9
1025 07:52:48.505438 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.505549 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7043497}
W20251025 07:52:48.514658 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 2f26667f053a4f16b5da330c0b5e40f9 because the tablet was not found. No further retry: Not found: Tablet not found: 2f26667f053a4f16b5da330c0b5e40f9
I20251025 07:52:48.510124 2146 ts_tablet_manager.cc:1950] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:48.514699 2169 tablet_replica.cc:333] T f97acc9a638b4d5a968dcc3412a384bc P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:48.515477 2169 raft_consensus.cc:2243] T f97acc9a638b4d5a968dcc3412a384bc P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:48.515708 2169 raft_consensus.cc:2272] T f97acc9a638b4d5a968dcc3412a384bc P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.515976 2686 consensus_queue.cc:1048] T a0ea1edeca7347fc844faa5e4be1ae94 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:48.516927 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 189) took 7093 ms. Trace:
I20251025 07:52:48.517005 2146 rpcz_store.cc:276] 1025 07:52:41.423384 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.423612 (+ 228us) service_pool.cc:225] Handling call
1025 07:52:48.472824 (+7049212us) ts_tablet_manager.cc:1151] Deleting tablet 48ff4ce7846b4e56a37eee9bcb17bd5e
1025 07:52:48.472839 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.506484 (+ 33645us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.509573 (+ 3089us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.516907 (+ 7334us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7049060}
I20251025 07:52:48.517308 2146 tablet_replica.cc:333] T e54fe7cbcffa40c4b04144b47bee2b48 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:48.517460 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 48ff4ce7846b4e56a37eee9bcb17bd5e (table default.Table_5eaf42285e474860a44c19683f1fa70f [id=0da8afa0245842cfa80b852852e5235a]) successfully deleted
I20251025 07:52:48.517505 2146 raft_consensus.cc:2243] T e54fe7cbcffa40c4b04144b47bee2b48 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:48.517807 2146 raft_consensus.cc:2272] T e54fe7cbcffa40c4b04144b47bee2b48 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.518437 2146 ts_tablet_manager.cc:1916] T e54fe7cbcffa40c4b04144b47bee2b48 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.519186 1169 tablet_bootstrap.cc:492] T 716fb61312e94a4d9eb8ab8336ca140c P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:48.519353 1169 ts_tablet_manager.cc:1403] T 716fb61312e94a4d9eb8ab8336ca140c P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.170s user 0.020s sys 0.026s
I20251025 07:52:48.520291 1169 raft_consensus.cc:359] T 716fb61312e94a4d9eb8ab8336ca140c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.520515 1169 raft_consensus.cc:385] T 716fb61312e94a4d9eb8ab8336ca140c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.520855 1169 raft_consensus.cc:740] T 716fb61312e94a4d9eb8ab8336ca140c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.521957 2169 ts_tablet_manager.cc:1916] T f97acc9a638b4d5a968dcc3412a384bc P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.526271 1166 tablet_bootstrap.cc:492] T 437da5ded9b648abb668a8cb4896452c P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:48.526448 1166 ts_tablet_manager.cc:1403] T 437da5ded9b648abb668a8cb4896452c P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.079s user 0.002s sys 0.007s
I20251025 07:52:48.526623 2597 raft_consensus.cc:493] T 9a8e2658a81e436298fe2190726080d4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:48.526765 2597 raft_consensus.cc:515] T 9a8e2658a81e436298fe2190726080d4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.527446 2597 leader_election.cc:290] T 9a8e2658a81e436298fe2190726080d4 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:48.528456 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a8e2658a81e436298fe2190726080d4" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:48.528748 1166 raft_consensus.cc:359] T 437da5ded9b648abb668a8cb4896452c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.528909 1166 raft_consensus.cc:385] T 437da5ded9b648abb668a8cb4896452c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.528965 1166 raft_consensus.cc:740] T 437da5ded9b648abb668a8cb4896452c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.529177 1166 consensus_queue.cc:260] T 437da5ded9b648abb668a8cb4896452c P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.529994 1169 consensus_queue.cc:260] T 716fb61312e94a4d9eb8ab8336ca140c P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.530629 1166 ts_tablet_manager.cc:1434] T 437da5ded9b648abb668a8cb4896452c P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251025 07:52:48.531901 1166 tablet_bootstrap.cc:492] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:48.533701 2146 ts_tablet_manager.cc:1929] T e54fe7cbcffa40c4b04144b47bee2b48 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.533807 2146 log.cc:1199] T e54fe7cbcffa40c4b04144b47bee2b48 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/e54fe7cbcffa40c4b04144b47bee2b48
I20251025 07:52:48.534183 2146 ts_tablet_manager.cc:1950] T e54fe7cbcffa40c4b04144b47bee2b48 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:48.534171 1169 ts_tablet_manager.cc:1434] T 716fb61312e94a4d9eb8ab8336ca140c P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20251025 07:52:48.535804 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 188) took 7112 ms. Trace:
I20251025 07:52:48.535897 2146 rpcz_store.cc:276] 1025 07:52:41.423253 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.423686 (+ 433us) service_pool.cc:225] Handling call
1025 07:52:48.517257 (+7093571us) ts_tablet_manager.cc:1151] Deleting tablet e54fe7cbcffa40c4b04144b47bee2b48
1025 07:52:48.517268 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.522429 (+ 5161us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.533588 (+ 11159us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.535787 (+ 2199us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":796,"tablet-delete.queue_time_us":7093399}
I20251025 07:52:48.536273 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 190) took 7112 ms. Trace:
I20251025 07:52:48.536341 2146 rpcz_store.cc:276] 1025 07:52:41.423921 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.423960 (+ 39us) service_pool.cc:225] Handling call
1025 07:52:48.536161 (+7112201us) ts_tablet_manager.cc:1151] Deleting tablet b947ecc352e54f4caa73c6ca97372256
1025 07:52:48.536170 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.536260 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7112018}
I20251025 07:52:48.536394 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet e54fe7cbcffa40c4b04144b47bee2b48 (table default.Table_590cc281d8b042b19ef12c7f6436752a [id=b1eee5c0243a48c4bbdecc0b4f38ab2d]) successfully deleted
W20251025 07:52:48.537005 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet b947ecc352e54f4caa73c6ca97372256 because the tablet was not found. No further retry: Not found: Tablet not found: b947ecc352e54f4caa73c6ca97372256
I20251025 07:52:48.529704 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a8e2658a81e436298fe2190726080d4" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:48.538997 2169 ts_tablet_manager.cc:1929] T f97acc9a638b4d5a968dcc3412a384bc P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.539175 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 185) took 7116 ms. Trace:
I20251025 07:52:48.539286 2146 rpcz_store.cc:276] 1025 07:52:41.422937 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.424072 (+ 1135us) service_pool.cc:225] Handling call
1025 07:52:48.536538 (+7112466us) ts_tablet_manager.cc:1151] Deleting tablet 2f26667f053a4f16b5da330c0b5e40f9
1025 07:52:48.536547 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.539159 (+ 2612us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7112317}
I20251025 07:52:48.539639 2146 tablet_replica.cc:333] T 26c849d376d143c6bef0c8e0e1d7d03a P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:48.539565 2886 consensus_queue.cc:1048] T a0ea1edeca7347fc844faa5e4be1ae94 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:48.539921 2146 raft_consensus.cc:2243] T 26c849d376d143c6bef0c8e0e1d7d03a P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
W20251025 07:52:48.540102 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 2f26667f053a4f16b5da330c0b5e40f9 because the tablet was not found. No further retry: Not found: Tablet not found: 2f26667f053a4f16b5da330c0b5e40f9
I20251025 07:52:48.543218 2146 raft_consensus.cc:2272] T 26c849d376d143c6bef0c8e0e1d7d03a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.543825 2146 ts_tablet_manager.cc:1916] T 26c849d376d143c6bef0c8e0e1d7d03a P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.539273 982 leader_election.cc:304] T 9a8e2658a81e436298fe2190726080d4 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:48.546691 2169 log.cc:1199] T f97acc9a638b4d5a968dcc3412a384bc P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/f97acc9a638b4d5a968dcc3412a384bc
I20251025 07:52:48.547115 2169 ts_tablet_manager.cc:1950] T f97acc9a638b4d5a968dcc3412a384bc P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:48.547876 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 191) took 7085 ms. Trace:
I20251025 07:52:48.546023 1169 tablet_bootstrap.cc:492] T 6d3c9954693d4e8782ee0128ef066aa9 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:48.548020 2169 rpcz_store.cc:276] 1025 07:52:41.462074 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.462115 (+ 41us) service_pool.cc:225] Handling call
1025 07:52:48.514647 (+7052532us) ts_tablet_manager.cc:1151] Deleting tablet f97acc9a638b4d5a968dcc3412a384bc
1025 07:52:48.514658 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.528713 (+ 14055us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.538897 (+ 10184us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.547857 (+ 8960us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7052311}
I20251025 07:52:48.548529 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 192) took 7086 ms. Trace:
I20251025 07:52:48.548646 2169 rpcz_store.cc:276] 1025 07:52:41.462481 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.462520 (+ 39us) service_pool.cc:225] Handling call
1025 07:52:48.548435 (+7085915us) ts_tablet_manager.cc:1151] Deleting tablet 410ef150a7014c6c867a021b544a2f4e
1025 07:52:48.548444 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.548516 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7085784}
I20251025 07:52:48.548983 2169 tablet_replica.cc:333] T 26c849d376d143c6bef0c8e0e1d7d03a P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:48.549042 2597 raft_consensus.cc:2749] T 9a8e2658a81e436298fe2190726080d4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:48.549204 2169 raft_consensus.cc:2243] T 26c849d376d143c6bef0c8e0e1d7d03a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:48.549358 2169 raft_consensus.cc:2272] T 26c849d376d143c6bef0c8e0e1d7d03a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.549819 1169 tablet_bootstrap.cc:654] T 6d3c9954693d4e8782ee0128ef066aa9 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.540050 1166 tablet_bootstrap.cc:654] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.554008 2169 ts_tablet_manager.cc:1916] T 26c849d376d143c6bef0c8e0e1d7d03a P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.556613 2146 ts_tablet_manager.cc:1929] T 26c849d376d143c6bef0c8e0e1d7d03a P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.556723 2146 log.cc:1199] T 26c849d376d143c6bef0c8e0e1d7d03a P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/26c849d376d143c6bef0c8e0e1d7d03a
I20251025 07:52:48.557093 2146 ts_tablet_manager.cc:1950] T 26c849d376d143c6bef0c8e0e1d7d03a P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:48.557829 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 186) took 7134 ms. Trace:
I20251025 07:52:48.557920 2146 rpcz_store.cc:276] 1025 07:52:41.423055 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.423971 (+ 916us) service_pool.cc:225] Handling call
1025 07:52:48.539594 (+7115623us) ts_tablet_manager.cc:1151] Deleting tablet 26c849d376d143c6bef0c8e0e1d7d03a
1025 07:52:48.539602 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.552792 (+ 13190us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.556518 (+ 3726us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.557809 (+ 1291us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7115278}
I20251025 07:52:48.558303 2146 tablet_replica.cc:333] T 88eb51447e0f4080baa49dd902deb24f P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:48.558493 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 26c849d376d143c6bef0c8e0e1d7d03a (table default.Table_f8c38604344c429295d6d0ab9cb85b13 [id=c2f7a6281b6f47c990b096412f2201a9]) successfully deleted
I20251025 07:52:48.560401 2146 raft_consensus.cc:2243] T 88eb51447e0f4080baa49dd902deb24f P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:48.560720 2146 raft_consensus.cc:2272] T 88eb51447e0f4080baa49dd902deb24f P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.561249 2146 ts_tablet_manager.cc:1916] T 88eb51447e0f4080baa49dd902deb24f P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.570417 1166 tablet_bootstrap.cc:492] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:48.570559 1166 ts_tablet_manager.cc:1403] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.002s
I20251025 07:52:48.571166 1166 raft_consensus.cc:359] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.571331 1166 raft_consensus.cc:385] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.571382 1166 raft_consensus.cc:740] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.571578 1166 consensus_queue.cc:260] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.572012 1166 ts_tablet_manager.cc:1434] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251025 07:52:48.572283 1166 tablet_bootstrap.cc:492] T 95efd553493e4966bc48a01cac6e1e10 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:48.580066 2146 ts_tablet_manager.cc:1929] T 88eb51447e0f4080baa49dd902deb24f P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.580183 2146 log.cc:1199] T 88eb51447e0f4080baa49dd902deb24f P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/88eb51447e0f4080baa49dd902deb24f
I20251025 07:52:48.580525 2146 ts_tablet_manager.cc:1950] T 88eb51447e0f4080baa49dd902deb24f P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:48.581248 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 184) took 7158 ms. Trace:
I20251025 07:52:48.581327 2146 rpcz_store.cc:276] 1025 07:52:41.422829 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.424389 (+ 1560us) service_pool.cc:225] Handling call
1025 07:52:48.558238 (+7133849us) ts_tablet_manager.cc:1151] Deleting tablet 88eb51447e0f4080baa49dd902deb24f
1025 07:52:48.558250 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.564475 (+ 6225us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.579963 (+ 15488us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.581219 (+ 1256us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1884,"tablet-delete.queue_time_us":7133675}
I20251025 07:52:48.581665 2146 tablet_replica.cc:333] T f661d5909657455581dcb5e6eadd3c0b P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:48.581847 2146 raft_consensus.cc:2243] T f661d5909657455581dcb5e6eadd3c0b P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:48.581904 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 88eb51447e0f4080baa49dd902deb24f (table default.Table_b210463b66db4877b9a791f7cd741492 [id=438936db6ab24eb09a48b31ca013031b]) successfully deleted
I20251025 07:52:48.583950 1166 tablet_bootstrap.cc:654] T 95efd553493e4966bc48a01cac6e1e10 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.589972 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5]). Expected version 1 got 0
I20251025 07:52:48.591493 2169 ts_tablet_manager.cc:1929] T 26c849d376d143c6bef0c8e0e1d7d03a P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.591643 2169 log.cc:1199] T 26c849d376d143c6bef0c8e0e1d7d03a P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/26c849d376d143c6bef0c8e0e1d7d03a
I20251025 07:52:48.591989 2169 ts_tablet_manager.cc:1950] T 26c849d376d143c6bef0c8e0e1d7d03a P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:48.592545 2146 raft_consensus.cc:2272] T f661d5909657455581dcb5e6eadd3c0b P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.592859 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 193) took 7130 ms. Trace:
I20251025 07:52:48.592958 2169 rpcz_store.cc:276] 1025 07:52:41.462781 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.462820 (+ 39us) service_pool.cc:225] Handling call
1025 07:52:48.548935 (+7086115us) ts_tablet_manager.cc:1151] Deleting tablet 26c849d376d143c6bef0c8e0e1d7d03a
1025 07:52:48.548943 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.577666 (+ 28723us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.591370 (+ 13704us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.592838 (+ 1468us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7085983}
I20251025 07:52:48.593361 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 194) took 7130 ms. Trace:
I20251025 07:52:48.593447 2169 rpcz_store.cc:276] 1025 07:52:41.463101 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.463140 (+ 39us) service_pool.cc:225] Handling call
1025 07:52:48.593222 (+7130082us) ts_tablet_manager.cc:1151] Deleting tablet 48ff4ce7846b4e56a37eee9bcb17bd5e
1025 07:52:48.593232 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.593347 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7129938}
I20251025 07:52:48.593605 2146 ts_tablet_manager.cc:1916] T f661d5909657455581dcb5e6eadd3c0b P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.593683 2169 tablet_replica.cc:333] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:48.593858 2169 raft_consensus.cc:2243] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:48.594161 2169 pending_rounds.cc:70] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b: Trying to abort 1 pending ops.
I20251025 07:52:48.594208 2169 pending_rounds.cc:77] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214607403026329600 op_type: NO_OP noop_request { }
I20251025 07:52:48.594300 2169 raft_consensus.cc:2889] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 07:52:48.594388 2169 raft_consensus.cc:2272] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.594846 2169 ts_tablet_manager.cc:1916] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.598896 1166 tablet_bootstrap.cc:492] T 95efd553493e4966bc48a01cac6e1e10 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:48.599018 1166 ts_tablet_manager.cc:1403] T 95efd553493e4966bc48a01cac6e1e10 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.000s
I20251025 07:52:48.599627 1166 raft_consensus.cc:359] T 95efd553493e4966bc48a01cac6e1e10 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.599798 1166 raft_consensus.cc:385] T 95efd553493e4966bc48a01cac6e1e10 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.599841 1166 raft_consensus.cc:740] T 95efd553493e4966bc48a01cac6e1e10 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.600015 1166 consensus_queue.cc:260] T 95efd553493e4966bc48a01cac6e1e10 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.600711 2146 ts_tablet_manager.cc:1929] T f661d5909657455581dcb5e6eadd3c0b P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.600751 1166 ts_tablet_manager.cc:1434] T 95efd553493e4966bc48a01cac6e1e10 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:48.600809 2146 log.cc:1199] T f661d5909657455581dcb5e6eadd3c0b P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/f661d5909657455581dcb5e6eadd3c0b
I20251025 07:52:48.601008 1166 tablet_bootstrap.cc:492] T d46673a283fe42b78ddeb99fbd2678eb P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:48.601137 2146 ts_tablet_manager.cc:1950] T f661d5909657455581dcb5e6eadd3c0b P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:48.601850 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 183) took 7179 ms. Trace:
I20251025 07:52:48.601938 2146 rpcz_store.cc:276] 1025 07:52:41.422723 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.424840 (+ 2117us) service_pool.cc:225] Handling call
1025 07:52:48.581622 (+7156782us) ts_tablet_manager.cc:1151] Deleting tablet f661d5909657455581dcb5e6eadd3c0b
1025 07:52:48.581631 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.597713 (+ 16082us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.600627 (+ 2914us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.601830 (+ 1203us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7156634}
I20251025 07:52:48.603892 2989 consensus_queue.cc:1048] T 1b5ac7e845a24b2eb5b2429d47361232 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:48.606707 1166 tablet_bootstrap.cc:654] T d46673a283fe42b78ddeb99fbd2678eb P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.608978 2886 raft_consensus.cc:493] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:48.609103 2886 raft_consensus.cc:515] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.609813 2886 leader_election.cc:290] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:48.608417 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet f97acc9a638b4d5a968dcc3412a384bc (table default.Table_e50bf47b78d94bbc811e6f3fb3551edb [id=0827603890c8466082bc6c306c9c66af]) successfully deleted
W20251025 07:52:48.611073 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 410ef150a7014c6c867a021b544a2f4e because the tablet was not found. No further retry: Not found: Tablet not found: 410ef150a7014c6c867a021b544a2f4e
I20251025 07:52:48.611270 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 26c849d376d143c6bef0c8e0e1d7d03a (table default.Table_f8c38604344c429295d6d0ab9cb85b13 [id=c2f7a6281b6f47c990b096412f2201a9]) successfully deleted
W20251025 07:52:48.611454 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 48ff4ce7846b4e56a37eee9bcb17bd5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48ff4ce7846b4e56a37eee9bcb17bd5e already in progress: creating tablet
I20251025 07:52:48.612182 771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc4a788dca9d4a8ba40f8fdf1a4b915e" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:48.612352 771 raft_consensus.cc:2468] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:48.612988 847 leader_election.cc:304] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:48.613335 2886 raft_consensus.cc:2804] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:48.613413 2886 raft_consensus.cc:493] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:48.613488 2886 raft_consensus.cc:3060] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.615187 1166 tablet_bootstrap.cc:492] T d46673a283fe42b78ddeb99fbd2678eb P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:48.615295 1166 ts_tablet_manager.cc:1403] T d46673a283fe42b78ddeb99fbd2678eb P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20251025 07:52:48.615924 1166 raft_consensus.cc:359] T d46673a283fe42b78ddeb99fbd2678eb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.616061 1166 raft_consensus.cc:385] T d46673a283fe42b78ddeb99fbd2678eb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.616106 1166 raft_consensus.cc:740] T d46673a283fe42b78ddeb99fbd2678eb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.616307 1166 consensus_queue.cc:260] T d46673a283fe42b78ddeb99fbd2678eb P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.616881 1166 ts_tablet_manager.cc:1434] T d46673a283fe42b78ddeb99fbd2678eb P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:48.617147 1166 tablet_bootstrap.cc:492] T e47c4f741f354a19b49777bd06e54764 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:48.617309 2886 raft_consensus.cc:515] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.618803 1166 tablet_bootstrap.cc:654] T e47c4f741f354a19b49777bd06e54764 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.619867 2886 leader_election.cc:290] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:48.623570 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 192) took 7196 ms. Trace:
I20251025 07:52:48.623651 2146 rpcz_store.cc:276] 1025 07:52:41.426681 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.426846 (+ 165us) service_pool.cc:225] Handling call
1025 07:52:48.623477 (+7196631us) ts_tablet_manager.cc:1151] Deleting tablet b94634ac5077439fadfbbce569f60e8b
1025 07:52:48.623484 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.623555 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7196454}
I20251025 07:52:48.623908 2146 tablet_replica.cc:333] T c11f674385184ae6b235f1db8bf1fb91 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:48.624099 2146 raft_consensus.cc:2243] T c11f674385184ae6b235f1db8bf1fb91 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:48.624071 771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc4a788dca9d4a8ba40f8fdf1a4b915e" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:48.624234 771 raft_consensus.cc:3060] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.624290 2146 pending_rounds.cc:70] T c11f674385184ae6b235f1db8bf1fb91 P 3aea2869ff814cc7b142012faa274382: Trying to abort 1 pending ops.
I20251025 07:52:48.624322 2146 pending_rounds.cc:77] T c11f674385184ae6b235f1db8bf1fb91 P 3aea2869ff814cc7b142012faa274382: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214607434318782464 op_type: NO_OP noop_request { }
I20251025 07:52:48.624404 2146 raft_consensus.cc:2889] T c11f674385184ae6b235f1db8bf1fb91 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 07:52:48.624466 2146 raft_consensus.cc:2272] T c11f674385184ae6b235f1db8bf1fb91 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.625324 1167 tablet_bootstrap.cc:492] T 48b73e0c0dce4be686d05f991a21bd19 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:48.625452 1167 ts_tablet_manager.cc:1403] T 48b73e0c0dce4be686d05f991a21bd19 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.126s user 0.013s sys 0.004s
I20251025 07:52:48.626436 2146 ts_tablet_manager.cc:1916] T c11f674385184ae6b235f1db8bf1fb91 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.626684 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet f661d5909657455581dcb5e6eadd3c0b (table default.Table_b8a12e9a7a974006904fcef12bb8225a [id=9bfd1a38f56e4b70850139909aff091f]) successfully deleted
W20251025 07:52:48.626960 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet b94634ac5077439fadfbbce569f60e8b because the tablet was not found. No further retry: Not found: Tablet not found: b94634ac5077439fadfbbce569f60e8b
I20251025 07:52:48.628435 1166 tablet_bootstrap.cc:492] T e47c4f741f354a19b49777bd06e54764 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:48.628551 1166 ts_tablet_manager.cc:1403] T e47c4f741f354a19b49777bd06e54764 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20251025 07:52:48.629184 1166 raft_consensus.cc:359] T e47c4f741f354a19b49777bd06e54764 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:48.629324 1166 raft_consensus.cc:385] T e47c4f741f354a19b49777bd06e54764 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.629367 1166 raft_consensus.cc:740] T e47c4f741f354a19b49777bd06e54764 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.630110 2886 raft_consensus.cc:493] T 437da5ded9b648abb668a8cb4896452c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:48.630244 2886 raft_consensus.cc:515] T 437da5ded9b648abb668a8cb4896452c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.630965 2886 leader_election.cc:290] T 437da5ded9b648abb668a8cb4896452c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:48.631482 1167 raft_consensus.cc:359] T 48b73e0c0dce4be686d05f991a21bd19 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:48.631656 1167 raft_consensus.cc:385] T 48b73e0c0dce4be686d05f991a21bd19 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.631706 1167 raft_consensus.cc:740] T 48b73e0c0dce4be686d05f991a21bd19 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.631753 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "437da5ded9b648abb668a8cb4896452c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:48.631902 772 raft_consensus.cc:2468] T 437da5ded9b648abb668a8cb4896452c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:48.631922 1167 consensus_queue.cc:260] T 48b73e0c0dce4be686d05f991a21bd19 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:48.632738 1167 ts_tablet_manager.cc:1434] T 48b73e0c0dce4be686d05f991a21bd19 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251025 07:52:48.633005 1167 tablet_bootstrap.cc:492] T 3e79d69d5d004cc395b8e35569e25440 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:48.633009 771 raft_consensus.cc:2468] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
W20251025 07:52:48.634004 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet d37758f606344127b3db74c87babe69c because the tablet was not found. No further retry: Not found: Tablet not found: d37758f606344127b3db74c87babe69c
I20251025 07:52:48.634433 2166 tablet_replica.cc:333] T 3b940fa5ae26427dafbcfd26372754b3 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:48.634670 2166 raft_consensus.cc:2243] T 3b940fa5ae26427dafbcfd26372754b3 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:48.634925 2166 raft_consensus.cc:2272] T 3b940fa5ae26427dafbcfd26372754b3 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.635031 1167 tablet_bootstrap.cc:654] T 3e79d69d5d004cc395b8e35569e25440 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.635543 2166 ts_tablet_manager.cc:1916] T 3b940fa5ae26427dafbcfd26372754b3 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.635802 1166 consensus_queue.cc:260] T e47c4f741f354a19b49777bd06e54764 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:48.636274 1166 ts_tablet_manager.cc:1434] T e47c4f741f354a19b49777bd06e54764 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20251025 07:52:48.636562 1166 tablet_bootstrap.cc:492] T 72618756f756484e895e885ef12545c8 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:48.637900 847 leader_election.cc:304] T 437da5ded9b648abb668a8cb4896452c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:48.638123 1166 tablet_bootstrap.cc:654] T 72618756f756484e895e885ef12545c8 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.638482 847 leader_election.cc:304] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:48.638784 2989 raft_consensus.cc:2804] T 437da5ded9b648abb668a8cb4896452c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:48.638857 2989 raft_consensus.cc:493] T 437da5ded9b648abb668a8cb4896452c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:48.638934 2989 raft_consensus.cc:3060] T 437da5ded9b648abb668a8cb4896452c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.641523 2989 raft_consensus.cc:515] T 437da5ded9b648abb668a8cb4896452c P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.642279 2991 raft_consensus.cc:2804] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:48.642403 2991 raft_consensus.cc:697] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:48.642679 2991 consensus_queue.cc:237] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.643597 2989 leader_election.cc:290] T 437da5ded9b648abb668a8cb4896452c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:48.644275 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "437da5ded9b648abb668a8cb4896452c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:48.644446 772 raft_consensus.cc:3060] T 437da5ded9b648abb668a8cb4896452c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.647010 772 raft_consensus.cc:2468] T 437da5ded9b648abb668a8cb4896452c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:48.648159 2146 ts_tablet_manager.cc:1929] T c11f674385184ae6b235f1db8bf1fb91 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.648298 2146 log.cc:1199] T c11f674385184ae6b235f1db8bf1fb91 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/c11f674385184ae6b235f1db8bf1fb91
I20251025 07:52:48.648437 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc4a788dca9d4a8ba40f8fdf1a4b915e" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:48.648679 2146 ts_tablet_manager.cc:1950] T c11f674385184ae6b235f1db8bf1fb91 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:48.648758 847 leader_election.cc:304] T 437da5ded9b648abb668a8cb4896452c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:48.649178 2988 raft_consensus.cc:2804] T 437da5ded9b648abb668a8cb4896452c P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:48.649327 2988 raft_consensus.cc:697] T 437da5ded9b648abb668a8cb4896452c P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:48.649498 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 193) took 7222 ms. Trace:
I20251025 07:52:48.649564 2988 consensus_queue.cc:237] T 437da5ded9b648abb668a8cb4896452c P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.649596 2146 rpcz_store.cc:276] 1025 07:52:41.426808 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.427060 (+ 252us) service_pool.cc:225] Handling call
1025 07:52:48.623867 (+7196807us) ts_tablet_manager.cc:1151] Deleting tablet c11f674385184ae6b235f1db8bf1fb91
1025 07:52:48.623874 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.639956 (+ 16082us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.648062 (+ 8106us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.649441 (+ 1379us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7196056}
I20251025 07:52:48.650063 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet c11f674385184ae6b235f1db8bf1fb91 (table default.Table_ce27d8ed3ede46579f2ed3ae81536572 [id=c4c41c3b236141eb8e03dbb171f6b56d]) successfully deleted
I20251025 07:52:48.651607 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc4a788dca9d4a8ba40f8fdf1a4b915e" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:48.652204 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "437da5ded9b648abb668a8cb4896452c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:48.653473 2146 tablet_replica.cc:333] T d77704b4fe6c4456a2ba40ffec0c60fc P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:48.654352 2146 raft_consensus.cc:2243] T d77704b4fe6c4456a2ba40ffec0c60fc P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:48.654865 2146 raft_consensus.cc:2272] T d77704b4fe6c4456a2ba40ffec0c60fc P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.654881 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "437da5ded9b648abb668a8cb4896452c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:48.656786 2146 ts_tablet_manager.cc:1916] T d77704b4fe6c4456a2ba40ffec0c60fc P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:48.656875 850 leader_election.cc:343] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.658239 2166 ts_tablet_manager.cc:1929] T 3b940fa5ae26427dafbcfd26372754b3 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:48.658351 2166 log.cc:1199] T 3b940fa5ae26427dafbcfd26372754b3 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/3b940fa5ae26427dafbcfd26372754b3
I20251025 07:52:48.658664 2166 ts_tablet_manager.cc:1950] T 3b940fa5ae26427dafbcfd26372754b3 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:48.659354 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 174) took 7169 ms. Trace:
I20251025 07:52:48.659739 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3]). Expected version 1 got 0
I20251025 07:52:48.659984 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_fa83d0528b4b481989bfc0ca35c6de44 [id=bf77b5ff6f1048dd996d948437f9ff5e]). Expected version 3 got 0
I20251025 07:52:48.660279 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 783487b60c144f719ed2eb3c67b5ccab (table default.Table_09095c0829ed4d1cb8b763fc695e7fa8 [id=e372ca4384644779a6b37f389f549dec]). Expected version 1 got 0
I20251025 07:52:48.660564 453 catalog_manager.cc:5649] T 2e2e6893a48449969b71ee1c08d1960e P db42be7ad3c741a0aa427f27e245993b reported cstate change: term changed from 0 to 1, leader changed from <none> to db42be7ad3c741a0aa427f27e245993b (127.0.104.3). New cstate: current_term: 1 leader_uuid: "db42be7ad3c741a0aa427f27e245993b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } }
W20251025 07:52:48.661846 850 leader_election.cc:343] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:48.662345 850 leader_election.cc:343] T 437da5ded9b648abb668a8cb4896452c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:48.662746 850 leader_election.cc:343] T 437da5ded9b648abb668a8cb4896452c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:48.663196 850 consensus_peers.cc:597] T 1b5ac7e845a24b2eb5b2429d47361232 P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:48.663506 2169 ts_tablet_manager.cc:1929] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.664178 1166 tablet_bootstrap.cc:492] T 72618756f756484e895e885ef12545c8 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:48.664357 1166 ts_tablet_manager.cc:1403] T 72618756f756484e895e885ef12545c8 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20251025 07:52:48.664541 2169 log.cc:1199] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/b947ecc352e54f4caa73c6ca97372256
I20251025 07:52:48.665019 2169 ts_tablet_manager.cc:1950] T b947ecc352e54f4caa73c6ca97372256 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:48.665066 1166 raft_consensus.cc:359] T 72618756f756484e895e885ef12545c8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.665257 1166 raft_consensus.cc:385] T 72618756f756484e895e885ef12545c8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.665334 1166 raft_consensus.cc:740] T 72618756f756484e895e885ef12545c8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.665568 1166 consensus_queue.cc:260] T 72618756f756484e895e885ef12545c8 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.667130 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f]). Expected version 1 got 0
I20251025 07:52:48.659446 2166 rpcz_store.cc:276] 1025 07:52:41.489929 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.490148 (+ 219us) service_pool.cc:225] Handling call
1025 07:52:48.634370 (+7144222us) ts_tablet_manager.cc:1151] Deleting tablet 3b940fa5ae26427dafbcfd26372754b3
1025 07:52:48.634390 (+ 20us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.639766 (+ 5376us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.658125 (+ 18359us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.659336 (+ 1211us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7144090}
I20251025 07:52:48.667765 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7]). Expected version 1 got 0
I20251025 07:52:48.668730 1166 ts_tablet_manager.cc:1434] T 72618756f756484e895e885ef12545c8 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251025 07:52:48.669039 1166 tablet_bootstrap.cc:492] T 2d24e3e7a9e4431db609331ad1fd57c9 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:48.671315 1166 tablet_bootstrap.cc:654] T 2d24e3e7a9e4431db609331ad1fd57c9 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.671492 2146 ts_tablet_manager.cc:1929] T d77704b4fe6c4456a2ba40ffec0c60fc P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:48.671633 2146 log.cc:1199] T d77704b4fe6c4456a2ba40ffec0c60fc P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/d77704b4fe6c4456a2ba40ffec0c60fc
I20251025 07:52:48.672006 2146 ts_tablet_manager.cc:1950] T d77704b4fe6c4456a2ba40ffec0c60fc P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:48.672549 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 3b940fa5ae26427dafbcfd26372754b3 (table default.Table_d57e7ea8e59641f781c564e5b01479bc [id=fbaf826daa7c448eb92c7f93826260c8]) successfully deleted
I20251025 07:52:48.673115 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 872cb7512d104827b1f1f186d33a4a44 (table default.Table_d5c65dce8b1b4214ba519d964012bfc0 [id=d4be274b36a845a99a4b96b8846248fd]). Expected version 1 got 0
I20251025 07:52:48.673417 2991 raft_consensus.cc:493] T e47c4f741f354a19b49777bd06e54764 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:48.673574 2991 raft_consensus.cc:515] T e47c4f741f354a19b49777bd06e54764 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:48.674157 2991 leader_election.cc:290] T e47c4f741f354a19b49777bd06e54764 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:48.674536 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 195) took 7246 ms. Trace:
I20251025 07:52:48.674657 2146 rpcz_store.cc:276] 1025 07:52:41.428049 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.428996 (+ 947us) service_pool.cc:225] Handling call
1025 07:52:48.653350 (+7224354us) ts_tablet_manager.cc:1151] Deleting tablet d77704b4fe6c4456a2ba40ffec0c60fc
1025 07:52:48.653364 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.664082 (+ 10718us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.671330 (+ 7248us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.674493 (+ 3163us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7224179}
I20251025 07:52:48.674811 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e47c4f741f354a19b49777bd06e54764" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:48.675002 772 raft_consensus.cc:2468] T e47c4f741f354a19b49777bd06e54764 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:48.675083 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet d77704b4fe6c4456a2ba40ffec0c60fc (table default.Table_e42a1f7287a34753b7e4daf36893d9c2 [id=ff115adf26ec4e499d44caa9b0366a49]) successfully deleted
I20251025 07:52:48.675420 847 leader_election.cc:304] T e47c4f741f354a19b49777bd06e54764 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:48.675755 2991 raft_consensus.cc:2804] T e47c4f741f354a19b49777bd06e54764 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:48.675848 2991 raft_consensus.cc:493] T e47c4f741f354a19b49777bd06e54764 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:48.675944 2991 raft_consensus.cc:3060] T e47c4f741f354a19b49777bd06e54764 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.676380 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 196) took 7247 ms. Trace:
I20251025 07:52:48.676494 2146 rpcz_store.cc:276] 1025 07:52:41.429363 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.429702 (+ 339us) service_pool.cc:225] Handling call
1025 07:52:48.676236 (+7246534us) ts_tablet_manager.cc:1151] Deleting tablet e4da89f863cc43f3bf6df908bb9dc6bf
1025 07:52:48.676245 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.676338 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7246366}
W20251025 07:52:48.676918 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet e4da89f863cc43f3bf6df908bb9dc6bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4da89f863cc43f3bf6df908bb9dc6bf already in progress: creating tablet
I20251025 07:52:48.677906 2146 tablet_replica.cc:333] T fd265cdbf6a8440393a2b5d8240d1872 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:48.678525 2146 raft_consensus.cc:2243] T fd265cdbf6a8440393a2b5d8240d1872 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:48.678776 2146 pending_rounds.cc:70] T fd265cdbf6a8440393a2b5d8240d1872 P 3aea2869ff814cc7b142012faa274382: Trying to abort 1 pending ops.
I20251025 07:52:48.678860 2146 pending_rounds.cc:77] T fd265cdbf6a8440393a2b5d8240d1872 P 3aea2869ff814cc7b142012faa274382: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214607435429658624 op_type: NO_OP noop_request { }
I20251025 07:52:48.678967 2146 raft_consensus.cc:2889] T fd265cdbf6a8440393a2b5d8240d1872 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 07:52:48.679113 2991 raft_consensus.cc:515] T e47c4f741f354a19b49777bd06e54764 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:48.680641 2991 leader_election.cc:290] T e47c4f741f354a19b49777bd06e54764 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:48.681190 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e47c4f741f354a19b49777bd06e54764" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:48.681365 772 raft_consensus.cc:3060] T e47c4f741f354a19b49777bd06e54764 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.679105 2146 raft_consensus.cc:2272] T fd265cdbf6a8440393a2b5d8240d1872 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.682276 1166 tablet_bootstrap.cc:492] T 2d24e3e7a9e4431db609331ad1fd57c9 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:48.682437 1166 ts_tablet_manager.cc:1403] T 2d24e3e7a9e4431db609331ad1fd57c9 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20251025 07:52:48.682475 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e47c4f741f354a19b49777bd06e54764" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:48.682942 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e47c4f741f354a19b49777bd06e54764" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:48.683341 850 leader_election.cc:343] T e47c4f741f354a19b49777bd06e54764 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.683277 1166 raft_consensus.cc:359] T 2d24e3e7a9e4431db609331ad1fd57c9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.683531 1166 raft_consensus.cc:385] T 2d24e3e7a9e4431db609331ad1fd57c9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.683652 1166 raft_consensus.cc:740] T 2d24e3e7a9e4431db609331ad1fd57c9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
W20251025 07:52:48.684033 850 leader_election.cc:343] T e47c4f741f354a19b49777bd06e54764 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.684298 2166 tablet_replica.cc:333] T f97acc9a638b4d5a968dcc3412a384bc P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:48.683951 1166 consensus_queue.cc:260] T 2d24e3e7a9e4431db609331ad1fd57c9 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.684505 2166 raft_consensus.cc:2243] T f97acc9a638b4d5a968dcc3412a384bc P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:48.684659 2166 raft_consensus.cc:2272] T f97acc9a638b4d5a968dcc3412a384bc P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.684669 2146 ts_tablet_manager.cc:1916] T fd265cdbf6a8440393a2b5d8240d1872 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.684784 1166 ts_tablet_manager.cc:1434] T 2d24e3e7a9e4431db609331ad1fd57c9 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:48.685078 1166 tablet_bootstrap.cc:492] T 1c40b2e06d894a06af9e58285892ea8d P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:48.685261 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 195) took 7221 ms. Trace:
I20251025 07:52:48.685360 2169 rpcz_store.cc:276] 1025 07:52:41.463441 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.463611 (+ 170us) service_pool.cc:225] Handling call
1025 07:52:48.593644 (+7130033us) ts_tablet_manager.cc:1151] Deleting tablet b947ecc352e54f4caa73c6ca97372256
1025 07:52:48.593652 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.597880 (+ 4228us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.654650 (+ 56770us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.685240 (+ 30590us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7129867}
I20251025 07:52:48.685513 772 raft_consensus.cc:2468] T e47c4f741f354a19b49777bd06e54764 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:48.685699 2169 tablet_replica.cc:333] T f656945454fe4921ab819c772a089eb3 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:48.685904 2169 raft_consensus.cc:2243] T f656945454fe4921ab819c772a089eb3 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:48.686069 2169 raft_consensus.cc:2272] T f656945454fe4921ab819c772a089eb3 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.686628 2169 ts_tablet_manager.cc:1916] T f656945454fe4921ab819c772a089eb3 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.686707 1166 tablet_bootstrap.cc:654] T 1c40b2e06d894a06af9e58285892ea8d P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.687527 847 leader_election.cc:304] T e47c4f741f354a19b49777bd06e54764 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:48.688033 2991 raft_consensus.cc:2804] T e47c4f741f354a19b49777bd06e54764 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:48.688367 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet b947ecc352e54f4caa73c6ca97372256 (table default.Table_503b86ee2ab545afa9829bebcc9ea11d [id=395c91146476423dad0455a12eb5a8af]) successfully deleted
I20251025 07:52:48.688673 2991 raft_consensus.cc:697] T e47c4f741f354a19b49777bd06e54764 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:48.688999 2991 consensus_queue.cc:237] T e47c4f741f354a19b49777bd06e54764 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:48.691053 2166 ts_tablet_manager.cc:1916] T f97acc9a638b4d5a968dcc3412a384bc P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.693292 1169 tablet_bootstrap.cc:492] T 6d3c9954693d4e8782ee0128ef066aa9 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:48.693562 1169 ts_tablet_manager.cc:1403] T 6d3c9954693d4e8782ee0128ef066aa9 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.148s user 0.005s sys 0.004s
I20251025 07:52:48.694286 1169 raft_consensus.cc:359] T 6d3c9954693d4e8782ee0128ef066aa9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.694499 1169 raft_consensus.cc:385] T 6d3c9954693d4e8782ee0128ef066aa9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.694586 1169 raft_consensus.cc:740] T 6d3c9954693d4e8782ee0128ef066aa9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.694876 1169 consensus_queue.cc:260] T 6d3c9954693d4e8782ee0128ef066aa9 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.695094 2146 ts_tablet_manager.cc:1929] T fd265cdbf6a8440393a2b5d8240d1872 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.695253 2146 log.cc:1199] T fd265cdbf6a8440393a2b5d8240d1872 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/fd265cdbf6a8440393a2b5d8240d1872
I20251025 07:52:48.695742 2146 ts_tablet_manager.cc:1950] T fd265cdbf6a8440393a2b5d8240d1872 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:48.696707 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 194) took 7268 ms. Trace:
I20251025 07:52:48.696822 2146 rpcz_store.cc:276] 1025 07:52:41.427944 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.430252 (+ 2308us) service_pool.cc:225] Handling call
1025 07:52:48.677842 (+7247590us) ts_tablet_manager.cc:1151] Deleting tablet fd265cdbf6a8440393a2b5d8240d1872
1025 07:52:48.677849 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.687950 (+ 10101us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.694916 (+ 6966us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.696665 (+ 1749us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7247418}
I20251025 07:52:48.697182 2146 tablet_replica.cc:333] T 4f7c15529be54a11961f385a953b0b03 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:48.697412 2146 raft_consensus.cc:2243] T 4f7c15529be54a11961f385a953b0b03 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:48.697466 1166 tablet_bootstrap.cc:492] T 1c40b2e06d894a06af9e58285892ea8d P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:48.697582 2146 raft_consensus.cc:2272] T 4f7c15529be54a11961f385a953b0b03 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.697608 1166 ts_tablet_manager.cc:1403] T 1c40b2e06d894a06af9e58285892ea8d P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20251025 07:52:48.698555 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet fd265cdbf6a8440393a2b5d8240d1872 (table default.Table_434ef1ade0a14af987864563993afae8 [id=3a3af35e11944a6f97fc0144cb9d08d4]) successfully deleted
I20251025 07:52:48.698838 1166 raft_consensus.cc:359] T 1c40b2e06d894a06af9e58285892ea8d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.699086 1166 raft_consensus.cc:385] T 1c40b2e06d894a06af9e58285892ea8d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.699182 1166 raft_consensus.cc:740] T 1c40b2e06d894a06af9e58285892ea8d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.699514 1166 consensus_queue.cc:260] T 1c40b2e06d894a06af9e58285892ea8d P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.700134 1169 ts_tablet_manager.cc:1434] T 6d3c9954693d4e8782ee0128ef066aa9 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20251025 07:52:48.700313 1166 ts_tablet_manager.cc:1434] T 1c40b2e06d894a06af9e58285892ea8d P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251025 07:52:48.700727 1166 tablet_bootstrap.cc:492] T ed243540f2d746b980c25687fbca7414 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:48.701345 1169 tablet_bootstrap.cc:492] T 48ff4ce7846b4e56a37eee9bcb17bd5e P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:48.701912 2886 raft_consensus.cc:493] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:48.702095 2886 raft_consensus.cc:515] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:48.702854 2886 leader_election.cc:290] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:48.702986 1166 tablet_bootstrap.cc:654] T ed243540f2d746b980c25687fbca7414 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.703181 2146 ts_tablet_manager.cc:1916] T 4f7c15529be54a11961f385a953b0b03 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.703377 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29dbaf9149134f548a974a5779f7da67" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:48.703496 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29dbaf9149134f548a974a5779f7da67" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:48.703560 772 raft_consensus.cc:2468] T 29dbaf9149134f548a974a5779f7da67 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
W20251025 07:52:48.703944 850 leader_election.cc:343] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.704492 847 leader_election.cc:304] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:48.704965 2886 raft_consensus.cc:2804] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:48.705127 2886 raft_consensus.cc:493] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:48.705272 2886 raft_consensus.cc:3060] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.710880 2886 raft_consensus.cc:515] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:48.712009 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29dbaf9149134f548a974a5779f7da67" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:48.712576 850 leader_election.cc:343] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.712837 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29dbaf9149134f548a974a5779f7da67" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:48.713053 772 raft_consensus.cc:3060] T 29dbaf9149134f548a974a5779f7da67 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.714772 2166 ts_tablet_manager.cc:1929] T f97acc9a638b4d5a968dcc3412a384bc P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.714897 2166 log.cc:1199] T f97acc9a638b4d5a968dcc3412a384bc P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/f97acc9a638b4d5a968dcc3412a384bc
I20251025 07:52:48.714987 1166 tablet_bootstrap.cc:492] T ed243540f2d746b980c25687fbca7414 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:48.715160 1166 ts_tablet_manager.cc:1403] T ed243540f2d746b980c25687fbca7414 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.003s
I20251025 07:52:48.715821 2146 ts_tablet_manager.cc:1929] T 4f7c15529be54a11961f385a953b0b03 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:48.715973 2146 log.cc:1199] T 4f7c15529be54a11961f385a953b0b03 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/4f7c15529be54a11961f385a953b0b03
I20251025 07:52:48.716131 772 raft_consensus.cc:2468] T 29dbaf9149134f548a974a5779f7da67 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:48.716961 1169 tablet_bootstrap.cc:654] T 48ff4ce7846b4e56a37eee9bcb17bd5e P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.717104 2886 leader_election.cc:290] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:48.717593 2146 ts_tablet_manager.cc:1950] T 4f7c15529be54a11961f385a953b0b03 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:48.717612 1166 raft_consensus.cc:359] T ed243540f2d746b980c25687fbca7414 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.717830 1166 raft_consensus.cc:385] T ed243540f2d746b980c25687fbca7414 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.717932 1166 raft_consensus.cc:740] T ed243540f2d746b980c25687fbca7414 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.718174 1166 consensus_queue.cc:260] T ed243540f2d746b980c25687fbca7414 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.718453 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 197) took 7288 ms. Trace:
I20251025 07:52:48.718554 2146 rpcz_store.cc:276] 1025 07:52:41.430028 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.430496 (+ 468us) service_pool.cc:225] Handling call
1025 07:52:48.697100 (+7266604us) ts_tablet_manager.cc:1151] Deleting tablet 4f7c15529be54a11961f385a953b0b03
1025 07:52:48.697109 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.709878 (+ 12769us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.715718 (+ 5840us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.718393 (+ 2675us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7266433}
I20251025 07:52:48.718711 1166 ts_tablet_manager.cc:1434] T ed243540f2d746b980c25687fbca7414 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251025 07:52:48.719029 1166 tablet_bootstrap.cc:492] T 288e543bb279410c91c01dd1623e110f P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:48.719013 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 4f7c15529be54a11961f385a953b0b03 (table default.Table_06053a14dcc84e79917273c10b06b751 [id=c42ad6bf27aa4744b66b4462ac97e57d]) successfully deleted
I20251025 07:52:48.720232 847 leader_election.cc:304] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:48.720326 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 198) took 7289 ms. Trace:
I20251025 07:52:48.720413 2146 rpcz_store.cc:276] 1025 07:52:41.430853 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.431038 (+ 185us) service_pool.cc:225] Handling call
1025 07:52:48.720157 (+7289119us) ts_tablet_manager.cc:1151] Deleting tablet 72dc084f8efc45478771c247e07a16f4
1025 07:52:48.720166 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.720270 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7288921}
W20251025 07:52:48.720829 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 72dc084f8efc45478771c247e07a16f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72dc084f8efc45478771c247e07a16f4 already in progress: creating tablet
I20251025 07:52:48.721096 2166 ts_tablet_manager.cc:1950] T f97acc9a638b4d5a968dcc3412a384bc P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:48.721681 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 199) took 7290 ms. Trace:
I20251025 07:52:48.721763 2146 rpcz_store.cc:276] 1025 07:52:41.431003 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.431412 (+ 409us) service_pool.cc:225] Handling call
1025 07:52:48.721558 (+7290146us) ts_tablet_manager.cc:1151] Deleting tablet be5ab0db6ac34c2c9a370445e251eb46
1025 07:52:48.721566 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.721626 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7289989}
I20251025 07:52:48.722164 2886 raft_consensus.cc:2804] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
W20251025 07:52:48.722186 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet be5ab0db6ac34c2c9a370445e251eb46 because the tablet was not found. No further retry: Not found: Tablet not found: be5ab0db6ac34c2c9a370445e251eb46
I20251025 07:52:48.722389 2886 raft_consensus.cc:697] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:48.722754 2886 consensus_queue.cc:237] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:48.722910 2146 tablet_replica.cc:333] T e47c4f741f354a19b49777bd06e54764 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:48.723461 2169 ts_tablet_manager.cc:1929] T f656945454fe4921ab819c772a089eb3 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.723565 2169 log.cc:1199] T f656945454fe4921ab819c772a089eb3 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/f656945454fe4921ab819c772a089eb3
I20251025 07:52:48.724459 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 175) took 7234 ms. Trace:
I20251025 07:52:48.724545 2166 rpcz_store.cc:276] 1025 07:52:41.490390 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.490429 (+ 39us) service_pool.cc:225] Handling call
1025 07:52:48.672746 (+7182317us) ts_tablet_manager.cc:1151] Deleting tablet f97acc9a638b4d5a968dcc3412a384bc
1025 07:52:48.672849 (+ 103us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.695900 (+ 23051us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.714673 (+ 18773us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.724441 (+ 9768us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2577,"tablet-delete.queue_time_us":7182184}
I20251025 07:52:48.725059 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet f97acc9a638b4d5a968dcc3412a384bc (table default.Table_e50bf47b78d94bbc811e6f3fb3551edb [id=0827603890c8466082bc6c306c9c66af]) successfully deleted
I20251025 07:52:48.723904 2169 ts_tablet_manager.cc:1950] T f656945454fe4921ab819c772a089eb3 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:48.726317 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 176) took 7235 ms. Trace:
I20251025 07:52:48.727205 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet f70091024c034c118fb86578db19a31c (table default.Table_b7da40b96d6c4359bed6f898e8cf6275 [id=e041946c093b4c789ce175c736cc230b]). Expected version 2 got 0
I20251025 07:52:48.728256 2166 rpcz_store.cc:276] 1025 07:52:41.490666 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.490706 (+ 40us) service_pool.cc:225] Handling call
1025 07:52:48.726230 (+7235524us) ts_tablet_manager.cc:1151] Deleting tablet 410ef150a7014c6c867a021b544a2f4e
1025 07:52:48.726240 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.726301 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7235369}
I20251025 07:52:48.728917 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 177) took 7237 ms. Trace:
I20251025 07:52:48.728982 2166 rpcz_store.cc:276] 1025 07:52:41.490971 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.491280 (+ 309us) service_pool.cc:225] Handling call
1025 07:52:48.728831 (+7237551us) ts_tablet_manager.cc:1151] Deleting tablet 2f26667f053a4f16b5da330c0b5e40f9
1025 07:52:48.728840 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.728901 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7237416}
I20251025 07:52:48.729120 1166 tablet_bootstrap.cc:654] T 288e543bb279410c91c01dd1623e110f P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.729353 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 196) took 7265 ms. Trace:
I20251025 07:52:48.729439 2169 rpcz_store.cc:276] 1025 07:52:41.463574 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.463885 (+ 311us) service_pool.cc:225] Handling call
1025 07:52:48.685652 (+7221767us) ts_tablet_manager.cc:1151] Deleting tablet f656945454fe4921ab819c772a089eb3
1025 07:52:48.685661 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.708148 (+ 22487us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.723333 (+ 15185us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.729333 (+ 6000us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7221622}
I20251025 07:52:48.729811 2166 tablet_replica.cc:333] T 26c849d376d143c6bef0c8e0e1d7d03a P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:48.729987 2166 raft_consensus.cc:2243] T 26c849d376d143c6bef0c8e0e1d7d03a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:48.730288 2169 tablet_replica.cc:333] T e54fe7cbcffa40c4b04144b47bee2b48 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:48.730461 2169 raft_consensus.cc:2243] T e54fe7cbcffa40c4b04144b47bee2b48 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:48.730582 2169 raft_consensus.cc:2272] T e54fe7cbcffa40c4b04144b47bee2b48 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.731060 2169 ts_tablet_manager.cc:1916] T e54fe7cbcffa40c4b04144b47bee2b48 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.733407 1166 tablet_bootstrap.cc:492] T 288e543bb279410c91c01dd1623e110f P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:48.733503 2888 raft_consensus.cc:493] T ed243540f2d746b980c25687fbca7414 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:48.733561 1166 ts_tablet_manager.cc:1403] T 288e543bb279410c91c01dd1623e110f P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.004s
I20251025 07:52:48.733621 2888 raft_consensus.cc:515] T ed243540f2d746b980c25687fbca7414 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.734205 2888 leader_election.cc:290] T ed243540f2d746b980c25687fbca7414 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:48.734284 1166 raft_consensus.cc:359] T 288e543bb279410c91c01dd1623e110f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.734483 1166 raft_consensus.cc:385] T 288e543bb279410c91c01dd1623e110f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.734551 1166 raft_consensus.cc:740] T 288e543bb279410c91c01dd1623e110f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.734767 1166 consensus_queue.cc:260] T 288e543bb279410c91c01dd1623e110f P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.734786 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed243540f2d746b980c25687fbca7414" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:48.735213 1166 ts_tablet_manager.cc:1434] T 288e543bb279410c91c01dd1623e110f P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:48.735507 1166 tablet_bootstrap.cc:492] T 075f0a151e024220b51ba7fcaecdcc3d P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
W20251025 07:52:48.736042 712 leader_election.cc:343] T ed243540f2d746b980c25687fbca7414 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.730149 2166 raft_consensus.cc:2272] T 26c849d376d143c6bef0c8e0e1d7d03a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.736892 2166 ts_tablet_manager.cc:1916] T 26c849d376d143c6bef0c8e0e1d7d03a P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.737900 1166 tablet_bootstrap.cc:654] T 075f0a151e024220b51ba7fcaecdcc3d P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.743134 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed243540f2d746b980c25687fbca7414" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:48.743427 910 raft_consensus.cc:2468] T ed243540f2d746b980c25687fbca7414 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
W20251025 07:52:48.728688 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 410ef150a7014c6c867a021b544a2f4e because the tablet was not found. No further retry: Not found: Tablet not found: 410ef150a7014c6c867a021b544a2f4e
W20251025 07:52:48.743786 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 2f26667f053a4f16b5da330c0b5e40f9 because the tablet was not found. No further retry: Not found: Tablet not found: 2f26667f053a4f16b5da330c0b5e40f9
I20251025 07:52:48.734308 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet f656945454fe4921ab819c772a089eb3 (table default.Table_7ce7627f71134d5c989f9ba5c6766a2a [id=b5bf17121a4e4e3a8f21dccf1dba4a99]) successfully deleted
I20251025 07:52:48.744287 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 5dab439bc50d4e869548154e1d222ad6 (table default.Table_fb9def2302784c999bf0f5707c60eb4c [id=45325c188eac4d34bb016d01de091f87]). Expected version 3 got 0
I20251025 07:52:48.744738 710 leader_election.cc:304] T ed243540f2d746b980c25687fbca7414 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:48.745118 2888 raft_consensus.cc:2804] T ed243540f2d746b980c25687fbca7414 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:48.745177 2888 raft_consensus.cc:493] T ed243540f2d746b980c25687fbca7414 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:48.745229 2888 raft_consensus.cc:3060] T ed243540f2d746b980c25687fbca7414 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.747802 2888 raft_consensus.cc:515] T ed243540f2d746b980c25687fbca7414 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.748733 2888 leader_election.cc:290] T ed243540f2d746b980c25687fbca7414 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:48.748745 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed243540f2d746b980c25687fbca7414" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:48.749264 712 leader_election.cc:343] T ed243540f2d746b980c25687fbca7414 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.749893 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed243540f2d746b980c25687fbca7414" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:48.750125 910 raft_consensus.cc:3060] T ed243540f2d746b980c25687fbca7414 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.751261 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 5fc092c2bb2d41ac8c5b4fc07460c8ec (table default.Table_cd20e202397645f8821dcde13b5c0ce3 [id=630c274339664b9db18ab4ef7c29c16f]). Expected version 2 got 0
I20251025 07:52:48.751488 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]). Expected version 3 got 0
I20251025 07:52:48.751658 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 22f7db660616457b83b24e947a0b0bae (table default.Table_4a86191163f2408f9bcc795cd8d1bbf6 [id=bb316a7e000748ca902a9bfb494802d7]). Expected version 1 got 0
I20251025 07:52:48.751835 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]). Expected version 1 got 0
I20251025 07:52:48.753271 1167 tablet_bootstrap.cc:492] T 3e79d69d5d004cc395b8e35569e25440 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:48.753389 1167 ts_tablet_manager.cc:1403] T 3e79d69d5d004cc395b8e35569e25440 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.120s user 0.023s sys 0.031s
I20251025 07:52:48.753913 910 raft_consensus.cc:2468] T ed243540f2d746b980c25687fbca7414 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:48.754060 1167 raft_consensus.cc:359] T 3e79d69d5d004cc395b8e35569e25440 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.754223 1167 raft_consensus.cc:385] T 3e79d69d5d004cc395b8e35569e25440 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.754276 1167 raft_consensus.cc:740] T 3e79d69d5d004cc395b8e35569e25440 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.754444 1167 consensus_queue.cc:260] T 3e79d69d5d004cc395b8e35569e25440 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.755095 1167 ts_tablet_manager.cc:1434] T 3e79d69d5d004cc395b8e35569e25440 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:48.755359 710 leader_election.cc:304] T ed243540f2d746b980c25687fbca7414 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:48.755734 2888 raft_consensus.cc:2804] T ed243540f2d746b980c25687fbca7414 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:48.755972 2888 raft_consensus.cc:697] T ed243540f2d746b980c25687fbca7414 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:48.756242 2888 consensus_queue.cc:237] T ed243540f2d746b980c25687fbca7414 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.759171 1167 tablet_bootstrap.cc:492] T 8917a289bd9b4428a4603429edd94581 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:48.761402 1167 tablet_bootstrap.cc:654] T 8917a289bd9b4428a4603429edd94581 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.761408 2146 raft_consensus.cc:2243] T e47c4f741f354a19b49777bd06e54764 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:48.761888 2146 pending_rounds.cc:70] T e47c4f741f354a19b49777bd06e54764 P 3aea2869ff814cc7b142012faa274382: Trying to abort 1 pending ops.
I20251025 07:52:48.761952 2146 pending_rounds.cc:77] T e47c4f741f354a19b49777bd06e54764 P 3aea2869ff814cc7b142012faa274382: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214607436552515584 op_type: NO_OP noop_request { }
I20251025 07:52:48.762070 2146 raft_consensus.cc:2889] T e47c4f741f354a19b49777bd06e54764 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 07:52:48.762204 2146 raft_consensus.cc:2272] T e47c4f741f354a19b49777bd06e54764 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.763288 2146 ts_tablet_manager.cc:1916] T e47c4f741f354a19b49777bd06e54764 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.763396 2166 ts_tablet_manager.cc:1929] T 26c849d376d143c6bef0c8e0e1d7d03a P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.763501 2166 log.cc:1199] T 26c849d376d143c6bef0c8e0e1d7d03a P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/26c849d376d143c6bef0c8e0e1d7d03a
I20251025 07:52:48.763855 2166 ts_tablet_manager.cc:1950] T 26c849d376d143c6bef0c8e0e1d7d03a P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:48.764312 1169 tablet_bootstrap.cc:492] T 48ff4ce7846b4e56a37eee9bcb17bd5e P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:48.764495 1169 ts_tablet_manager.cc:1403] T 48ff4ce7846b4e56a37eee9bcb17bd5e P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.063s user 0.004s sys 0.000s
I20251025 07:52:48.764590 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 178) took 7273 ms. Trace:
I20251025 07:52:48.764665 2166 rpcz_store.cc:276] 1025 07:52:41.491526 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.491568 (+ 42us) service_pool.cc:225] Handling call
1025 07:52:48.729768 (+7238200us) ts_tablet_manager.cc:1151] Deleting tablet 26c849d376d143c6bef0c8e0e1d7d03a
1025 07:52:48.729776 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.741116 (+ 11340us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.763272 (+ 22156us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.764571 (+ 1299us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7238075}
I20251025 07:52:48.764971 2166 tablet_replica.cc:333] T 04449cc97f404a19b6a1b20dea074f95 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:48.765151 2166 raft_consensus.cc:2243] T 04449cc97f404a19b6a1b20dea074f95 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:48.765192 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 26c849d376d143c6bef0c8e0e1d7d03a (table default.Table_f8c38604344c429295d6d0ab9cb85b13 [id=c2f7a6281b6f47c990b096412f2201a9]) successfully deleted
I20251025 07:52:48.765295 2166 raft_consensus.cc:2272] T 04449cc97f404a19b6a1b20dea074f95 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.765947 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 3a0f50c8000144a4acb5219484b01800 (table default.Table_c62e0dc1ae504f9bb912013506ff3031 [id=6f395892b09240fea02482b3199cfb5e]). Expected version 1 got 0
I20251025 07:52:48.766911 2169 ts_tablet_manager.cc:1929] T e54fe7cbcffa40c4b04144b47bee2b48 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.767846 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525]). Expected version 1 got 0
I20251025 07:52:48.769723 2146 ts_tablet_manager.cc:1929] T e47c4f741f354a19b49777bd06e54764 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.769871 2146 log.cc:1199] T e47c4f741f354a19b49777bd06e54764 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/e47c4f741f354a19b49777bd06e54764
I20251025 07:52:48.769958 2169 log.cc:1199] T e54fe7cbcffa40c4b04144b47bee2b48 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/e54fe7cbcffa40c4b04144b47bee2b48
I20251025 07:52:48.767477 1169 raft_consensus.cc:359] T 48ff4ce7846b4e56a37eee9bcb17bd5e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.770187 1169 raft_consensus.cc:385] T 48ff4ce7846b4e56a37eee9bcb17bd5e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.770291 2146 ts_tablet_manager.cc:1950] T e47c4f741f354a19b49777bd06e54764 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:48.770295 1169 raft_consensus.cc:740] T 48ff4ce7846b4e56a37eee9bcb17bd5e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.770385 2169 ts_tablet_manager.cc:1950] T e54fe7cbcffa40c4b04144b47bee2b48 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:48.770712 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 200) took 7338 ms. Trace:
I20251025 07:52:48.770628 1169 consensus_queue.cc:260] T 48ff4ce7846b4e56a37eee9bcb17bd5e P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.770812 2146 rpcz_store.cc:276] 1025 07:52:41.431718 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.432084 (+ 366us) service_pool.cc:225] Handling call
1025 07:52:48.722830 (+7290746us) ts_tablet_manager.cc:1151] Deleting tablet e47c4f741f354a19b49777bd06e54764
1025 07:52:48.722839 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.761690 (+ 38851us) spinlock_profiling.cc:243] Waited 38.3 ms on lock 0x616000283354. stack: 00007ffaa94bd9f6 00007ffaa94bd7b0 00007ffaa84993ad 0000000000983f04 0000000000983ecd 000000000098389a 00007ffab5383100 00007ffab66d0ae3 00007ffab928d287 00007ffab92c02d4 00007ffab92a0d70 00007ffab92a0afd 0000000000a8867c 00007ffaa9503d7f 00007ffaa950704b 00007ffaa9506ec8
1025 07:52:48.766539 (+ 4849us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.769609 (+ 3070us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.770658 (+ 1049us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":88069504,"tablet-delete.queue_time_us":7290229}
I20251025 07:52:48.771136 1169 ts_tablet_manager.cc:1434] T 48ff4ce7846b4e56a37eee9bcb17bd5e P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20251025 07:52:48.771396 1169 tablet_bootstrap.cc:492] T e9e8fffb8e984463b5fa2c7dd9e93227 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:48.771636 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 197) took 7307 ms. Trace:
I20251025 07:52:48.771641 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 201) took 7339 ms. Trace:
I20251025 07:52:48.771729 2146 rpcz_store.cc:276] 1025 07:52:41.432350 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.432776 (+ 426us) service_pool.cc:225] Handling call
1025 07:52:48.771495 (+7338719us) ts_tablet_manager.cc:1151] Deleting tablet 7f29b64704e147aa9b0b5f6cda860f09
1025 07:52:48.771505 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.771584 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7338562}
I20251025 07:52:48.771763 2169 rpcz_store.cc:276] 1025 07:52:41.464198 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.464243 (+ 45us) service_pool.cc:225] Handling call
1025 07:52:48.730244 (+7266001us) ts_tablet_manager.cc:1151] Deleting tablet e54fe7cbcffa40c4b04144b47bee2b48
1025 07:52:48.730254 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.747574 (+ 17320us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.766805 (+ 19231us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.771617 (+ 4812us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":150,"tablet-delete.queue_time_us":7265871}
I20251025 07:52:48.772143 2146 tablet_replica.cc:333] T 95efd553493e4966bc48a01cac6e1e10 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:48.772353 2146 raft_consensus.cc:2243] T 95efd553493e4966bc48a01cac6e1e10 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:48.772394 2169 tablet_replica.cc:333] T b94634ac5077439fadfbbce569f60e8b P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:48.772554 2146 raft_consensus.cc:2272] T 95efd553493e4966bc48a01cac6e1e10 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.773289 2146 ts_tablet_manager.cc:1916] T 95efd553493e4966bc48a01cac6e1e10 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.773550 1169 tablet_bootstrap.cc:654] T e9e8fffb8e984463b5fa2c7dd9e93227 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.771317 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet e47c4f741f354a19b49777bd06e54764 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]) successfully deleted
I20251025 07:52:48.772312 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet e54fe7cbcffa40c4b04144b47bee2b48 (table default.Table_590cc281d8b042b19ef12c7f6436752a [id=b1eee5c0243a48c4bbdecc0b4f38ab2d]) successfully deleted
I20251025 07:52:48.774412 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_67b99749f3bf43228b40d8b9807e69b4 [id=fc8180c09d7b4e74b2e161f7acb8fe0e]). Expected version 2 got 0
W20251025 07:52:48.774461 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 7f29b64704e147aa9b0b5f6cda860f09 because the tablet was not found. No further retry: Not found: Tablet not found: 7f29b64704e147aa9b0b5f6cda860f09
I20251025 07:52:48.774819 2169 raft_consensus.cc:2243] T b94634ac5077439fadfbbce569f60e8b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:48.775014 2169 raft_consensus.cc:2272] T b94634ac5077439fadfbbce569f60e8b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.775082 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c]). Expected version 1 got 0
I20251025 07:52:48.775538 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 755f0d6c87044e20a8020f8410940123 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 4 got 0
I20251025 07:52:48.776055 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 830fdbc6812e416690d0db42a73af503 (table default.Table_92824b3f2a774564b3aea04aaa25d204 [id=fb21a45b86ab446e8c3dae4512a9598f]). Expected version 1 got 0
I20251025 07:52:48.775600 2169 ts_tablet_manager.cc:1916] T b94634ac5077439fadfbbce569f60e8b P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.776813 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236]). Expected version 1 got 0
I20251025 07:52:48.776995 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 8707b93d491e4e4386d148087a2f0870 (table default.Table_799dcb1cef2f4fe6b9b8f17f9f1026c5 [id=59840a567e5244e8a9024cbad42471e4]). Expected version 1 got 0
I20251025 07:52:48.785046 2146 ts_tablet_manager.cc:1929] T 95efd553493e4966bc48a01cac6e1e10 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:48.785218 2146 log.cc:1199] T 95efd553493e4966bc48a01cac6e1e10 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/95efd553493e4966bc48a01cac6e1e10
I20251025 07:52:48.785554 2146 ts_tablet_manager.cc:1950] T 95efd553493e4966bc48a01cac6e1e10 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:48.785864 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 202) took 7352 ms. Trace:
I20251025 07:52:48.786973 2888 raft_consensus.cc:493] T 2e098c9c4613459295b6043048cba587 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:48.787168 2888 raft_consensus.cc:515] T 2e098c9c4613459295b6043048cba587 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:48.787905 2888 leader_election.cc:290] T 2e098c9c4613459295b6043048cba587 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:48.786005 2146 rpcz_store.cc:276] 1025 07:52:41.433096 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.433304 (+ 208us) service_pool.cc:225] Handling call
1025 07:52:48.772060 (+7338756us) ts_tablet_manager.cc:1151] Deleting tablet 95efd553493e4966bc48a01cac6e1e10
1025 07:52:48.772068 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.777129 (+ 5061us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.784900 (+ 7771us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.785821 (+ 921us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7338605}
I20251025 07:52:48.788573 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 95efd553493e4966bc48a01cac6e1e10 (table default.Table_c4716673eec04f3baa6fc852c668fed8 [id=1d09f9e02b4a4a4d8d38ee522fea841b]) successfully deleted
I20251025 07:52:48.788784 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e098c9c4613459295b6043048cba587" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:48.789259 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 203) took 7355 ms. Trace:
I20251025 07:52:48.789731 2146 rpcz_store.cc:276] 1025 07:52:41.433602 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.433757 (+ 155us) service_pool.cc:225] Handling call
1025 07:52:48.789106 (+7355349us) ts_tablet_manager.cc:1151] Deleting tablet 896bd7870d87408e8adfdb7a78998111
1025 07:52:48.789118 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.789207 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7355172}
W20251025 07:52:48.791112 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 896bd7870d87408e8adfdb7a78998111 because the tablet was not found. No further retry: Not found: Tablet not found: 896bd7870d87408e8adfdb7a78998111
I20251025 07:52:48.790737 1166 tablet_bootstrap.cc:492] T 075f0a151e024220b51ba7fcaecdcc3d P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:48.791466 1166 ts_tablet_manager.cc:1403] T 075f0a151e024220b51ba7fcaecdcc3d P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.056s user 0.006s sys 0.020s
W20251025 07:52:48.792006 712 leader_election.cc:343] T 2e098c9c4613459295b6043048cba587 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.792225 1166 raft_consensus.cc:359] T 075f0a151e024220b51ba7fcaecdcc3d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.792438 1166 raft_consensus.cc:385] T 075f0a151e024220b51ba7fcaecdcc3d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.792544 1166 raft_consensus.cc:740] T 075f0a151e024220b51ba7fcaecdcc3d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.792810 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e098c9c4613459295b6043048cba587" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:48.793005 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 204) took 7358 ms. Trace:
I20251025 07:52:48.793113 2146 rpcz_store.cc:276] 1025 07:52:41.434087 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.434232 (+ 145us) service_pool.cc:225] Handling call
1025 07:52:48.792893 (+7358661us) ts_tablet_manager.cc:1151] Deleting tablet 27651312cc2e42a8bfcd245f5d301679
1025 07:52:48.792901 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.792964 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7358467}
I20251025 07:52:48.793277 2166 ts_tablet_manager.cc:1916] T 04449cc97f404a19b6a1b20dea074f95 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:48.793545 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 27651312cc2e42a8bfcd245f5d301679 because the tablet was not found. No further retry: Not found: Tablet not found: 27651312cc2e42a8bfcd245f5d301679
I20251025 07:52:48.792811 1166 consensus_queue.cc:260] T 075f0a151e024220b51ba7fcaecdcc3d P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.794677 1166 ts_tablet_manager.cc:1434] T 075f0a151e024220b51ba7fcaecdcc3d P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251025 07:52:48.795015 1166 tablet_bootstrap.cc:492] T 3b940fa5ae26427dafbcfd26372754b3 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:48.796890 1166 tablet_bootstrap.cc:654] T 3b940fa5ae26427dafbcfd26372754b3 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:48.797546 710 leader_election.cc:343] T 2e098c9c4613459295b6043048cba587 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.797698 710 leader_election.cc:304] T 2e098c9c4613459295b6043048cba587 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:48.797194 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 205) took 7362 ms. Trace:
I20251025 07:52:48.798244 2888 raft_consensus.cc:2749] T 2e098c9c4613459295b6043048cba587 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:48.798099 2146 rpcz_store.cc:276] 1025 07:52:41.434197 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.436215 (+ 2018us) service_pool.cc:225] Handling call
1025 07:52:48.797074 (+7360859us) ts_tablet_manager.cc:1151] Deleting tablet fab67b56eaa44b54886c17181d8c52b2
1025 07:52:48.797083 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.797150 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7360693}
I20251025 07:52:48.798779 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet b56209e4934e4407a8b4518ca4af0d46 (table default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de]). Expected version 1 got 0
I20251025 07:52:48.799094 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 206) took 7362 ms. Trace:
I20251025 07:52:48.799209 2146 rpcz_store.cc:276] 1025 07:52:41.436552 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.436726 (+ 174us) service_pool.cc:225] Handling call
1025 07:52:48.798975 (+7362249us) ts_tablet_manager.cc:1151] Deleting tablet 2e098c9c4613459295b6043048cba587
1025 07:52:48.798983 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.799064 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7362091}
I20251025 07:52:48.799640 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 207) took 7362 ms. Trace:
I20251025 07:52:48.799753 2146 rpcz_store.cc:276] 1025 07:52:41.437035 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.437203 (+ 168us) service_pool.cc:225] Handling call
1025 07:52:48.799549 (+7362346us) ts_tablet_manager.cc:1151] Deleting tablet 4f25c9caa05b4da4bd56a8a21fde423c
1025 07:52:48.799557 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.799612 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7362172}
I20251025 07:52:48.800027 1167 tablet_bootstrap.cc:492] T 8917a289bd9b4428a4603429edd94581 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:48.800132 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 208) took 7362 ms. Trace:
I20251025 07:52:48.800247 1167 ts_tablet_manager.cc:1403] T 8917a289bd9b4428a4603429edd94581 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.000s
I20251025 07:52:48.800247 2146 rpcz_store.cc:276] 1025 07:52:41.437528 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.437688 (+ 160us) service_pool.cc:225] Handling call
1025 07:52:48.800049 (+7362361us) ts_tablet_manager.cc:1151] Deleting tablet 887e7c7a5d0e4aaf88f26246306df19c
1025 07:52:48.800057 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.800103 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7362216}
I20251025 07:52:48.800621 2146 tablet_replica.cc:333] T 6dcab53a020847a1aa27930265ba0205 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:48.800854 2146 raft_consensus.cc:2243] T 6dcab53a020847a1aa27930265ba0205 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:48.801038 2146 raft_consensus.cc:2272] T 6dcab53a020847a1aa27930265ba0205 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.801195 1167 raft_consensus.cc:359] T 8917a289bd9b4428a4603429edd94581 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.801419 1167 raft_consensus.cc:385] T 8917a289bd9b4428a4603429edd94581 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.801508 1167 raft_consensus.cc:740] T 8917a289bd9b4428a4603429edd94581 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.801625 2146 ts_tablet_manager.cc:1916] T 6dcab53a020847a1aa27930265ba0205 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.801808 1167 consensus_queue.cc:260] T 8917a289bd9b4428a4603429edd94581 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.802481 1167 ts_tablet_manager.cc:1434] T 8917a289bd9b4428a4603429edd94581 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:48.802623 2169 ts_tablet_manager.cc:1929] T b94634ac5077439fadfbbce569f60e8b P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:48.802795 1167 tablet_bootstrap.cc:492] T ee1c8ea694284d80a30f9c05be4cbf49 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
W20251025 07:52:48.798830 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet fab67b56eaa44b54886c17181d8c52b2 because the tablet was not found. No further retry: Not found: Tablet not found: fab67b56eaa44b54886c17181d8c52b2
I20251025 07:52:48.803217 2169 log.cc:1199] T b94634ac5077439fadfbbce569f60e8b P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/b94634ac5077439fadfbbce569f60e8b
I20251025 07:52:48.803543 2169 ts_tablet_manager.cc:1950] T b94634ac5077439fadfbbce569f60e8b P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
W20251025 07:52:48.803543 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 2e098c9c4613459295b6043048cba587 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e098c9c4613459295b6043048cba587 already in progress: creating tablet
W20251025 07:52:48.803797 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 4f25c9caa05b4da4bd56a8a21fde423c because the tablet was not found. No further retry: Not found: Tablet not found: 4f25c9caa05b4da4bd56a8a21fde423c
W20251025 07:52:48.803982 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 887e7c7a5d0e4aaf88f26246306df19c because the tablet was not found. No further retry: Not found: Tablet not found: 887e7c7a5d0e4aaf88f26246306df19c
I20251025 07:52:48.804481 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84]). Expected version 1 got 0
W20251025 07:52:48.806305 850 consensus_peers.cc:597] T 1068f566477e4c6db7dde566fff4c2fb P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:48.806349 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_d11bba62f65740319701b4337a5c255a [id=ad812862698d421184ebf501ce79b7be]). Expected version 3 got 0
I20251025 07:52:48.807123 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 198) took 7342 ms. Trace:
I20251025 07:52:48.807228 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 288e543bb279410c91c01dd1623e110f (table default.Table_6e242ba55b194c23a6701f4b064fdbe5 [id=4c34b64b299f4162a53866cf01c5298f]). Expected version 1 got 0
I20251025 07:52:48.807264 2169 rpcz_store.cc:276] 1025 07:52:41.464519 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.464558 (+ 39us) service_pool.cc:225] Handling call
1025 07:52:48.772340 (+7307782us) ts_tablet_manager.cc:1151] Deleting tablet b94634ac5077439fadfbbce569f60e8b
1025 07:52:48.772350 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.783254 (+ 10904us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.802510 (+ 19256us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.807103 (+ 4593us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7307652}
I20251025 07:52:48.807776 2146 ts_tablet_manager.cc:1929] T 6dcab53a020847a1aa27930265ba0205 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.807804 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet b94634ac5077439fadfbbce569f60e8b (table default.Table_1ff54f4df7a2444489ae4e4060084156 [id=b88c37a678344c6b8e8017553e486cb2]) successfully deleted
I20251025 07:52:48.807919 2146 log.cc:1199] T 6dcab53a020847a1aa27930265ba0205 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/6dcab53a020847a1aa27930265ba0205
I20251025 07:52:48.808071 2169 tablet_replica.cc:333] T 92aa605fabbc4257a6a204c087e7cc7f P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:48.808138 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3]). Expected version 2 got 0
I20251025 07:52:48.808298 2146 ts_tablet_manager.cc:1950] T 6dcab53a020847a1aa27930265ba0205 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:48.809072 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 209) took 7371 ms. Trace:
I20251025 07:52:48.809168 2146 rpcz_store.cc:276] 1025 07:52:41.437987 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.438167 (+ 180us) service_pool.cc:225] Handling call
1025 07:52:48.800563 (+7362396us) ts_tablet_manager.cc:1151] Deleting tablet 6dcab53a020847a1aa27930265ba0205
1025 07:52:48.800571 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.804727 (+ 4156us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.807672 (+ 2945us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.809019 (+ 1347us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7362213}
I20251025 07:52:48.809590 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 6dcab53a020847a1aa27930265ba0205 (table default.Table_9b7fc5a1f4944e749c90c1119d0b0054 [id=cb79dc5b6680450cadf2350095389c9e]) successfully deleted
I20251025 07:52:48.809687 2169 raft_consensus.cc:2243] T 92aa605fabbc4257a6a204c087e7cc7f P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:48.809990 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet cd23b8094d7e472eba6bd170248858a9 (table default.Table_cea16af97a2242968d72900d3a84a86d [id=d82b3e1317434cf08a0d51e8a9427224]). Expected version 4 got 3
I20251025 07:52:48.810031 2169 raft_consensus.cc:2272] T 92aa605fabbc4257a6a204c087e7cc7f P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.810442 1169 tablet_bootstrap.cc:492] T e9e8fffb8e984463b5fa2c7dd9e93227 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:48.810563 2169 ts_tablet_manager.cc:1916] T 92aa605fabbc4257a6a204c087e7cc7f P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.810573 1169 ts_tablet_manager.cc:1403] T e9e8fffb8e984463b5fa2c7dd9e93227 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.000s
I20251025 07:52:48.811244 1169 raft_consensus.cc:359] T e9e8fffb8e984463b5fa2c7dd9e93227 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.811419 1169 raft_consensus.cc:385] T e9e8fffb8e984463b5fa2c7dd9e93227 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.811470 1169 raft_consensus.cc:740] T e9e8fffb8e984463b5fa2c7dd9e93227 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.811658 1169 consensus_queue.cc:260] T e9e8fffb8e984463b5fa2c7dd9e93227 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.812091 1169 ts_tablet_manager.cc:1434] T e9e8fffb8e984463b5fa2c7dd9e93227 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251025 07:52:48.818351 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 210) took 7379 ms. Trace:
I20251025 07:52:48.818486 2146 rpcz_store.cc:276] 1025 07:52:41.438547 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.438752 (+ 205us) service_pool.cc:225] Handling call
1025 07:52:48.818187 (+7379435us) ts_tablet_manager.cc:1151] Deleting tablet abf94c32ff2945119b710905c075d63c
1025 07:52:48.818197 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.818280 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7379265}
I20251025 07:52:48.818727 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e]). Expected version 1 got 0
W20251025 07:52:48.818980 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet abf94c32ff2945119b710905c075d63c because the tablet was not found. No further retry: Not found: Tablet not found: abf94c32ff2945119b710905c075d63c
I20251025 07:52:48.819041 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]). Expected version 1 got 0
I20251025 07:52:48.819159 2169 ts_tablet_manager.cc:1929] T 92aa605fabbc4257a6a204c087e7cc7f P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.820031 2146 tablet_replica.cc:333] T 1912e59548874e0e8e989050ec0064c5 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:48.820263 2146 raft_consensus.cc:2243] T 1912e59548874e0e8e989050ec0064c5 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:48.820669 2146 raft_consensus.cc:2272] T 1912e59548874e0e8e989050ec0064c5 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.821754 2166 ts_tablet_manager.cc:1929] T 04449cc97f404a19b6a1b20dea074f95 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.821867 2166 log.cc:1199] T 04449cc97f404a19b6a1b20dea074f95 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/04449cc97f404a19b6a1b20dea074f95
I20251025 07:52:48.822207 2166 ts_tablet_manager.cc:1950] T 04449cc97f404a19b6a1b20dea074f95 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:48.824316 1167 tablet_bootstrap.cc:654] T ee1c8ea694284d80a30f9c05be4cbf49 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.819348 453 catalog_manager.cc:5649] T c0c6324cbae34dcebade3ef9f89a7150 P db42be7ad3c741a0aa427f27e245993b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } }
I20251025 07:52:48.826524 453 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]). Expected version 1 got 0
I20251025 07:52:48.827906 2169 log.cc:1199] T 92aa605fabbc4257a6a204c087e7cc7f P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/92aa605fabbc4257a6a204c087e7cc7f
I20251025 07:52:48.824005 1169 tablet_bootstrap.cc:492] T c187a40f8ff14c6a8c184fbeb030ad13 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:48.827776 3011 raft_consensus.cc:493] T e9e8fffb8e984463b5fa2c7dd9e93227 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:48.828328 2169 ts_tablet_manager.cc:1950] T 92aa605fabbc4257a6a204c087e7cc7f P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:48.828411 3011 raft_consensus.cc:515] T e9e8fffb8e984463b5fa2c7dd9e93227 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.829245 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 199) took 7364 ms. Trace:
I20251025 07:52:48.829293 3011 leader_election.cc:290] T e9e8fffb8e984463b5fa2c7dd9e93227 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:48.830505 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9e8fffb8e984463b5fa2c7dd9e93227" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:48.830725 772 raft_consensus.cc:2393] T e9e8fffb8e984463b5fa2c7dd9e93227 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:48.831110 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9e8fffb8e984463b5fa2c7dd9e93227" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:48.831445 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 171) took 7342 ms. Trace:
I20251025 07:52:48.831516 2166 rpcz_store.cc:276] 1025 07:52:41.489010 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.491873 (+ 2863us) service_pool.cc:225] Handling call
1025 07:52:48.764922 (+7273049us) ts_tablet_manager.cc:1151] Deleting tablet 04449cc97f404a19b6a1b20dea074f95
1025 07:52:48.764933 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.796492 (+ 31559us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.821659 (+ 25167us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.831428 (+ 9769us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7272922}
I20251025 07:52:48.831938 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 04449cc97f404a19b6a1b20dea074f95 (table default.Table_b7da40b96d6c4359bed6f898e8cf6275 [id=e041946c093b4c789ce175c736cc230b]) successfully deleted
I20251025 07:52:48.832298 983 leader_election.cc:304] T e9e8fffb8e984463b5fa2c7dd9e93227 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:48.832523 2467 raft_consensus.cc:493] T 6d3c9954693d4e8782ee0128ef066aa9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:48.832679 3011 raft_consensus.cc:3060] T e9e8fffb8e984463b5fa2c7dd9e93227 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.832641 2467 raft_consensus.cc:515] T 6d3c9954693d4e8782ee0128ef066aa9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.833153 2467 leader_election.cc:290] T 6d3c9954693d4e8782ee0128ef066aa9 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:48.829319 2169 rpcz_store.cc:276] 1025 07:52:41.464817 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.464857 (+ 40us) service_pool.cc:225] Handling call
1025 07:52:48.807830 (+7342973us) ts_tablet_manager.cc:1151] Deleting tablet 92aa605fabbc4257a6a204c087e7cc7f
1025 07:52:48.807840 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.816127 (+ 8287us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.819071 (+ 2944us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.829225 (+ 10154us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7342789}
I20251025 07:52:48.834385 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 179) took 7342 ms. Trace:
I20251025 07:52:48.834759 2169 tablet_replica.cc:333] T 4b28215f61574cf48c447dd01955ac77 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:48.834735 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 92aa605fabbc4257a6a204c087e7cc7f (table default.Table_78724078b4574f35879fd2db388fc4f4 [id=187bde879b044237ada58158eb000123]) successfully deleted
I20251025 07:52:48.835067 2169 raft_consensus.cc:2243] T 4b28215f61574cf48c447dd01955ac77 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:48.835450 2169 raft_consensus.cc:2272] T 4b28215f61574cf48c447dd01955ac77 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.835649 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d3c9954693d4e8782ee0128ef066aa9" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:48.835763 2886 raft_consensus.cc:493] T a7ab3150a35b4caca9119a1374d3f2f7 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:48.835858 772 raft_consensus.cc:2393] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 23eeffd3de254ce18db5febfde971c6b in this term.
I20251025 07:52:48.835944 2886 raft_consensus.cc:515] T a7ab3150a35b4caca9119a1374d3f2f7 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.836133 2169 ts_tablet_manager.cc:1916] T 4b28215f61574cf48c447dd01955ac77 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.836701 2886 leader_election.cc:290] T a7ab3150a35b4caca9119a1374d3f2f7 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:48.837404 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7ab3150a35b4caca9119a1374d3f2f7" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:48.837450 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7ab3150a35b4caca9119a1374d3f2f7" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:48.837610 772 raft_consensus.cc:2468] T a7ab3150a35b4caca9119a1374d3f2f7 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
W20251025 07:52:48.837896 850 leader_election.cc:343] T a7ab3150a35b4caca9119a1374d3f2f7 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.838172 3011 raft_consensus.cc:2749] T e9e8fffb8e984463b5fa2c7dd9e93227 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:48.834465 2166 rpcz_store.cc:276] 1025 07:52:41.491836 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.492030 (+ 194us) service_pool.cc:225] Handling call
1025 07:52:48.834287 (+7342257us) ts_tablet_manager.cc:1151] Deleting tablet b3a0af893a8947848e12093d0a7f0229
1025 07:52:48.834297 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.834368 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7342144}
I20251025 07:52:48.838704 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d3c9954693d4e8782ee0128ef066aa9" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:48.838923 910 raft_consensus.cc:2393] T 6d3c9954693d4e8782ee0128ef066aa9 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:48.839397 983 leader_election.cc:304] T 6d3c9954693d4e8782ee0128ef066aa9 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:48.839638 2467 raft_consensus.cc:3060] T 6d3c9954693d4e8782ee0128ef066aa9 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.840159 847 leader_election.cc:304] T a7ab3150a35b4caca9119a1374d3f2f7 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:48.840523 2886 raft_consensus.cc:2804] T a7ab3150a35b4caca9119a1374d3f2f7 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:48.840590 2886 raft_consensus.cc:493] T a7ab3150a35b4caca9119a1374d3f2f7 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:48.840646 2886 raft_consensus.cc:3060] T a7ab3150a35b4caca9119a1374d3f2f7 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.842301 2467 raft_consensus.cc:2749] T 6d3c9954693d4e8782ee0128ef066aa9 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:48.840162 2888 raft_consensus.cc:493] T a4e4b99be0434baaa75d988281797846 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3aea2869ff814cc7b142012faa274382)
I20251025 07:52:48.842528 2888 raft_consensus.cc:515] T a4e4b99be0434baaa75d988281797846 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:48.843127 2888 leader_election.cc:290] T a4e4b99be0434baaa75d988281797846 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:48.843261 2886 raft_consensus.cc:515] T a7ab3150a35b4caca9119a1374d3f2f7 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.844514 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7ab3150a35b4caca9119a1374d3f2f7" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:48.844964 850 leader_election.cc:343] T a7ab3150a35b4caca9119a1374d3f2f7 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:48.844936 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet b3a0af893a8947848e12093d0a7f0229 because the tablet was not found. No further retry: Not found: Tablet not found: b3a0af893a8947848e12093d0a7f0229
I20251025 07:52:48.845299 2166 tablet_replica.cc:333] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:48.845510 2166 raft_consensus.cc:2243] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
W20251025 07:52:48.845616 712 leader_election.cc:343] T a4e4b99be0434baaa75d988281797846 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Not found: Tablet not found: a4e4b99be0434baaa75d988281797846
I20251025 07:52:48.845683 2166 raft_consensus.cc:2272] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:48.847235 710 leader_election.cc:343] T a4e4b99be0434baaa75d988281797846 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: a4e4b99be0434baaa75d988281797846
I20251025 07:52:48.847342 710 leader_election.cc:304] T a4e4b99be0434baaa75d988281797846 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:48.847720 2888 raft_consensus.cc:2749] T a4e4b99be0434baaa75d988281797846 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251025 07:52:48.848225 2886 leader_election.cc:290] T a7ab3150a35b4caca9119a1374d3f2f7 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:48.848452 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7ab3150a35b4caca9119a1374d3f2f7" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:48.848610 772 raft_consensus.cc:3060] T a7ab3150a35b4caca9119a1374d3f2f7 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.851526 772 raft_consensus.cc:2468] T a7ab3150a35b4caca9119a1374d3f2f7 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:48.852046 847 leader_election.cc:304] T a7ab3150a35b4caca9119a1374d3f2f7 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:48.852380 2991 raft_consensus.cc:2804] T a7ab3150a35b4caca9119a1374d3f2f7 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:48.852525 2991 raft_consensus.cc:697] T a7ab3150a35b4caca9119a1374d3f2f7 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:48.852763 2991 consensus_queue.cc:237] T a7ab3150a35b4caca9119a1374d3f2f7 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.858283 1169 tablet_bootstrap.cc:654] T c187a40f8ff14c6a8c184fbeb030ad13 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.859681 2166 ts_tablet_manager.cc:1916] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.863143 1166 tablet_bootstrap.cc:492] T 3b940fa5ae26427dafbcfd26372754b3 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:48.863284 1166 ts_tablet_manager.cc:1403] T 3b940fa5ae26427dafbcfd26372754b3 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.068s user 0.019s sys 0.006s
I20251025 07:52:48.865103 2798 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "b2dfbd0b44a441c9be9254f660974944" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2dfbd0b44a441c9be9254f660974944" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37421 } } peers { permanent_uuid: "8603bddd8a084ff7a73a25641768dfa9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42203 } } peers { permanent_uuid: "b4a999d754c941ec850c143e4e24c407" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 37611 } } }
I20251025 07:52:48.865353 2798 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 [sys.catalog]: This master's current role is: LEADER
I20251025 07:52:48.863991 1166 raft_consensus.cc:359] T 3b940fa5ae26427dafbcfd26372754b3 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.865559 1166 raft_consensus.cc:385] T 3b940fa5ae26427dafbcfd26372754b3 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.865645 1166 raft_consensus.cc:740] T 3b940fa5ae26427dafbcfd26372754b3 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.865929 1166 consensus_queue.cc:260] T 3b940fa5ae26427dafbcfd26372754b3 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.867213 1166 ts_tablet_manager.cc:1434] T 3b940fa5ae26427dafbcfd26372754b3 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251025 07:52:48.867846 1166 tablet_bootstrap.cc:492] T 67cf96a7dd3f42e19b86746c87e689f6 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:48.868575 2166 ts_tablet_manager.cc:1929] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.868817 2166 log.cc:1199] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/48ff4ce7846b4e56a37eee9bcb17bd5e
I20251025 07:52:48.869202 2166 ts_tablet_manager.cc:1950] T 48ff4ce7846b4e56a37eee9bcb17bd5e P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:48.870018 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 180) took 7377 ms. Trace:
I20251025 07:52:48.870182 2166 rpcz_store.cc:276] 1025 07:52:41.492274 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.492313 (+ 39us) service_pool.cc:225] Handling call
1025 07:52:48.845244 (+7352931us) ts_tablet_manager.cc:1151] Deleting tablet 48ff4ce7846b4e56a37eee9bcb17bd5e
1025 07:52:48.845255 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.864629 (+ 19374us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.868469 (+ 3840us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.869998 (+ 1529us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7352806}
I20251025 07:52:48.870673 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 181) took 7378 ms. Trace:
I20251025 07:52:48.870739 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 48ff4ce7846b4e56a37eee9bcb17bd5e (table default.Table_5eaf42285e474860a44c19683f1fa70f [id=0da8afa0245842cfa80b852852e5235a]) successfully deleted
I20251025 07:52:48.870821 2166 rpcz_store.cc:276] 1025 07:52:41.492547 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.492649 (+ 102us) service_pool.cc:225] Handling call
1025 07:52:48.870565 (+7377916us) ts_tablet_manager.cc:1151] Deleting tablet b947ecc352e54f4caa73c6ca97372256
1025 07:52:48.870573 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.870657 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7377783}
I20251025 07:52:48.871389 1169 tablet_bootstrap.cc:492] T c187a40f8ff14c6a8c184fbeb030ad13 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:48.871552 1169 ts_tablet_manager.cc:1403] T c187a40f8ff14c6a8c184fbeb030ad13 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.048s user 0.001s sys 0.001s
I20251025 07:52:48.871693 1167 tablet_bootstrap.cc:492] T ee1c8ea694284d80a30f9c05be4cbf49 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:48.871821 1167 ts_tablet_manager.cc:1403] T ee1c8ea694284d80a30f9c05be4cbf49 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.069s user 0.004s sys 0.000s
I20251025 07:52:48.867215 2169 ts_tablet_manager.cc:1929] T 4b28215f61574cf48c447dd01955ac77 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.872078 2169 log.cc:1199] T 4b28215f61574cf48c447dd01955ac77 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/4b28215f61574cf48c447dd01955ac77
I20251025 07:52:48.872473 1167 raft_consensus.cc:359] T ee1c8ea694284d80a30f9c05be4cbf49 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.872673 1167 raft_consensus.cc:385] T ee1c8ea694284d80a30f9c05be4cbf49 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.872728 1167 raft_consensus.cc:740] T ee1c8ea694284d80a30f9c05be4cbf49 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.872947 1167 consensus_queue.cc:260] T ee1c8ea694284d80a30f9c05be4cbf49 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.873337 1167 ts_tablet_manager.cc:1434] T ee1c8ea694284d80a30f9c05be4cbf49 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251025 07:52:48.873628 1167 tablet_bootstrap.cc:492] T 0582ee42ab564c07ae3facfc5935617c P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
W20251025 07:52:48.874704 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet b947ecc352e54f4caa73c6ca97372256 because the tablet was not found. No further retry: Not found: Tablet not found: b947ecc352e54f4caa73c6ca97372256
I20251025 07:52:48.877256 2169 ts_tablet_manager.cc:1950] T 4b28215f61574cf48c447dd01955ac77 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:48.878814 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 200) took 7413 ms. Trace:
I20251025 07:52:48.878899 2169 rpcz_store.cc:276] 1025 07:52:41.465262 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.465302 (+ 40us) service_pool.cc:225] Handling call
1025 07:52:48.834708 (+7369406us) ts_tablet_manager.cc:1151] Deleting tablet 4b28215f61574cf48c447dd01955ac77
1025 07:52:48.834721 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.860868 (+ 26147us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.867126 (+ 6258us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.878793 (+ 11667us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7369271}
I20251025 07:52:48.879246 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 201) took 7404 ms. Trace:
I20251025 07:52:48.879313 2169 rpcz_store.cc:276] 1025 07:52:41.474678 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.474981 (+ 303us) service_pool.cc:225] Handling call
1025 07:52:48.879149 (+7404168us) ts_tablet_manager.cc:1151] Deleting tablet 2784bbfb97d5466f9ec856add7d0b29a
1025 07:52:48.879158 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.879230 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7403966}
I20251025 07:52:48.879469 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 4b28215f61574cf48c447dd01955ac77 (table default.Table_a437a2623d704e3ea3c5b8c65c88b936 [id=51b12903e6024e1a9adc2f9fa7400937]) successfully deleted
W20251025 07:52:48.879786 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 2784bbfb97d5466f9ec856add7d0b29a because the tablet was not found. No further retry: Not found: Tablet not found: 2784bbfb97d5466f9ec856add7d0b29a
I20251025 07:52:48.880137 2888 raft_consensus.cc:493] T df9db9dd31d840e0b1394db04d662a62 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:48.880254 2888 raft_consensus.cc:515] T df9db9dd31d840e0b1394db04d662a62 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.880787 2888 leader_election.cc:290] T df9db9dd31d840e0b1394db04d662a62 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:48.881331 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df9db9dd31d840e0b1394db04d662a62" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:48.881847 712 leader_election.cc:343] T df9db9dd31d840e0b1394db04d662a62 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.882086 1166 tablet_bootstrap.cc:654] T 67cf96a7dd3f42e19b86746c87e689f6 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.872249 1169 raft_consensus.cc:359] T c187a40f8ff14c6a8c184fbeb030ad13 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.882521 1169 raft_consensus.cc:385] T c187a40f8ff14c6a8c184fbeb030ad13 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.882593 1169 raft_consensus.cc:740] T c187a40f8ff14c6a8c184fbeb030ad13 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.883224 1169 consensus_queue.cc:260] T c187a40f8ff14c6a8c184fbeb030ad13 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.883491 2467 consensus_queue.cc:1048] T 4f0c4f0e8f014e59be9e46d2afe0be9d P db42be7ad3c741a0aa427f27e245993b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:48.884546 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df9db9dd31d840e0b1394db04d662a62" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:48.886663 710 leader_election.cc:343] T df9db9dd31d840e0b1394db04d662a62 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.886771 710 leader_election.cc:304] T df9db9dd31d840e0b1394db04d662a62 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:48.887182 2888 raft_consensus.cc:2749] T df9db9dd31d840e0b1394db04d662a62 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:48.887341 2146 ts_tablet_manager.cc:1916] T 1912e59548874e0e8e989050ec0064c5 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.888113 1167 tablet_bootstrap.cc:654] T 0582ee42ab564c07ae3facfc5935617c P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:48.891799 442 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet b56209e4934e4407a8b4518ca4af0d46 (table default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet b56209e4934e4407a8b4518ca4af0d46 consensus state has no leader
I20251025 07:52:48.893602 1169 ts_tablet_manager.cc:1434] T c187a40f8ff14c6a8c184fbeb030ad13 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20251025 07:52:48.895918 1169 tablet_bootstrap.cc:492] T 2b594d4718344aafbdaec34040695589 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:48.896162 2686 raft_consensus.cc:493] T 288e543bb279410c91c01dd1623e110f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:48.896292 2686 raft_consensus.cc:515] T 288e543bb279410c91c01dd1623e110f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.896885 2686 leader_election.cc:290] T 288e543bb279410c91c01dd1623e110f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:48.897367 1169 tablet_bootstrap.cc:654] T 2b594d4718344aafbdaec34040695589 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.897454 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "288e543bb279410c91c01dd1623e110f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:48.897938 850 leader_election.cc:343] T 288e543bb279410c91c01dd1623e110f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.898906 771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "288e543bb279410c91c01dd1623e110f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:48.899082 771 raft_consensus.cc:2468] T 288e543bb279410c91c01dd1623e110f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:48.899493 847 leader_election.cc:304] T 288e543bb279410c91c01dd1623e110f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:48.899931 2686 raft_consensus.cc:2804] T 288e543bb279410c91c01dd1623e110f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:48.900002 2686 raft_consensus.cc:493] T 288e543bb279410c91c01dd1623e110f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:48.900063 2686 raft_consensus.cc:3060] T 288e543bb279410c91c01dd1623e110f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.900697 1166 tablet_bootstrap.cc:492] T 67cf96a7dd3f42e19b86746c87e689f6 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:48.900822 1166 ts_tablet_manager.cc:1403] T 67cf96a7dd3f42e19b86746c87e689f6 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.003s
I20251025 07:52:48.901417 1166 raft_consensus.cc:359] T 67cf96a7dd3f42e19b86746c87e689f6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.901556 1166 raft_consensus.cc:385] T 67cf96a7dd3f42e19b86746c87e689f6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.901597 1166 raft_consensus.cc:740] T 67cf96a7dd3f42e19b86746c87e689f6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.898056 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 202) took 7422 ms. Trace:
I20251025 07:52:48.902861 2169 rpcz_store.cc:276] 1025 07:52:41.475373 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.475673 (+ 300us) service_pool.cc:225] Handling call
1025 07:52:48.897969 (+7422296us) ts_tablet_manager.cc:1151] Deleting tablet 1d261e432cf5403ea162129706bc4d4d
1025 07:52:48.897980 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.898040 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7422097}
I20251025 07:52:48.903149 1166 consensus_queue.cc:260] T 67cf96a7dd3f42e19b86746c87e689f6 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:48.903674 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 1d261e432cf5403ea162129706bc4d4d because the tablet was not found. No further retry: Not found: Tablet not found: 1d261e432cf5403ea162129706bc4d4d
I20251025 07:52:48.904000 1166 ts_tablet_manager.cc:1434] T 67cf96a7dd3f42e19b86746c87e689f6 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251025 07:52:48.902742 2686 raft_consensus.cc:515] T 288e543bb279410c91c01dd1623e110f P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.905011 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "288e543bb279410c91c01dd1623e110f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:48.903234 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 203) took 7427 ms. Trace:
W20251025 07:52:48.905467 850 leader_election.cc:343] T 288e543bb279410c91c01dd1623e110f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.905854 2686 leader_election.cc:290] T 288e543bb279410c91c01dd1623e110f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:48.906008 1166 tablet_bootstrap.cc:492] T df0d86ad301245c1a6bcb44805936d25 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:48.905480 2169 rpcz_store.cc:276] 1025 07:52:41.476076 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.476523 (+ 447us) service_pool.cc:225] Handling call
1025 07:52:48.903147 (+7426624us) ts_tablet_manager.cc:1151] Deleting tablet 467324e2432f4caca148951329caa292
1025 07:52:48.903157 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.903219 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7426427}
I20251025 07:52:48.906742 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 204) took 7428 ms. Trace:
I20251025 07:52:48.907106 771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "288e543bb279410c91c01dd1623e110f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:48.907264 771 raft_consensus.cc:3060] T 288e543bb279410c91c01dd1623e110f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:48.907436 1166 tablet_bootstrap.cc:654] T df0d86ad301245c1a6bcb44805936d25 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.909837 2169 rpcz_store.cc:276] 1025 07:52:41.478515 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.478835 (+ 320us) service_pool.cc:225] Handling call
1025 07:52:48.906661 (+7427826us) ts_tablet_manager.cc:1151] Deleting tablet 8c76d1b509f04ab1a10b83cff675c11e
1025 07:52:48.906671 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.906727 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7427652}
I20251025 07:52:48.910208 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 205) took 7430 ms. Trace:
I20251025 07:52:48.910282 2169 rpcz_store.cc:276] 1025 07:52:41.479260 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.479779 (+ 519us) service_pool.cc:225] Handling call
1025 07:52:48.910099 (+7430320us) ts_tablet_manager.cc:1151] Deleting tablet b772025852794a49b072f37cccbd3291
1025 07:52:48.910110 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.910192 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7430173}
W20251025 07:52:48.910811 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 467324e2432f4caca148951329caa292 because the tablet was not found. No further retry: Not found: Tablet not found: 467324e2432f4caca148951329caa292
I20251025 07:52:48.910845 2146 ts_tablet_manager.cc:1929] T 1912e59548874e0e8e989050ec0064c5 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.910946 2146 log.cc:1199] T 1912e59548874e0e8e989050ec0064c5 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/1912e59548874e0e8e989050ec0064c5
W20251025 07:52:48.911123 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 8c76d1b509f04ab1a10b83cff675c11e because the tablet was not found. No further retry: Not found: Tablet not found: 8c76d1b509f04ab1a10b83cff675c11e
I20251025 07:52:48.911295 2146 ts_tablet_manager.cc:1950] T 1912e59548874e0e8e989050ec0064c5 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:48.911744 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 206) took 7431 ms. Trace:
I20251025 07:52:48.912006 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 211) took 7472 ms. Trace:
I20251025 07:52:48.912087 2146 rpcz_store.cc:276] 1025 07:52:41.439068 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.439235 (+ 167us) service_pool.cc:225] Handling call
1025 07:52:48.819942 (+7380707us) ts_tablet_manager.cc:1151] Deleting tablet 1912e59548874e0e8e989050ec0064c5
1025 07:52:48.819951 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.897719 (+ 77768us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.910736 (+ 13017us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.911981 (+ 1245us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":26496,"tablet-delete.queue_time_us":7380516}
I20251025 07:52:48.912541 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 1912e59548874e0e8e989050ec0064c5 (table default.Table_8d39f75f88854f528f5cac09c3b0423a [id=74ec1fa4786e40109252885ea225ed66]) successfully deleted
W20251025 07:52:48.912608 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet b772025852794a49b072f37cccbd3291 because the tablet was not found. No further retry: Not found: Tablet not found: b772025852794a49b072f37cccbd3291
I20251025 07:52:48.912156 2169 rpcz_store.cc:276] 1025 07:52:41.480115 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.480444 (+ 329us) service_pool.cc:225] Handling call
1025 07:52:48.911671 (+7431227us) ts_tablet_manager.cc:1151] Deleting tablet a6ddefa1738f4ff98586900c344359d2
1025 07:52:48.911680 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.911730 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7431091}
I20251025 07:52:48.914592 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 209) took 7405 ms. Trace:
I20251025 07:52:48.914733 2169 rpcz_store.cc:276] 1025 07:52:41.509575 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.509851 (+ 276us) service_pool.cc:225] Handling call
1025 07:52:48.914515 (+7404664us) ts_tablet_manager.cc:1151] Deleting tablet fa77085e81c7440d9e255c44d1e80c26
1025 07:52:48.914524 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.914577 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7404487}
I20251025 07:52:48.915181 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 210) took 7404 ms. Trace:
I20251025 07:52:48.915308 2169 rpcz_store.cc:276] 1025 07:52:41.510772 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.510966 (+ 194us) service_pool.cc:225] Handling call
1025 07:52:48.915109 (+7404143us) ts_tablet_manager.cc:1151] Deleting tablet 0b21b7da1e2f47829c3f6cdaee689e8a
1025 07:52:48.915117 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.915167 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7403994}
W20251025 07:52:48.915722 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet a6ddefa1738f4ff98586900c344359d2 because the tablet was not found. No further retry: Not found: Tablet not found: a6ddefa1738f4ff98586900c344359d2
I20251025 07:52:48.915788 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 211) took 7404 ms. Trace:
I20251025 07:52:48.915845 2169 rpcz_store.cc:276] 1025 07:52:41.511350 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.512315 (+ 965us) service_pool.cc:225] Handling call
1025 07:52:48.915685 (+7403370us) ts_tablet_manager.cc:1151] Deleting tablet fd265cdbf6a8440393a2b5d8240d1872
1025 07:52:48.915695 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.915773 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7403176}
W20251025 07:52:48.915972 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet fa77085e81c7440d9e255c44d1e80c26 because the tablet was not found. No further retry: Not found: Tablet not found: fa77085e81c7440d9e255c44d1e80c26
W20251025 07:52:48.916186 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 0b21b7da1e2f47829c3f6cdaee689e8a because the tablet was not found. No further retry: Not found: Tablet not found: 0b21b7da1e2f47829c3f6cdaee689e8a
I20251025 07:52:48.916199 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 212) took 7403 ms. Trace:
I20251025 07:52:48.916257 2169 rpcz_store.cc:276] 1025 07:52:41.513208 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.514051 (+ 843us) service_pool.cc:225] Handling call
1025 07:52:48.916128 (+7402077us) ts_tablet_manager.cc:1151] Deleting tablet 887e7c7a5d0e4aaf88f26246306df19c
1025 07:52:48.916136 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.916186 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7401878}
W20251025 07:52:48.916395 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet fd265cdbf6a8440393a2b5d8240d1872 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd265cdbf6a8440393a2b5d8240d1872 already in progress: creating tablet
I20251025 07:52:48.916654 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 213) took 7402 ms. Trace:
W20251025 07:52:48.916672 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 887e7c7a5d0e4aaf88f26246306df19c because the tablet was not found. No further retry: Not found: Tablet not found: 887e7c7a5d0e4aaf88f26246306df19c
I20251025 07:52:48.916724 2169 rpcz_store.cc:276] 1025 07:52:41.514440 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.515439 (+ 999us) service_pool.cc:225] Handling call
1025 07:52:48.916559 (+7401120us) ts_tablet_manager.cc:1151] Deleting tablet c11f674385184ae6b235f1db8bf1fb91
1025 07:52:48.916568 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.916636 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7400959}
W20251025 07:52:48.917109 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet c11f674385184ae6b235f1db8bf1fb91 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c11f674385184ae6b235f1db8bf1fb91 already in progress: creating tablet
I20251025 07:52:48.910089 771 raft_consensus.cc:2468] T 288e543bb279410c91c01dd1623e110f P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:48.917866 847 leader_election.cc:304] T 288e543bb279410c91c01dd1623e110f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:48.918182 2886 raft_consensus.cc:2804] T 288e543bb279410c91c01dd1623e110f P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:48.918332 2886 raft_consensus.cc:697] T 288e543bb279410c91c01dd1623e110f P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:48.918577 2886 consensus_queue.cc:237] T 288e543bb279410c91c01dd1623e110f P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.919422 1166 tablet_bootstrap.cc:492] T df0d86ad301245c1a6bcb44805936d25 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:48.919544 1166 ts_tablet_manager.cc:1403] T df0d86ad301245c1a6bcb44805936d25 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20251025 07:52:48.920279 1166 raft_consensus.cc:359] T df0d86ad301245c1a6bcb44805936d25 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.920455 1166 raft_consensus.cc:385] T df0d86ad301245c1a6bcb44805936d25 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.920511 1166 raft_consensus.cc:740] T df0d86ad301245c1a6bcb44805936d25 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.921597 2146 tablet_replica.cc:333] T 1b5ac7e845a24b2eb5b2429d47361232 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:48.921793 2146 raft_consensus.cc:2243] T 1b5ac7e845a24b2eb5b2429d47361232 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:48.922067 2146 raft_consensus.cc:2272] T 1b5ac7e845a24b2eb5b2429d47361232 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.922427 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 214) took 7406 ms. Trace:
I20251025 07:52:48.922601 2169 rpcz_store.cc:276] 1025 07:52:41.515795 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.515959 (+ 164us) service_pool.cc:225] Handling call
1025 07:52:48.922278 (+7406319us) ts_tablet_manager.cc:1151] Deleting tablet d77704b4fe6c4456a2ba40ffec0c60fc
1025 07:52:48.922292 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.922408 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7406125}
W20251025 07:52:48.923113 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet d77704b4fe6c4456a2ba40ffec0c60fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d77704b4fe6c4456a2ba40ffec0c60fc already in progress: creating tablet
I20251025 07:52:48.923162 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 215) took 7406 ms. Trace:
I20251025 07:52:48.923226 2169 rpcz_store.cc:276] 1025 07:52:41.516341 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.516630 (+ 289us) service_pool.cc:225] Handling call
1025 07:52:48.923061 (+7406431us) ts_tablet_manager.cc:1151] Deleting tablet e4da89f863cc43f3bf6df908bb9dc6bf
1025 07:52:48.923070 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.923147 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7406237}
I20251025 07:52:48.923621 2146 ts_tablet_manager.cc:1916] T 1b5ac7e845a24b2eb5b2429d47361232 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:48.923638 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet e4da89f863cc43f3bf6df908bb9dc6bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4da89f863cc43f3bf6df908bb9dc6bf already in progress: creating tablet
I20251025 07:52:48.926225 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 216) took 7404 ms. Trace:
I20251025 07:52:48.926375 2166 tablet_replica.cc:333] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:48.926344 2169 rpcz_store.cc:276] 1025 07:52:41.519536 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.519580 (+ 44us) service_pool.cc:225] Handling call
1025 07:52:48.923618 (+7404038us) ts_tablet_manager.cc:1151] Deleting tablet 4037dee385d04e85bb78c8e77cb00379
1025 07:52:48.923627 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.923676 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7403889}
I20251025 07:52:48.927269 2166 raft_consensus.cc:2243] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:48.926899 1169 tablet_bootstrap.cc:492] T 2b594d4718344aafbdaec34040695589 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:48.927441 2166 raft_consensus.cc:2272] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.927529 1169 ts_tablet_manager.cc:1403] T 2b594d4718344aafbdaec34040695589 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.001s
W20251025 07:52:48.927704 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 4037dee385d04e85bb78c8e77cb00379 because the tablet was not found. No further retry: Not found: Tablet not found: 4037dee385d04e85bb78c8e77cb00379
I20251025 07:52:48.927801 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 217) took 7407 ms. Trace:
I20251025 07:52:48.927873 2169 rpcz_store.cc:276] 1025 07:52:41.519847 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.519890 (+ 43us) service_pool.cc:225] Handling call
1025 07:52:48.927702 (+7407812us) ts_tablet_manager.cc:1151] Deleting tablet e47c4f741f354a19b49777bd06e54764
1025 07:52:48.927710 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.927785 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7407672}
W20251025 07:52:48.928216 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet e47c4f741f354a19b49777bd06e54764 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e47c4f741f354a19b49777bd06e54764 already in progress: creating tablet
I20251025 07:52:48.928153 1169 raft_consensus.cc:359] T 2b594d4718344aafbdaec34040695589 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.928300 1169 raft_consensus.cc:385] T 2b594d4718344aafbdaec34040695589 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.928351 1169 raft_consensus.cc:740] T 2b594d4718344aafbdaec34040695589 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.928566 1169 consensus_queue.cc:260] T 2b594d4718344aafbdaec34040695589 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.929490 2166 ts_tablet_manager.cc:1916] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.929951 1169 ts_tablet_manager.cc:1434] T 2b594d4718344aafbdaec34040695589 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:48.930397 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 218) took 7410 ms. Trace:
I20251025 07:52:48.930495 2169 rpcz_store.cc:276] 1025 07:52:41.520165 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.520442 (+ 277us) service_pool.cc:225] Handling call
1025 07:52:48.930265 (+7409823us) ts_tablet_manager.cc:1151] Deleting tablet 72dc084f8efc45478771c247e07a16f4
1025 07:52:48.930276 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.930368 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7409664}
W20251025 07:52:48.930891 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 72dc084f8efc45478771c247e07a16f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72dc084f8efc45478771c247e07a16f4 already in progress: creating tablet
I20251025 07:52:48.930958 1169 tablet_bootstrap.cc:492] T dff7f86ae8924685b26011e504e2247b P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:48.932291 1169 tablet_bootstrap.cc:654] T dff7f86ae8924685b26011e504e2247b P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.934603 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 219) took 7413 ms. Trace:
I20251025 07:52:48.934684 2169 rpcz_store.cc:276] 1025 07:52:41.520754 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.521130 (+ 376us) service_pool.cc:225] Handling call
1025 07:52:48.934526 (+7413396us) ts_tablet_manager.cc:1151] Deleting tablet be5ab0db6ac34c2c9a370445e251eb46
1025 07:52:48.934534 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.934589 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7413156}
W20251025 07:52:48.935117 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet be5ab0db6ac34c2c9a370445e251eb46 because the tablet was not found. No further retry: Not found: Tablet not found: be5ab0db6ac34c2c9a370445e251eb46
I20251025 07:52:48.935184 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 220) took 7413 ms. Trace:
I20251025 07:52:48.935261 2169 rpcz_store.cc:276] 1025 07:52:41.521554 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.529249 (+ 7695us) service_pool.cc:225] Handling call
1025 07:52:48.935089 (+7405840us) ts_tablet_manager.cc:1151] Deleting tablet 4f7c15529be54a11961f385a953b0b03
1025 07:52:48.935098 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.935170 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7405674}
W20251025 07:52:48.935619 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 4f7c15529be54a11961f385a953b0b03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f7c15529be54a11961f385a953b0b03 already in progress: creating tablet
I20251025 07:52:48.935665 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 221) took 7406 ms. Trace:
I20251025 07:52:48.935729 2169 rpcz_store.cc:276] 1025 07:52:41.529582 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.529624 (+ 42us) service_pool.cc:225] Handling call
1025 07:52:48.935589 (+7405965us) ts_tablet_manager.cc:1151] Deleting tablet 7f29b64704e147aa9b0b5f6cda860f09
1025 07:52:48.935599 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.935650 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7405704}
W20251025 07:52:48.936046 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 7f29b64704e147aa9b0b5f6cda860f09 because the tablet was not found. No further retry: Not found: Tablet not found: 7f29b64704e147aa9b0b5f6cda860f09
I20251025 07:52:48.927386 1166 consensus_queue.cc:260] T df0d86ad301245c1a6bcb44805936d25 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.941871 1166 ts_tablet_manager.cc:1434] T df0d86ad301245c1a6bcb44805936d25 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20251025 07:52:48.942195 1166 tablet_bootstrap.cc:492] T a6ae1315b28940d49b667e3c80347021 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:48.942325 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 223) took 7412 ms. Trace:
I20251025 07:52:48.942411 2169 rpcz_store.cc:276] 1025 07:52:41.530349 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.530391 (+ 42us) service_pool.cc:225] Handling call
1025 07:52:48.942243 (+7411852us) ts_tablet_manager.cc:1151] Deleting tablet abf94c32ff2945119b710905c075d63c
1025 07:52:48.942253 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.942309 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7411519}
I20251025 07:52:48.943513 1166 tablet_bootstrap.cc:654] T a6ae1315b28940d49b667e3c80347021 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.944762 1169 tablet_bootstrap.cc:492] T dff7f86ae8924685b26011e504e2247b P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:48.944886 1169 ts_tablet_manager.cc:1403] T dff7f86ae8924685b26011e504e2247b P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.002s
I20251025 07:52:48.945240 2888 raft_consensus.cc:493] T 48b73e0c0dce4be686d05f991a21bd19 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:48.945361 2888 raft_consensus.cc:515] T 48b73e0c0dce4be686d05f991a21bd19 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:48.945499 1169 raft_consensus.cc:359] T dff7f86ae8924685b26011e504e2247b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.945645 1169 raft_consensus.cc:385] T dff7f86ae8924685b26011e504e2247b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.945688 1169 raft_consensus.cc:740] T dff7f86ae8924685b26011e504e2247b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.945880 2888 leader_election.cc:290] T 48b73e0c0dce4be686d05f991a21bd19 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:48.945856 1169 consensus_queue.cc:260] T dff7f86ae8924685b26011e504e2247b P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.946122 2169 tablet_replica.cc:333] T 1912e59548874e0e8e989050ec0064c5 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:48.946331 1169 ts_tablet_manager.cc:1434] T dff7f86ae8924685b26011e504e2247b P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251025 07:52:48.946740 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "48b73e0c0dce4be686d05f991a21bd19" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:48.947149 712 leader_election.cc:343] T 48b73e0c0dce4be686d05f991a21bd19 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.947418 2169 raft_consensus.cc:2243] T 1912e59548874e0e8e989050ec0064c5 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:48.947607 2169 raft_consensus.cc:2272] T 1912e59548874e0e8e989050ec0064c5 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.948151 2169 ts_tablet_manager.cc:1916] T 1912e59548874e0e8e989050ec0064c5 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:48.949437 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "48b73e0c0dce4be686d05f991a21bd19" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:48.950263 1169 tablet_bootstrap.cc:492] T fd265cdbf6a8440393a2b5d8240d1872 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
W20251025 07:52:48.951658 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet abf94c32ff2945119b710905c075d63c because the tablet was not found. No further retry: Not found: Tablet not found: abf94c32ff2945119b710905c075d63c
I20251025 07:52:48.951895 1169 tablet_bootstrap.cc:654] T fd265cdbf6a8440393a2b5d8240d1872 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:48.952451 710 leader_election.cc:343] T 48b73e0c0dce4be686d05f991a21bd19 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:48.952569 710 leader_election.cc:304] T 48b73e0c0dce4be686d05f991a21bd19 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:48.952986 2888 raft_consensus.cc:2749] T 48b73e0c0dce4be686d05f991a21bd19 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:48.954456 1166 tablet_bootstrap.cc:492] T a6ae1315b28940d49b667e3c80347021 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:48.954574 1166 ts_tablet_manager.cc:1403] T a6ae1315b28940d49b667e3c80347021 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20251025 07:52:48.955158 1166 raft_consensus.cc:359] T a6ae1315b28940d49b667e3c80347021 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.955307 1166 raft_consensus.cc:385] T a6ae1315b28940d49b667e3c80347021 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.955353 1166 raft_consensus.cc:740] T a6ae1315b28940d49b667e3c80347021 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.957134 1166 consensus_queue.cc:260] T a6ae1315b28940d49b667e3c80347021 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.959281 1166 ts_tablet_manager.cc:1434] T a6ae1315b28940d49b667e3c80347021 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251025 07:52:48.959573 1166 tablet_bootstrap.cc:492] T 062c0f3e51a3487880cea373f8cf1f54 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:48.961076 1166 tablet_bootstrap.cc:654] T 062c0f3e51a3487880cea373f8cf1f54 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.963975 2146 ts_tablet_manager.cc:1929] T 1b5ac7e845a24b2eb5b2429d47361232 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.964076 2146 log.cc:1199] T 1b5ac7e845a24b2eb5b2429d47361232 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/1b5ac7e845a24b2eb5b2429d47361232
W20251025 07:52:48.964043 983 consensus_peers.cc:597] T 4f0c4f0e8f014e59be9e46d2afe0be9d P db42be7ad3c741a0aa427f27e245993b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f0c4f0e8f014e59be9e46d2afe0be9d. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:48.964401 2146 ts_tablet_manager.cc:1950] T 1b5ac7e845a24b2eb5b2429d47361232 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:48.965029 1167 tablet_bootstrap.cc:492] T 0582ee42ab564c07ae3facfc5935617c P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:48.965097 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 212) took 7525 ms. Trace:
I20251025 07:52:48.965186 2146 rpcz_store.cc:276] 1025 07:52:41.439596 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.439864 (+ 268us) service_pool.cc:225] Handling call
1025 07:52:48.921550 (+7481686us) ts_tablet_manager.cc:1151] Deleting tablet 1b5ac7e845a24b2eb5b2429d47361232
1025 07:52:48.921559 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.947137 (+ 25578us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.963873 (+ 16736us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.965081 (+ 1208us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7481526}
I20251025 07:52:48.965559 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 213) took 7507 ms. Trace:
I20251025 07:52:48.965628 2146 rpcz_store.cc:276] 1025 07:52:41.458256 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.458458 (+ 202us) service_pool.cc:225] Handling call
1025 07:52:48.965476 (+7507018us) ts_tablet_manager.cc:1151] Deleting tablet db99a1e68daf4e948a3d66ad276cc5f9
1025 07:52:48.965485 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.965546 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7506746}
I20251025 07:52:48.965729 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 1b5ac7e845a24b2eb5b2429d47361232 (table default.Table_f6781cf6e9104f51a278cab3d1781a1a [id=8b4207028ba74a5ebd5d4f6690f46893]) successfully deleted
I20251025 07:52:48.965860 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 215) took 7499 ms. Trace:
I20251025 07:52:48.965914 2146 rpcz_store.cc:276] 1025 07:52:41.466308 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.466626 (+ 318us) service_pool.cc:225] Handling call
1025 07:52:48.965803 (+7499177us) ts_tablet_manager.cc:1151] Deleting tablet 8c76d1b509f04ab1a10b83cff675c11e
1025 07:52:48.965810 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.965850 (+ 40us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7498988}
W20251025 07:52:48.966526 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet db99a1e68daf4e948a3d66ad276cc5f9 because the tablet was not found. No further retry: Not found: Tablet not found: db99a1e68daf4e948a3d66ad276cc5f9
I20251025 07:52:48.966712 1169 tablet_bootstrap.cc:492] T fd265cdbf6a8440393a2b5d8240d1872 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
W20251025 07:52:48.966780 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 8c76d1b509f04ab1a10b83cff675c11e because the tablet was not found. No further retry: Not found: Tablet not found: 8c76d1b509f04ab1a10b83cff675c11e
I20251025 07:52:48.966840 1169 ts_tablet_manager.cc:1403] T fd265cdbf6a8440393a2b5d8240d1872 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20251025 07:52:48.967069 1167 ts_tablet_manager.cc:1403] T 0582ee42ab564c07ae3facfc5935617c P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.094s user 0.003s sys 0.007s
I20251025 07:52:48.967445 1169 raft_consensus.cc:359] T fd265cdbf6a8440393a2b5d8240d1872 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:48.967608 1169 raft_consensus.cc:385] T fd265cdbf6a8440393a2b5d8240d1872 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.967664 1169 raft_consensus.cc:740] T fd265cdbf6a8440393a2b5d8240d1872 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.967736 2169 ts_tablet_manager.cc:1929] T 1912e59548874e0e8e989050ec0064c5 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:48.967851 2169 log.cc:1199] T 1912e59548874e0e8e989050ec0064c5 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/1912e59548874e0e8e989050ec0064c5
I20251025 07:52:48.967880 1169 consensus_queue.cc:260] T fd265cdbf6a8440393a2b5d8240d1872 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:48.968212 2169 ts_tablet_manager.cc:1950] T 1912e59548874e0e8e989050ec0064c5 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:48.968971 1169 ts_tablet_manager.cc:1434] T fd265cdbf6a8440393a2b5d8240d1872 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:48.969214 1169 tablet_bootstrap.cc:492] T 1aeba18be0cb4eb9949aeaf6ad73a812 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:48.970530 1169 tablet_bootstrap.cc:654] T 1aeba18be0cb4eb9949aeaf6ad73a812 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.971966 1166 tablet_bootstrap.cc:492] T 062c0f3e51a3487880cea373f8cf1f54 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:48.972074 1166 ts_tablet_manager.cc:1403] T 062c0f3e51a3487880cea373f8cf1f54 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20251025 07:52:48.972741 1166 raft_consensus.cc:359] T 062c0f3e51a3487880cea373f8cf1f54 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.972903 1166 raft_consensus.cc:385] T 062c0f3e51a3487880cea373f8cf1f54 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.972956 1166 raft_consensus.cc:740] T 062c0f3e51a3487880cea373f8cf1f54 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.973116 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 216) took 7506 ms. Trace:
I20251025 07:52:48.973178 2146 rpcz_store.cc:276] 1025 07:52:41.466942 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.467092 (+ 150us) service_pool.cc:225] Handling call
1025 07:52:48.966070 (+7498978us) ts_tablet_manager.cc:1151] Deleting tablet b772025852794a49b072f37cccbd3291
1025 07:52:48.966080 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.973102 (+ 7022us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7498802}
I20251025 07:52:48.973670 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 217) took 7506 ms. Trace:
W20251025 07:52:48.973742 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet b772025852794a49b072f37cccbd3291 because the tablet was not found. No further retry: Not found: Tablet not found: b772025852794a49b072f37cccbd3291
I20251025 07:52:48.973809 2146 rpcz_store.cc:276] 1025 07:52:41.467381 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.467521 (+ 140us) service_pool.cc:225] Handling call
1025 07:52:48.973579 (+7506058us) ts_tablet_manager.cc:1151] Deleting tablet fa77085e81c7440d9e255c44d1e80c26
1025 07:52:48.973588 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.973655 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7505927}
W20251025 07:52:48.974440 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet fa77085e81c7440d9e255c44d1e80c26 because the tablet was not found. No further retry: Not found: Tablet not found: fa77085e81c7440d9e255c44d1e80c26
I20251025 07:52:48.974786 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 218) took 7507 ms. Trace:
I20251025 07:52:48.974820 1167 raft_consensus.cc:359] T 0582ee42ab564c07ae3facfc5935617c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.974995 1167 raft_consensus.cc:385] T 0582ee42ab564c07ae3facfc5935617c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.975055 1167 raft_consensus.cc:740] T 0582ee42ab564c07ae3facfc5935617c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.975276 1166 consensus_queue.cc:260] T 062c0f3e51a3487880cea373f8cf1f54 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:48.975884 1166 ts_tablet_manager.cc:1434] T 062c0f3e51a3487880cea373f8cf1f54 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251025 07:52:48.976188 1166 tablet_bootstrap.cc:492] T 22fc90c704f64d2f9f4b8c54dee45576 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:48.975041 2146 rpcz_store.cc:276] 1025 07:52:41.467751 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.467887 (+ 136us) service_pool.cc:225] Handling call
1025 07:52:48.974713 (+7506826us) ts_tablet_manager.cc:1151] Deleting tablet a6ddefa1738f4ff98586900c344359d2
1025 07:52:48.974722 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.974772 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7506654}
I20251025 07:52:48.978263 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 219) took 7510 ms. Trace:
I20251025 07:52:48.978410 2146 rpcz_store.cc:276] 1025 07:52:41.468189 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.468346 (+ 157us) service_pool.cc:225] Handling call
1025 07:52:48.978188 (+7509842us) ts_tablet_manager.cc:1151] Deleting tablet 0b21b7da1e2f47829c3f6cdaee689e8a
1025 07:52:48.978195 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.978249 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7509706}
I20251025 07:52:48.978857 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 220) took 7510 ms. Trace:
I20251025 07:52:48.978974 2146 rpcz_store.cc:276] 1025 07:52:41.468613 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.468782 (+ 169us) service_pool.cc:225] Handling call
1025 07:52:48.978789 (+7510007us) ts_tablet_manager.cc:1151] Deleting tablet f0bdaa8cfd764f1b83f47881353d26b6
1025 07:52:48.978797 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.978846 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7509837}
I20251025 07:52:48.979429 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 221) took 7510 ms. Trace:
I20251025 07:52:48.979547 2146 rpcz_store.cc:276] 1025 07:52:41.469114 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.469318 (+ 204us) service_pool.cc:225] Handling call
1025 07:52:48.979347 (+7510029us) ts_tablet_manager.cc:1151] Deleting tablet 8956ad28b35d45f28c683ac055bba441
1025 07:52:48.979355 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.979415 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7509856}
I20251025 07:52:48.980479 1166 tablet_bootstrap.cc:654] T 22fc90c704f64d2f9f4b8c54dee45576 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.982262 1169 tablet_bootstrap.cc:492] T 1aeba18be0cb4eb9949aeaf6ad73a812 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
W20251025 07:52:48.982349 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet a6ddefa1738f4ff98586900c344359d2 because the tablet was not found. No further retry: Not found: Tablet not found: a6ddefa1738f4ff98586900c344359d2
I20251025 07:52:48.982508 1169 ts_tablet_manager.cc:1403] T 1aeba18be0cb4eb9949aeaf6ad73a812 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
W20251025 07:52:48.982609 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 0b21b7da1e2f47829c3f6cdaee689e8a because the tablet was not found. No further retry: Not found: Tablet not found: 0b21b7da1e2f47829c3f6cdaee689e8a
W20251025 07:52:48.982854 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet f0bdaa8cfd764f1b83f47881353d26b6 because the tablet was not found. No further retry: Not found: Tablet not found: f0bdaa8cfd764f1b83f47881353d26b6
W20251025 07:52:48.983074 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 8956ad28b35d45f28c683ac055bba441 because the tablet was not found. No further retry: Not found: Tablet not found: 8956ad28b35d45f28c683ac055bba441
I20251025 07:52:48.983110 2686 consensus_queue.cc:1048] T c501ef5d70e742598c1a902efe47feb1 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:48.983520 1169 raft_consensus.cc:359] T 1aeba18be0cb4eb9949aeaf6ad73a812 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.983682 1169 raft_consensus.cc:385] T 1aeba18be0cb4eb9949aeaf6ad73a812 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:48.983728 1169 raft_consensus.cc:740] T 1aeba18be0cb4eb9949aeaf6ad73a812 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:48.983927 1169 consensus_queue.cc:260] T 1aeba18be0cb4eb9949aeaf6ad73a812 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:48.987232 1169 ts_tablet_manager.cc:1434] T 1aeba18be0cb4eb9949aeaf6ad73a812 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20251025 07:52:48.987839 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 224) took 7457 ms. Trace:
I20251025 07:52:48.988111 1169 tablet_bootstrap.cc:492] T c501ef5d70e742598c1a902efe47feb1 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:48.988214 2169 rpcz_store.cc:276] 1025 07:52:41.530603 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.530900 (+ 297us) service_pool.cc:225] Handling call
1025 07:52:48.946074 (+7415174us) ts_tablet_manager.cc:1151] Deleting tablet 1912e59548874e0e8e989050ec0064c5
1025 07:52:48.946085 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.955298 (+ 9213us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.967617 (+ 12319us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:48.987818 (+ 20201us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6687,"tablet-delete.queue_time_us":7415032}
I20251025 07:52:48.989104 2169 tablet_replica.cc:333] T 2b594d4718344aafbdaec34040695589 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:48.993497 2169 raft_consensus.cc:2243] T 2b594d4718344aafbdaec34040695589 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:48.993593 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 1912e59548874e0e8e989050ec0064c5 (table default.Table_8d39f75f88854f528f5cac09c3b0423a [id=74ec1fa4786e40109252885ea225ed66]) successfully deleted
I20251025 07:52:48.994022 2169 raft_consensus.cc:2272] T 2b594d4718344aafbdaec34040695589 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:48.989650 1169 tablet_bootstrap.cc:654] T c501ef5d70e742598c1a902efe47feb1 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:48.997524 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 222) took 7527 ms. Trace:
I20251025 07:52:48.997668 2146 rpcz_store.cc:276] 1025 07:52:41.469629 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.469821 (+ 192us) service_pool.cc:225] Handling call
1025 07:52:48.979902 (+7510081us) ts_tablet_manager.cc:1151] Deleting tablet cc407c7eb12e4eadb41fb21385611c94
1025 07:52:48.997388 (+ 17486us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.997503 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7507011}
I20251025 07:52:48.997917 1167 consensus_queue.cc:260] T 0582ee42ab564c07ae3facfc5935617c P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:48.998217 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet cc407c7eb12e4eadb41fb21385611c94 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc407c7eb12e4eadb41fb21385611c94 already in progress: creating tablet
I20251025 07:52:48.998550 2146 tablet_replica.cc:333] T 716fb61312e94a4d9eb8ab8336ca140c P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:48.998797 2146 raft_consensus.cc:2243] T 716fb61312e94a4d9eb8ab8336ca140c P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:48.999897 2146 raft_consensus.cc:2272] T 716fb61312e94a4d9eb8ab8336ca140c P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:49.000155 1167 ts_tablet_manager.cc:1434] T 0582ee42ab564c07ae3facfc5935617c P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.032s user 0.001s sys 0.001s
I20251025 07:52:49.000159 2991 consensus_queue.cc:1048] T fa826b85f0c94150a17d4e417dd3d62f P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:49.000864 2146 ts_tablet_manager.cc:1916] T 716fb61312e94a4d9eb8ab8336ca140c P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:49.003314 1167 tablet_bootstrap.cc:492] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:49.006115 3011 raft_consensus.cc:493] T 716fb61312e94a4d9eb8ab8336ca140c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.006304 3011 raft_consensus.cc:515] T 716fb61312e94a4d9eb8ab8336ca140c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:49.006893 3011 leader_election.cc:290] T 716fb61312e94a4d9eb8ab8336ca140c P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:49.007488 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "716fb61312e94a4d9eb8ab8336ca140c" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:49.007917 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "716fb61312e94a4d9eb8ab8336ca140c" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:49.008705 2888 raft_consensus.cc:493] T 4f7c15529be54a11961f385a953b0b03 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.008849 2888 raft_consensus.cc:515] T 4f7c15529be54a11961f385a953b0b03 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.009202 2146 ts_tablet_manager.cc:1929] T 716fb61312e94a4d9eb8ab8336ca140c P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:49.009289 2146 log.cc:1199] T 716fb61312e94a4d9eb8ab8336ca140c P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/716fb61312e94a4d9eb8ab8336ca140c
I20251025 07:52:49.009502 2888 leader_election.cc:290] T 4f7c15529be54a11961f385a953b0b03 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:49.009620 2146 ts_tablet_manager.cc:1950] T 716fb61312e94a4d9eb8ab8336ca140c P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
W20251025 07:52:49.010212 983 leader_election.cc:343] T 716fb61312e94a4d9eb8ab8336ca140c P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: Tablet not RUNNING: STOPPED
I20251025 07:52:49.010228 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f7c15529be54a11961f385a953b0b03" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:49.010347 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 223) took 7540 ms. Trace:
I20251025 07:52:49.010419 2146 rpcz_store.cc:276] 1025 07:52:41.469747 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.511502 (+ 41755us) service_pool.cc:225] Handling call
1025 07:52:48.998502 (+7487000us) ts_tablet_manager.cc:1151] Deleting tablet 716fb61312e94a4d9eb8ab8336ca140c
1025 07:52:48.998513 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.004310 (+ 5797us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.009113 (+ 4803us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.010331 (+ 1218us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":720,"tablet-delete.queue_time_us":7486744}
W20251025 07:52:49.010694 712 leader_election.cc:343] T 4f7c15529be54a11961f385a953b0b03 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.010754 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 224) took 7498 ms. Trace:
I20251025 07:52:49.010814 2146 rpcz_store.cc:276] 1025 07:52:41.512013 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.513383 (+ 1370us) service_pool.cc:225] Handling call
1025 07:52:49.010678 (+7497295us) ts_tablet_manager.cc:1151] Deleting tablet bef0ef23d90d480d9c99aef21d7271fd
1025 07:52:49.010687 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.010742 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7497096}
I20251025 07:52:49.010950 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 716fb61312e94a4d9eb8ab8336ca140c (table default.Table_ba46ca0cf5d941e0879dead1c3544190 [id=3cf1d7f589ff457b9d47c88a1ccc3b2b]) successfully deleted
I20251025 07:52:49.011049 2146 tablet_replica.cc:333] T a7ab3150a35b4caca9119a1374d3f2f7 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:49.011236 2146 raft_consensus.cc:2243] T a7ab3150a35b4caca9119a1374d3f2f7 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:49.011433 2146 pending_rounds.cc:70] T a7ab3150a35b4caca9119a1374d3f2f7 P 3aea2869ff814cc7b142012faa274382: Trying to abort 1 pending ops.
I20251025 07:52:49.011466 2146 pending_rounds.cc:77] T a7ab3150a35b4caca9119a1374d3f2f7 P 3aea2869ff814cc7b142012faa274382: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214607437223612416 op_type: NO_OP noop_request { }
I20251025 07:52:49.011533 2146 raft_consensus.cc:2889] T a7ab3150a35b4caca9119a1374d3f2f7 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 07:52:49.011607 2146 raft_consensus.cc:2272] T a7ab3150a35b4caca9119a1374d3f2f7 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:49.011572 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet bef0ef23d90d480d9c99aef21d7271fd because the tablet was not found. No further retry: Not found: Tablet not found: bef0ef23d90d480d9c99aef21d7271fd
W20251025 07:52:49.012293 710 leader_election.cc:343] T 4f7c15529be54a11961f385a953b0b03 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 4f7c15529be54a11961f385a953b0b03
I20251025 07:52:49.012423 710 leader_election.cc:304] T 4f7c15529be54a11961f385a953b0b03 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.012869 2888 raft_consensus.cc:2749] T 4f7c15529be54a11961f385a953b0b03 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:49.013204 982 leader_election.cc:304] T 716fb61312e94a4d9eb8ab8336ca140c P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:49.013654 3011 raft_consensus.cc:2749] T 716fb61312e94a4d9eb8ab8336ca140c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:49.014545 2169 ts_tablet_manager.cc:1916] T 2b594d4718344aafbdaec34040695589 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:49.019285 1169 tablet_bootstrap.cc:492] T c501ef5d70e742598c1a902efe47feb1 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:49.019418 1169 ts_tablet_manager.cc:1403] T c501ef5d70e742598c1a902efe47feb1 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
W20251025 07:52:49.020247 850 consensus_peers.cc:597] T fa826b85f0c94150a17d4e417dd3d62f P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251025 07:52:49.020435 850 consensus_peers.cc:597] T c501ef5d70e742598c1a902efe47feb1 P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:49.020411 1169 raft_consensus.cc:359] T c501ef5d70e742598c1a902efe47feb1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:49.020560 1169 raft_consensus.cc:385] T c501ef5d70e742598c1a902efe47feb1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.020622 1169 raft_consensus.cc:740] T c501ef5d70e742598c1a902efe47feb1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.020805 1169 consensus_queue.cc:260] T c501ef5d70e742598c1a902efe47feb1 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:49.021275 1169 ts_tablet_manager.cc:1434] T c501ef5d70e742598c1a902efe47feb1 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:49.021535 1169 tablet_bootstrap.cc:492] T 1068f566477e4c6db7dde566fff4c2fb P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:49.022933 1169 tablet_bootstrap.cc:654] T 1068f566477e4c6db7dde566fff4c2fb P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.025936 1042 raft_consensus.cc:3060] T 9a8e2658a81e436298fe2190726080d4 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.026356 2146 ts_tablet_manager.cc:1916] T a7ab3150a35b4caca9119a1374d3f2f7 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:49.029269 2169 ts_tablet_manager.cc:1929] T 2b594d4718344aafbdaec34040695589 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:49.029367 2169 log.cc:1199] T 2b594d4718344aafbdaec34040695589 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/2b594d4718344aafbdaec34040695589
I20251025 07:52:49.029640 2169 ts_tablet_manager.cc:1950] T 2b594d4718344aafbdaec34040695589 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:49.031394 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 225) took 7500 ms. Trace:
I20251025 07:52:49.031558 1166 tablet_bootstrap.cc:492] T 22fc90c704f64d2f9f4b8c54dee45576 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:49.031495 2169 rpcz_store.cc:276] 1025 07:52:41.531157 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.531204 (+ 47us) service_pool.cc:225] Handling call
1025 07:52:48.989049 (+7457845us) ts_tablet_manager.cc:1151] Deleting tablet 2b594d4718344aafbdaec34040695589
1025 07:52:48.989063 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.025060 (+ 35997us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.029169 (+ 4109us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.031369 (+ 2200us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7457709}
I20251025 07:52:49.031694 1166 ts_tablet_manager.cc:1403] T 22fc90c704f64d2f9f4b8c54dee45576 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.056s user 0.009s sys 0.000s
I20251025 07:52:49.032027 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 2b594d4718344aafbdaec34040695589 (table default.Table_e08a3ff70f3a411e82b4bb292d768961 [id=8d075a6652b14a37a2bfbfddc6139eb9]) successfully deleted
I20251025 07:52:49.032334 1166 raft_consensus.cc:359] T 22fc90c704f64d2f9f4b8c54dee45576 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:49.032490 1166 raft_consensus.cc:385] T 22fc90c704f64d2f9f4b8c54dee45576 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.032539 1166 raft_consensus.cc:740] T 22fc90c704f64d2f9f4b8c54dee45576 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.032754 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 226) took 7501 ms. Trace:
I20251025 07:52:49.032704 1166 consensus_queue.cc:260] T 22fc90c704f64d2f9f4b8c54dee45576 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:49.032817 2169 rpcz_store.cc:276] 1025 07:52:41.531457 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.531497 (+ 40us) service_pool.cc:225] Handling call
1025 07:52:49.032631 (+7501134us) ts_tablet_manager.cc:1151] Deleting tablet ee1c8ea694284d80a30f9c05be4cbf49
1025 07:52:49.032644 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.032741 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7500744}
I20251025 07:52:49.033169 1166 ts_tablet_manager.cc:1434] T 22fc90c704f64d2f9f4b8c54dee45576 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251025 07:52:49.033402 1166 tablet_bootstrap.cc:492] T af743ee344e543a0aa6da3cd7a30e467 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:49.033615 2169 tablet_replica.cc:333] T 6dcab53a020847a1aa27930265ba0205 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:49.033789 2169 raft_consensus.cc:2243] T 6dcab53a020847a1aa27930265ba0205 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:49.034070 2169 raft_consensus.cc:2272] T 6dcab53a020847a1aa27930265ba0205 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:49.034566 2169 ts_tablet_manager.cc:1916] T 6dcab53a020847a1aa27930265ba0205 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:49.036231 850 consensus_peers.cc:597] T 645263a61b654f04985c40d04506c087 P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251025 07:52:49.036895 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet ee1c8ea694284d80a30f9c05be4cbf49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee1c8ea694284d80a30f9c05be4cbf49 already in progress: creating tablet
I20251025 07:52:49.039852 1166 tablet_bootstrap.cc:654] T af743ee344e543a0aa6da3cd7a30e467 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.041260 3011 raft_consensus.cc:493] T c501ef5d70e742598c1a902efe47feb1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.041411 3011 raft_consensus.cc:515] T c501ef5d70e742598c1a902efe47feb1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:49.042095 3011 leader_election.cc:290] T c501ef5d70e742598c1a902efe47feb1 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:49.042752 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c501ef5d70e742598c1a902efe47feb1" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:49.043720 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c501ef5d70e742598c1a902efe47feb1" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:49.044246 982 leader_election.cc:304] T c501ef5d70e742598c1a902efe47feb1 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:49.044590 2146 ts_tablet_manager.cc:1929] T a7ab3150a35b4caca9119a1374d3f2f7 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:49.044699 3011 raft_consensus.cc:2749] T c501ef5d70e742598c1a902efe47feb1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:49.045876 2169 ts_tablet_manager.cc:1929] T 6dcab53a020847a1aa27930265ba0205 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:49.045979 2169 log.cc:1199] T 6dcab53a020847a1aa27930265ba0205 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/6dcab53a020847a1aa27930265ba0205
I20251025 07:52:49.046402 2169 ts_tablet_manager.cc:1950] T 6dcab53a020847a1aa27930265ba0205 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:49.052201 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 227) took 7514 ms. Trace:
I20251025 07:52:49.052408 2146 log.cc:1199] T a7ab3150a35b4caca9119a1374d3f2f7 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/a7ab3150a35b4caca9119a1374d3f2f7
I20251025 07:52:49.052299 2169 rpcz_store.cc:276] 1025 07:52:41.537849 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.538071 (+ 222us) service_pool.cc:225] Handling call
1025 07:52:49.033570 (+7495499us) ts_tablet_manager.cc:1151] Deleting tablet 6dcab53a020847a1aa27930265ba0205
1025 07:52:49.033577 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.042528 (+ 8951us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.045797 (+ 3269us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.052187 (+ 6390us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7495295}
I20251025 07:52:49.052733 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 6dcab53a020847a1aa27930265ba0205 (table default.Table_9b7fc5a1f4944e749c90c1119d0b0054 [id=cb79dc5b6680450cadf2350095389c9e]) successfully deleted
I20251025 07:52:49.052954 2146 ts_tablet_manager.cc:1950] T a7ab3150a35b4caca9119a1374d3f2f7 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:49.053159 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 228) took 7514 ms. Trace:
I20251025 07:52:49.053244 2169 rpcz_store.cc:276] 1025 07:52:41.538965 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.539050 (+ 85us) service_pool.cc:225] Handling call
1025 07:52:49.053062 (+7514012us) ts_tablet_manager.cc:1151] Deleting tablet 4f25c9caa05b4da4bd56a8a21fde423c
1025 07:52:49.053073 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.053143 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7513843}
I20251025 07:52:49.054095 1166 tablet_bootstrap.cc:492] T af743ee344e543a0aa6da3cd7a30e467 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:49.054831 1166 ts_tablet_manager.cc:1403] T af743ee344e543a0aa6da3cd7a30e467 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.003s
W20251025 07:52:49.054831 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 4f25c9caa05b4da4bd56a8a21fde423c because the tablet was not found. No further retry: Not found: Tablet not found: 4f25c9caa05b4da4bd56a8a21fde423c
I20251025 07:52:49.055546 1166 raft_consensus.cc:359] T af743ee344e543a0aa6da3cd7a30e467 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.055677 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 225) took 7541 ms. Trace:
I20251025 07:52:49.055755 1166 raft_consensus.cc:385] T af743ee344e543a0aa6da3cd7a30e467 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.055804 1166 raft_consensus.cc:740] T af743ee344e543a0aa6da3cd7a30e467 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.055778 2146 rpcz_store.cc:276] 1025 07:52:41.513812 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.514672 (+ 860us) service_pool.cc:225] Handling call
1025 07:52:49.011008 (+7496336us) ts_tablet_manager.cc:1151] Deleting tablet a7ab3150a35b4caca9119a1374d3f2f7
1025 07:52:49.011016 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.029487 (+ 18471us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.044503 (+ 15016us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.055644 (+ 11141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7496143}
I20251025 07:52:49.056576 1166 consensus_queue.cc:260] T af743ee344e543a0aa6da3cd7a30e467 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.056806 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet a7ab3150a35b4caca9119a1374d3f2f7 (table default.Table_8129e15bf5c549bd853c04b227a7ef26 [id=e9e8547061ba4ac987d0aa7616c7f3ea]) successfully deleted
I20251025 07:52:49.057147 1166 ts_tablet_manager.cc:1434] T af743ee344e543a0aa6da3cd7a30e467 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:49.057173 1042 raft_consensus.cc:3060] T c187a40f8ff14c6a8c184fbeb030ad13 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.057174 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 226) took 7542 ms. Trace:
I20251025 07:52:49.057355 2146 rpcz_store.cc:276] 1025 07:52:41.515227 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.516941 (+ 1714us) service_pool.cc:225] Handling call
1025 07:52:49.057079 (+7540138us) ts_tablet_manager.cc:1151] Deleting tablet cb824bd780524832b2b60f5618a39da5
1025 07:52:49.057090 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.057159 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7539964}
I20251025 07:52:49.058959 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 229) took 7519 ms. Trace:
I20251025 07:52:49.059038 2169 rpcz_store.cc:276] 1025 07:52:41.539963 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.540057 (+ 94us) service_pool.cc:225] Handling call
1025 07:52:49.057652 (+7517595us) ts_tablet_manager.cc:1151] Deleting tablet cc407c7eb12e4eadb41fb21385611c94
1025 07:52:49.058845 (+ 1193us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.058939 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7517434}
W20251025 07:52:49.059401 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet cc407c7eb12e4eadb41fb21385611c94 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc407c7eb12e4eadb41fb21385611c94 already in progress: creating tablet
W20251025 07:52:49.059614 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet cb824bd780524832b2b60f5618a39da5 because the tablet was not found. No further retry: Not found: Tablet not found: cb824bd780524832b2b60f5618a39da5
I20251025 07:52:49.057461 1166 tablet_bootstrap.cc:492] T e429b94326cc4827a03968fd92de38e0 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:49.060112 2988 consensus_queue.cc:1048] T 1068f566477e4c6db7dde566fff4c2fb P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:49.061591 1166 tablet_bootstrap.cc:654] T e429b94326cc4827a03968fd92de38e0 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.061559 2988 consensus_queue.cc:1048] T c187a40f8ff14c6a8c184fbeb030ad13 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:49.062582 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 227) took 7545 ms. Trace:
I20251025 07:52:49.062669 2146 rpcz_store.cc:276] 1025 07:52:41.517455 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.517498 (+ 43us) service_pool.cc:225] Handling call
1025 07:52:49.062480 (+7544982us) ts_tablet_manager.cc:1151] Deleting tablet ee1c8ea694284d80a30f9c05be4cbf49
1025 07:52:49.062489 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.062567 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7544803}
W20251025 07:52:49.063036 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet ee1c8ea694284d80a30f9c05be4cbf49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee1c8ea694284d80a30f9c05be4cbf49 already in progress: creating tablet
I20251025 07:52:49.063311 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 228) took 7545 ms. Trace:
I20251025 07:52:49.063417 2146 rpcz_store.cc:276] 1025 07:52:41.517865 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.518150 (+ 285us) service_pool.cc:225] Handling call
1025 07:52:49.063235 (+7545085us) ts_tablet_manager.cc:1151] Deleting tablet 4037dee385d04e85bb78c8e77cb00379
1025 07:52:49.063243 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.063297 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7544920}
W20251025 07:52:49.065929 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 4037dee385d04e85bb78c8e77cb00379 because the tablet was not found. No further retry: Not found: Tablet not found: 4037dee385d04e85bb78c8e77cb00379
I20251025 07:52:49.066344 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 231) took 7541 ms. Trace:
I20251025 07:52:49.066455 2146 rpcz_store.cc:276] 1025 07:52:41.524634 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.524674 (+ 40us) service_pool.cc:225] Handling call
1025 07:52:49.066263 (+7541589us) ts_tablet_manager.cc:1151] Deleting tablet 467324e2432f4caca148951329caa292
1025 07:52:49.066273 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.066328 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7541383}
W20251025 07:52:49.066915 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 467324e2432f4caca148951329caa292 because the tablet was not found. No further retry: Not found: Tablet not found: 467324e2432f4caca148951329caa292
I20251025 07:52:49.067796 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 232) took 7542 ms. Trace:
I20251025 07:52:49.067910 2146 rpcz_store.cc:276] 1025 07:52:41.525032 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.525073 (+ 41us) service_pool.cc:225] Handling call
1025 07:52:49.067167 (+7542094us) ts_tablet_manager.cc:1151] Deleting tablet 1d261e432cf5403ea162129706bc4d4d
1025 07:52:49.067176 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.067782 (+ 606us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7541917}
W20251025 07:52:49.075865 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 1d261e432cf5403ea162129706bc4d4d because the tablet was not found. No further retry: Not found: Tablet not found: 1d261e432cf5403ea162129706bc4d4d
I20251025 07:52:49.076282 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 233) took 7550 ms. Trace:
I20251025 07:52:49.085752 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 230) took 7545 ms. Trace:
I20251025 07:52:49.085855 2169 rpcz_store.cc:276] 1025 07:52:41.540876 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.541074 (+ 198us) service_pool.cc:225] Handling call
1025 07:52:49.059355 (+7518281us) ts_tablet_manager.cc:1151] Deleting tablet 1b5ac7e845a24b2eb5b2429d47361232
1025 07:52:49.085592 (+ 26237us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.085732 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7518121}
I20251025 07:52:49.085233 2991 consensus_queue.cc:1048] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:49.086374 2146 rpcz_store.cc:276] 1025 07:52:41.525372 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.525413 (+ 41us) service_pool.cc:225] Handling call
1025 07:52:49.076194 (+7550781us) ts_tablet_manager.cc:1151] Deleting tablet 2784bbfb97d5466f9ec856add7d0b29a
1025 07:52:49.076204 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.076265 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7550587}
I20251025 07:52:49.086740 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 237) took 7559 ms. Trace:
I20251025 07:52:49.086822 2146 rpcz_store.cc:276] 1025 07:52:41.527016 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.527314 (+ 298us) service_pool.cc:225] Handling call
1025 07:52:49.086650 (+7559336us) ts_tablet_manager.cc:1151] Deleting tablet 627a10a0714f4be8bdbced503fd6388d
1025 07:52:49.086660 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.086724 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7559161}
I20251025 07:52:49.087086 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 238) took 7559 ms. Trace:
I20251025 07:52:49.087154 2146 rpcz_store.cc:276] 1025 07:52:41.527648 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.527993 (+ 345us) service_pool.cc:225] Handling call
1025 07:52:49.087020 (+7559027us) ts_tablet_manager.cc:1151] Deleting tablet 35ba9d23262248cda6fdf03c51be11a4
1025 07:52:49.087029 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.087074 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7558856}
I20251025 07:52:49.087370 2146 tablet_replica.cc:333] T 4c155aa7fa5b4d68bbab1918b8432372 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:49.087545 2146 raft_consensus.cc:2243] T 4c155aa7fa5b4d68bbab1918b8432372 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:49.087859 2146 raft_consensus.cc:2272] T 4c155aa7fa5b4d68bbab1918b8432372 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:49.088446 2146 ts_tablet_manager.cc:1916] T 4c155aa7fa5b4d68bbab1918b8432372 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:49.088443 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 2784bbfb97d5466f9ec856add7d0b29a because the tablet was not found. No further retry: Not found: Tablet not found: 2784bbfb97d5466f9ec856add7d0b29a
W20251025 07:52:49.088873 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 627a10a0714f4be8bdbced503fd6388d because the tablet was not found. No further retry: Not found: Tablet not found: 627a10a0714f4be8bdbced503fd6388d
W20251025 07:52:49.089149 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 35ba9d23262248cda6fdf03c51be11a4 because the tablet was not found. No further retry: Not found: Tablet not found: 35ba9d23262248cda6fdf03c51be11a4
W20251025 07:52:49.089813 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 1b5ac7e845a24b2eb5b2429d47361232 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b5ac7e845a24b2eb5b2429d47361232 already in progress: creating tablet
I20251025 07:52:49.090457 1169 tablet_bootstrap.cc:492] T 1068f566477e4c6db7dde566fff4c2fb P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:49.090590 1169 ts_tablet_manager.cc:1403] T 1068f566477e4c6db7dde566fff4c2fb P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.069s user 0.004s sys 0.014s
I20251025 07:52:49.091256 1169 raft_consensus.cc:359] T 1068f566477e4c6db7dde566fff4c2fb P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.091421 1169 raft_consensus.cc:385] T 1068f566477e4c6db7dde566fff4c2fb P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.091480 1169 raft_consensus.cc:740] T 1068f566477e4c6db7dde566fff4c2fb P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.091670 1169 consensus_queue.cc:260] T 1068f566477e4c6db7dde566fff4c2fb P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.097649 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 239) took 7525 ms. Trace:
I20251025 07:52:49.097759 2169 rpcz_store.cc:276] 1025 07:52:41.572252 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.572336 (+ 84us) service_pool.cc:225] Handling call
1025 07:52:49.086100 (+7513764us) ts_tablet_manager.cc:1151] Deleting tablet 21e52928580c4762b0ae7152490ef955
1025 07:52:49.096808 (+ 10708us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.097626 (+ 818us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7513534}
I20251025 07:52:49.098371 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 243) took 7516 ms. Trace:
I20251025 07:52:49.098466 2169 rpcz_store.cc:276] 1025 07:52:41.581644 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.586080 (+ 4436us) service_pool.cc:225] Handling call
1025 07:52:49.098256 (+7512176us) ts_tablet_manager.cc:1151] Deleting tablet 5361f08f50b745eb98bf02e979d7c71e
1025 07:52:49.098268 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.098341 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7511969}
W20251025 07:52:49.098698 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 21e52928580c4762b0ae7152490ef955 because the tablet was not found. No further retry: Not found: Tablet not found: 21e52928580c4762b0ae7152490ef955
I20251025 07:52:49.098877 2169 tablet_replica.cc:333] T 4c155aa7fa5b4d68bbab1918b8432372 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
W20251025 07:52:49.099069 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 5361f08f50b745eb98bf02e979d7c71e because the tablet was not found. No further retry: Not found: Tablet not found: 5361f08f50b745eb98bf02e979d7c71e
I20251025 07:52:49.099133 2169 raft_consensus.cc:2243] T 4c155aa7fa5b4d68bbab1918b8432372 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:49.107719 2169 raft_consensus.cc:2272] T 4c155aa7fa5b4d68bbab1918b8432372 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:49.107668 3011 raft_consensus.cc:493] T dda0b26e4f1a44928bc4b84f333901c0 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.107854 3011 raft_consensus.cc:515] T dda0b26e4f1a44928bc4b84f333901c0 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:49.108444 3011 leader_election.cc:290] T dda0b26e4f1a44928bc4b84f333901c0 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:49.108615 1169 ts_tablet_manager.cc:1434] T 1068f566477e4c6db7dde566fff4c2fb P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20251025 07:52:49.108868 1169 tablet_bootstrap.cc:492] T fa826b85f0c94150a17d4e417dd3d62f P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:49.109424 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dda0b26e4f1a44928bc4b84f333901c0" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:49.109640 768 raft_consensus.cc:2393] T dda0b26e4f1a44928bc4b84f333901c0 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:49.109642 2169 ts_tablet_manager.cc:1916] T 4c155aa7fa5b4d68bbab1918b8432372 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:49.112912 2888 raft_consensus.cc:493] T 4b278ce7768349e49dcd1131e404ab34 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.113045 2888 raft_consensus.cc:515] T 4b278ce7768349e49dcd1131e404ab34 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.113566 2888 leader_election.cc:290] T 4b278ce7768349e49dcd1131e404ab34 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:49.113739 2597 raft_consensus.cc:493] T 48ff4ce7846b4e56a37eee9bcb17bd5e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.113854 2597 raft_consensus.cc:515] T 48ff4ce7846b4e56a37eee9bcb17bd5e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:49.114319 2597 leader_election.cc:290] T 48ff4ce7846b4e56a37eee9bcb17bd5e P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
W20251025 07:52:49.114464 983 leader_election.cc:343] T dda0b26e4f1a44928bc4b84f333901c0 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: dda0b26e4f1a44928bc4b84f333901c0
I20251025 07:52:49.114560 983 leader_election.cc:304] T dda0b26e4f1a44928bc4b84f333901c0 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:49.114856 2597 raft_consensus.cc:3060] T dda0b26e4f1a44928bc4b84f333901c0 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.115145 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b278ce7768349e49dcd1131e404ab34" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:49.115947 983 leader_election.cc:343] T 48ff4ce7846b4e56a37eee9bcb17bd5e P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 48ff4ce7846b4e56a37eee9bcb17bd5e
I20251025 07:52:49.117633 2597 raft_consensus.cc:2749] T dda0b26e4f1a44928bc4b84f333901c0 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:49.117942 712 leader_election.cc:343] T 4b278ce7768349e49dcd1131e404ab34 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.120074 1169 tablet_bootstrap.cc:654] T fa826b85f0c94150a17d4e417dd3d62f P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.121450 2169 ts_tablet_manager.cc:1929] T 4c155aa7fa5b4d68bbab1918b8432372 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:49.121539 2169 log.cc:1199] T 4c155aa7fa5b4d68bbab1918b8432372 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/4c155aa7fa5b4d68bbab1918b8432372
I20251025 07:52:49.121882 2169 ts_tablet_manager.cc:1950] T 4c155aa7fa5b4d68bbab1918b8432372 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
W20251025 07:52:49.122606 982 leader_election.cc:343] T 48ff4ce7846b4e56a37eee9bcb17bd5e P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: 48ff4ce7846b4e56a37eee9bcb17bd5e
I20251025 07:52:49.122766 982 leader_election.cc:304] T 48ff4ce7846b4e56a37eee9bcb17bd5e P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:49.123143 2597 raft_consensus.cc:2749] T 48ff4ce7846b4e56a37eee9bcb17bd5e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:49.123711 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b278ce7768349e49dcd1131e404ab34" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:49.124425 710 leader_election.cc:343] T 4b278ce7768349e49dcd1131e404ab34 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.124531 710 leader_election.cc:304] T 4b278ce7768349e49dcd1131e404ab34 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.124893 2888 raft_consensus.cc:2749] T 4b278ce7768349e49dcd1131e404ab34 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:49.122613 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 253) took 5528 ms. Trace:
I20251025 07:52:49.127007 2169 rpcz_store.cc:276] 1025 07:52:43.594324 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:43.594379 (+ 55us) service_pool.cc:225] Handling call
1025 07:52:49.098824 (+5504445us) ts_tablet_manager.cc:1151] Deleting tablet 4c155aa7fa5b4d68bbab1918b8432372
1025 07:52:49.098835 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.117252 (+ 18417us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.121365 (+ 4113us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.122593 (+ 1228us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9618,"tablet-delete.queue_time_us":5504277}
I20251025 07:52:49.127394 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 257) took 5353 ms. Trace:
I20251025 07:52:49.127704 2169 rpcz_store.cc:276] 1025 07:52:43.774380 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:43.774442 (+ 62us) service_pool.cc:225] Handling call
1025 07:52:49.127289 (+5352847us) ts_tablet_manager.cc:1151] Deleting tablet fce535a0c0bd45c6bf17f81add297f40
1025 07:52:49.127298 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.127381 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5352628}
W20251025 07:52:49.128116 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet fce535a0c0bd45c6bf17f81add297f40 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fce535a0c0bd45c6bf17f81add297f40 already in progress: creating tablet
I20251025 07:52:49.128386 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 258) took 5216 ms. Trace:
I20251025 07:52:49.128475 2169 rpcz_store.cc:276] 1025 07:52:43.911706 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:43.911860 (+ 154us) service_pool.cc:225] Handling call
1025 07:52:49.127926 (+5216066us) ts_tablet_manager.cc:1151] Deleting tablet 32a143a715cb485cb4501f4e28c68252
1025 07:52:49.127934 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.128371 (+ 437us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5215894}
W20251025 07:52:49.128810 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 32a143a715cb485cb4501f4e28c68252 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32a143a715cb485cb4501f4e28c68252 already in progress: creating tablet
I20251025 07:52:49.129009 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 259) took 5160 ms. Trace:
I20251025 07:52:49.129076 2169 rpcz_store.cc:276] 1025 07:52:43.968342 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:43.968424 (+ 82us) service_pool.cc:225] Handling call
1025 07:52:49.128661 (+5160237us) ts_tablet_manager.cc:1151] Deleting tablet 0540d6a10cf04fa89756623e9aaaeb68
1025 07:52:49.128669 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.128996 (+ 327us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5160001}
W20251025 07:52:49.129462 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 0540d6a10cf04fa89756623e9aaaeb68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0540d6a10cf04fa89756623e9aaaeb68 already in progress: creating tablet
I20251025 07:52:49.129690 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 264) took 4620 ms. Trace:
I20251025 07:52:49.129761 2169 rpcz_store.cc:276] 1025 07:52:44.508982 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.509026 (+ 44us) service_pool.cc:225] Handling call
1025 07:52:49.129268 (+4620242us) ts_tablet_manager.cc:1151] Deleting tablet 60c60a9ee37f43978910bd067a098bb7
1025 07:52:49.129275 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.129677 (+ 402us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4620019}
W20251025 07:52:49.130107 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 60c60a9ee37f43978910bd067a098bb7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60c60a9ee37f43978910bd067a098bb7 already in progress: creating tablet
I20251025 07:52:49.130334 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 271) took 4555 ms. Trace:
I20251025 07:52:49.130412 2169 rpcz_store.cc:276] 1025 07:52:44.575077 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.575134 (+ 57us) service_pool.cc:225] Handling call
1025 07:52:49.129954 (+4554820us) ts_tablet_manager.cc:1151] Deleting tablet 95efd553493e4966bc48a01cac6e1e10
1025 07:52:49.129961 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.130321 (+ 360us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4554616}
W20251025 07:52:49.130774 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 95efd553493e4966bc48a01cac6e1e10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95efd553493e4966bc48a01cac6e1e10 already in progress: creating tablet
I20251025 07:52:49.130950 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 274) took 4548 ms. Trace:
I20251025 07:52:49.131009 2169 rpcz_store.cc:276] 1025 07:52:44.582815 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.582859 (+ 44us) service_pool.cc:225] Handling call
1025 07:52:49.130599 (+4547740us) ts_tablet_manager.cc:1151] Deleting tablet 27651312cc2e42a8bfcd245f5d301679
1025 07:52:49.130607 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.130936 (+ 329us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4547506}
W20251025 07:52:49.132828 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 27651312cc2e42a8bfcd245f5d301679 because the tablet was not found. No further retry: Not found: Tablet not found: 27651312cc2e42a8bfcd245f5d301679
I20251025 07:52:49.134855 1169 tablet_bootstrap.cc:492] T fa826b85f0c94150a17d4e417dd3d62f P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:49.134999 1169 ts_tablet_manager.cc:1403] T fa826b85f0c94150a17d4e417dd3d62f P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.001s
I20251025 07:52:49.135742 1169 raft_consensus.cc:359] T fa826b85f0c94150a17d4e417dd3d62f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:49.135922 1169 raft_consensus.cc:385] T fa826b85f0c94150a17d4e417dd3d62f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.135985 1169 raft_consensus.cc:740] T fa826b85f0c94150a17d4e417dd3d62f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.137347 2146 ts_tablet_manager.cc:1929] T 4c155aa7fa5b4d68bbab1918b8432372 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:49.138343 2146 log.cc:1199] T 4c155aa7fa5b4d68bbab1918b8432372 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/4c155aa7fa5b4d68bbab1918b8432372
I20251025 07:52:49.138974 2146 ts_tablet_manager.cc:1950] T 4c155aa7fa5b4d68bbab1918b8432372 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:49.141165 1169 consensus_queue.cc:260] T fa826b85f0c94150a17d4e417dd3d62f P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:49.142192 1169 ts_tablet_manager.cc:1434] T fa826b85f0c94150a17d4e417dd3d62f P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251025 07:52:49.143232 2988 raft_consensus.cc:493] T 1c40b2e06d894a06af9e58285892ea8d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.143376 2988 raft_consensus.cc:515] T 1c40b2e06d894a06af9e58285892ea8d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.143899 2988 leader_election.cc:290] T 1c40b2e06d894a06af9e58285892ea8d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:49.145174 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 256) took 5551 ms. Trace:
I20251025 07:52:49.148010 1042 raft_consensus.cc:3060] T 1aeba18be0cb4eb9949aeaf6ad73a812 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.148384 2146 rpcz_store.cc:276] 1025 07:52:43.593717 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:43.593948 (+ 231us) service_pool.cc:225] Handling call
1025 07:52:49.087330 (+5493382us) ts_tablet_manager.cc:1151] Deleting tablet 4c155aa7fa5b4d68bbab1918b8432372
1025 07:52:49.087337 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.122320 (+ 34983us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.137199 (+ 14879us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.145148 (+ 7949us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5493208}
I20251025 07:52:49.148643 2988 raft_consensus.cc:493] T 2d24e3e7a9e4431db609331ad1fd57c9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.148934 2988 raft_consensus.cc:515] T 2d24e3e7a9e4431db609331ad1fd57c9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.149473 2988 leader_election.cc:290] T 2d24e3e7a9e4431db609331ad1fd57c9 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:49.150305 2146 tablet_replica.cc:333] T fce535a0c0bd45c6bf17f81add297f40 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:49.150707 2146 raft_consensus.cc:2243] T fce535a0c0bd45c6bf17f81add297f40 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:49.150664 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d24e3e7a9e4431db609331ad1fd57c9" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:49.150835 768 raft_consensus.cc:2468] T 2d24e3e7a9e4431db609331ad1fd57c9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:49.151201 2146 pending_rounds.cc:70] T fce535a0c0bd45c6bf17f81add297f40 P 3aea2869ff814cc7b142012faa274382: Trying to abort 1 pending ops.
I20251025 07:52:49.151516 2146 pending_rounds.cc:77] T fce535a0c0bd45c6bf17f81add297f40 P 3aea2869ff814cc7b142012faa274382: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214607435503513600 op_type: NO_OP noop_request { }
I20251025 07:52:49.151914 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c40b2e06d894a06af9e58285892ea8d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:49.152113 2146 raft_consensus.cc:2889] T fce535a0c0bd45c6bf17f81add297f40 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 07:52:49.152202 2146 raft_consensus.cc:2272] T fce535a0c0bd45c6bf17f81add297f40 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:49.152837 847 leader_election.cc:304] T 2d24e3e7a9e4431db609331ad1fd57c9 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
W20251025 07:52:49.154006 850 leader_election.cc:343] T 1c40b2e06d894a06af9e58285892ea8d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.154101 2886 consensus_queue.cc:1048] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:49.154325 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 275) took 4571 ms. Trace:
I20251025 07:52:49.155591 1169 tablet_bootstrap.cc:492] T 9f039d9107ae493486d799ab6a1d731d P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:49.156266 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c40b2e06d894a06af9e58285892ea8d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:49.156862 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d24e3e7a9e4431db609331ad1fd57c9" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:49.156939 2886 raft_consensus.cc:2804] T 2d24e3e7a9e4431db609331ad1fd57c9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:49.157008 2886 raft_consensus.cc:493] T 2d24e3e7a9e4431db609331ad1fd57c9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:49.157064 2886 raft_consensus.cc:3060] T 2d24e3e7a9e4431db609331ad1fd57c9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.157397 1169 tablet_bootstrap.cc:654] T 9f039d9107ae493486d799ab6a1d731d P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.160046 2886 raft_consensus.cc:515] T 2d24e3e7a9e4431db609331ad1fd57c9 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:49.160614 850 leader_election.cc:343] T 2d24e3e7a9e4431db609331ad1fd57c9 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.161676 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d24e3e7a9e4431db609331ad1fd57c9" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:49.162169 850 leader_election.cc:343] T 2d24e3e7a9e4431db609331ad1fd57c9 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.162350 2886 leader_election.cc:290] T 2d24e3e7a9e4431db609331ad1fd57c9 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:49.167022 772 raft_consensus.cc:2468] T 1c40b2e06d894a06af9e58285892ea8d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:49.169128 1169 tablet_bootstrap.cc:492] T 9f039d9107ae493486d799ab6a1d731d P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:49.169266 1169 ts_tablet_manager.cc:1403] T 9f039d9107ae493486d799ab6a1d731d P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.002s
I20251025 07:52:49.154443 2169 rpcz_store.cc:276] 1025 07:52:44.582928 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.583132 (+ 204us) service_pool.cc:225] Handling call
1025 07:52:49.131174 (+4548042us) ts_tablet_manager.cc:1151] Deleting tablet 2e098c9c4613459295b6043048cba587
1025 07:52:49.147350 (+ 16176us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.154308 (+ 6958us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4547831}
W20251025 07:52:49.169706 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 2e098c9c4613459295b6043048cba587 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e098c9c4613459295b6043048cba587 already in progress: creating tablet
I20251025 07:52:49.169881 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 273) took 4587 ms. Trace:
I20251025 07:52:49.170053 1169 raft_consensus.cc:359] T 9f039d9107ae493486d799ab6a1d731d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:49.170248 1169 raft_consensus.cc:385] T 9f039d9107ae493486d799ab6a1d731d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.170310 1169 raft_consensus.cc:740] T 9f039d9107ae493486d799ab6a1d731d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.169991 2169 rpcz_store.cc:276] 1025 07:52:44.582731 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.583212 (+ 481us) service_pool.cc:225] Handling call
1025 07:52:49.169768 (+4586556us) ts_tablet_manager.cc:1151] Deleting tablet fab67b56eaa44b54886c17181d8c52b2
1025 07:52:49.169780 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.169858 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4586395}
I20251025 07:52:49.170500 1169 consensus_queue.cc:260] T 9f039d9107ae493486d799ab6a1d731d P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
W20251025 07:52:49.170876 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet fab67b56eaa44b54886c17181d8c52b2 because the tablet was not found. No further retry: Not found: Tablet not found: fab67b56eaa44b54886c17181d8c52b2
I20251025 07:52:49.171176 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 272) took 4588 ms. Trace:
I20251025 07:52:49.171272 2169 rpcz_store.cc:276] 1025 07:52:44.582613 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.583148 (+ 535us) service_pool.cc:225] Handling call
1025 07:52:49.171087 (+4587939us) ts_tablet_manager.cc:1151] Deleting tablet 896bd7870d87408e8adfdb7a78998111
1025 07:52:49.171099 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.171161 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4587576}
W20251025 07:52:49.171645 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 896bd7870d87408e8adfdb7a78998111 because the tablet was not found. No further retry: Not found: Tablet not found: 896bd7870d87408e8adfdb7a78998111
I20251025 07:52:49.172325 1169 ts_tablet_manager.cc:1434] T 9f039d9107ae493486d799ab6a1d731d P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251025 07:52:49.173569 1167 tablet_bootstrap.cc:700] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b: Time spent opening tablet: real 0.155s user 0.001s sys 0.000s
I20251025 07:52:49.173568 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 276) took 4590 ms. Trace:
I20251025 07:52:49.173657 2169 rpcz_store.cc:276] 1025 07:52:44.583102 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.583544 (+ 442us) service_pool.cc:225] Handling call
1025 07:52:49.173339 (+4589795us) ts_tablet_manager.cc:1151] Deleting tablet 5fca357d7e24494fb6af4bd73d98833c
1025 07:52:49.173350 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.173551 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4589613}
W20251025 07:52:49.174139 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 5fca357d7e24494fb6af4bd73d98833c because the tablet was not found. No further retry: Not found: Tablet not found: 5fca357d7e24494fb6af4bd73d98833c
I20251025 07:52:49.174300 1169 tablet_bootstrap.cc:492] T 1b5ac7e845a24b2eb5b2429d47361232 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:49.175516 1167 tablet_bootstrap.cc:654] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.176538 1169 tablet_bootstrap.cc:654] T 1b5ac7e845a24b2eb5b2429d47361232 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.178114 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 277) took 4594 ms. Trace:
I20251025 07:52:49.178232 2169 rpcz_store.cc:276] 1025 07:52:44.583717 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.583771 (+ 54us) service_pool.cc:225] Handling call
1025 07:52:49.177741 (+4593970us) ts_tablet_manager.cc:1151] Deleting tablet ac0ea298bc114c56a8093e12def21ce1
1025 07:52:49.177752 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.178091 (+ 339us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4593828}
I20251025 07:52:49.179113 2169 tablet_replica.cc:333] T 570201e7bf05406a97c81dde071d894b P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:49.179349 2169 raft_consensus.cc:2243] T 570201e7bf05406a97c81dde071d894b P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:49.179534 2169 raft_consensus.cc:2272] T 570201e7bf05406a97c81dde071d894b P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:49.181272 2169 ts_tablet_manager.cc:1916] T 570201e7bf05406a97c81dde071d894b P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:49.189239 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRBHlYt/build/asan/bin/kudu
/tmp/dist-test-taskRBHlYt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:42203
--webserver_interface=127.0.104.61
--webserver_port=45311
--builtin_ntp_servers=127.0.104.20:40091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:37421,127.0.104.61:42203,127.0.104.60:37611
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251025 07:52:49.204109 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet ac0ea298bc114c56a8093e12def21ce1 because the tablet was not found. No further retry: Not found: Tablet not found: ac0ea298bc114c56a8093e12def21ce1
I20251025 07:52:49.175112 771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d24e3e7a9e4431db609331ad1fd57c9" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:49.210916 771 raft_consensus.cc:3060] T 2d24e3e7a9e4431db609331ad1fd57c9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.215341 847 leader_election.cc:304] T 1c40b2e06d894a06af9e58285892ea8d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.217207 1166 tablet_bootstrap.cc:492] T e429b94326cc4827a03968fd92de38e0 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:49.217458 1166 ts_tablet_manager.cc:1403] T e429b94326cc4827a03968fd92de38e0 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.160s user 0.004s sys 0.033s
I20251025 07:52:49.217686 2886 raft_consensus.cc:2804] T 1c40b2e06d894a06af9e58285892ea8d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:49.217770 2886 raft_consensus.cc:493] T 1c40b2e06d894a06af9e58285892ea8d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:49.217828 2886 raft_consensus.cc:3060] T 1c40b2e06d894a06af9e58285892ea8d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.218204 1166 raft_consensus.cc:359] T e429b94326cc4827a03968fd92de38e0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:49.218360 1166 raft_consensus.cc:385] T e429b94326cc4827a03968fd92de38e0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.218415 1166 raft_consensus.cc:740] T e429b94326cc4827a03968fd92de38e0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.218592 1166 consensus_queue.cc:260] T e429b94326cc4827a03968fd92de38e0 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:49.219213 1166 ts_tablet_manager.cc:1434] T e429b94326cc4827a03968fd92de38e0 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:49.219545 1166 tablet_bootstrap.cc:492] T 243b8828fc344bacab46d7bab6bd2dec P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:49.220539 2886 raft_consensus.cc:515] T 1c40b2e06d894a06af9e58285892ea8d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.221077 1166 tablet_bootstrap.cc:654] T 243b8828fc344bacab46d7bab6bd2dec P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.221518 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c40b2e06d894a06af9e58285892ea8d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:49.222013 850 leader_election.cc:343] T 1c40b2e06d894a06af9e58285892ea8d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.222203 2886 leader_election.cc:290] T 1c40b2e06d894a06af9e58285892ea8d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:49.226071 1166 tablet_bootstrap.cc:492] T 243b8828fc344bacab46d7bab6bd2dec P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:49.226470 1166 ts_tablet_manager.cc:1403] T 243b8828fc344bacab46d7bab6bd2dec P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20251025 07:52:49.233785 2169 ts_tablet_manager.cc:1929] T 570201e7bf05406a97c81dde071d894b P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:49.235673 2169 log.cc:1199] T 570201e7bf05406a97c81dde071d894b P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/570201e7bf05406a97c81dde071d894b
I20251025 07:52:49.236364 2169 ts_tablet_manager.cc:1950] T 570201e7bf05406a97c81dde071d894b P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:49.236771 2886 consensus_queue.cc:1048] T c187a40f8ff14c6a8c184fbeb030ad13 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:49.236883 1166 raft_consensus.cc:359] T 243b8828fc344bacab46d7bab6bd2dec P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.237012 1166 raft_consensus.cc:385] T 243b8828fc344bacab46d7bab6bd2dec P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.237061 1166 raft_consensus.cc:740] T 243b8828fc344bacab46d7bab6bd2dec P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.237242 1166 consensus_queue.cc:260] T 243b8828fc344bacab46d7bab6bd2dec P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.238195 1166 ts_tablet_manager.cc:1434] T 243b8828fc344bacab46d7bab6bd2dec P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20251025 07:52:49.238509 1166 tablet_bootstrap.cc:492] T d0240d20f9904b4bbc8bf7f0f05e8850 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:49.240559 1166 tablet_bootstrap.cc:654] T d0240d20f9904b4bbc8bf7f0f05e8850 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.240540 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 278) took 4652 ms. Trace:
I20251025 07:52:49.242005 2169 rpcz_store.cc:276] 1025 07:52:44.588211 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.588262 (+ 51us) service_pool.cc:225] Handling call
1025 07:52:49.179051 (+4590789us) ts_tablet_manager.cc:1151] Deleting tablet 570201e7bf05406a97c81dde071d894b
1025 07:52:49.179062 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.213185 (+ 34123us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.233618 (+ 20433us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.240501 (+ 6883us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4590554}
I20251025 07:52:49.242993 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 570201e7bf05406a97c81dde071d894b (table default.Table_c62e0dc1ae504f9bb912013506ff3031 [id=6f395892b09240fea02482b3199cfb5e]) successfully deleted
I20251025 07:52:49.243353 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 279) took 4647 ms. Trace:
I20251025 07:52:49.243448 2169 rpcz_store.cc:276] 1025 07:52:44.595549 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.595604 (+ 55us) service_pool.cc:225] Handling call
1025 07:52:49.243237 (+4647633us) ts_tablet_manager.cc:1151] Deleting tablet 943ec703e65f4fd3b0c690e08c0bd226
1025 07:52:49.243248 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.243316 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4647430}
W20251025 07:52:49.243826 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 943ec703e65f4fd3b0c690e08c0bd226 because the tablet was not found. No further retry: Not found: Tablet not found: 943ec703e65f4fd3b0c690e08c0bd226
I20251025 07:52:49.244771 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 280) took 4648 ms. Trace:
I20251025 07:52:49.244849 2169 rpcz_store.cc:276] 1025 07:52:44.596640 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.596690 (+ 50us) service_pool.cc:225] Handling call
1025 07:52:49.244693 (+4648003us) ts_tablet_manager.cc:1151] Deleting tablet 3775a2e08d4e40bd9f57e4cda759d535
1025 07:52:49.244705 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.244758 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4647788}
W20251025 07:52:49.245438 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 3775a2e08d4e40bd9f57e4cda759d535 because the tablet was not found. No further retry: Not found: Tablet not found: 3775a2e08d4e40bd9f57e4cda759d535
I20251025 07:52:49.246042 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 281) took 4646 ms. Trace:
I20251025 07:52:49.246124 2169 rpcz_store.cc:276] 1025 07:52:44.599389 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.599530 (+ 141us) service_pool.cc:225] Handling call
1025 07:52:49.245972 (+4646442us) ts_tablet_manager.cc:1151] Deleting tablet 627a10a0714f4be8bdbced503fd6388d
1025 07:52:49.245981 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.246029 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4646180}
I20251025 07:52:49.247339 1042 raft_consensus.cc:3060] T 9f039d9107ae493486d799ab6a1d731d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.247668 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 282) took 4648 ms. Trace:
I20251025 07:52:49.248181 2169 rpcz_store.cc:276] 1025 07:52:44.599524 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.599771 (+ 247us) service_pool.cc:225] Handling call
1025 07:52:49.247588 (+4647817us) ts_tablet_manager.cc:1151] Deleting tablet 35ba9d23262248cda6fdf03c51be11a4
1025 07:52:49.247597 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.247654 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4647618}
I20251025 07:52:49.249172 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 283) took 4648 ms. Trace:
I20251025 07:52:49.249676 2169 rpcz_store.cc:276] 1025 07:52:44.600380 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.600423 (+ 43us) service_pool.cc:225] Handling call
1025 07:52:49.249105 (+4648682us) ts_tablet_manager.cc:1151] Deleting tablet 74e2d78c024849b0af5c0bde1f48c36f
1025 07:52:49.249114 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.249160 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4648499}
I20251025 07:52:49.250516 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 284) took 4648 ms. Trace:
W20251025 07:52:49.252224 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 627a10a0714f4be8bdbced503fd6388d because the tablet was not found. No further retry: Not found: Tablet not found: 627a10a0714f4be8bdbced503fd6388d
I20251025 07:52:49.251858 2169 rpcz_store.cc:276] 1025 07:52:44.601565 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.601844 (+ 279us) service_pool.cc:225] Handling call
1025 07:52:49.250450 (+4648606us) ts_tablet_manager.cc:1151] Deleting tablet 02e4c4d6ffb34cecaf17beb5ba197e81
1025 07:52:49.250458 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.250505 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4648395}
W20251025 07:52:49.252583 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 35ba9d23262248cda6fdf03c51be11a4 because the tablet was not found. No further retry: Not found: Tablet not found: 35ba9d23262248cda6fdf03c51be11a4
W20251025 07:52:49.252816 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 74e2d78c024849b0af5c0bde1f48c36f because the tablet was not found. No further retry: Not found: Tablet not found: 74e2d78c024849b0af5c0bde1f48c36f
I20251025 07:52:49.253005 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 286) took 4590 ms. Trace:
W20251025 07:52:49.253059 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 02e4c4d6ffb34cecaf17beb5ba197e81 because the tablet was not found. No further retry: Not found: Tablet not found: 02e4c4d6ffb34cecaf17beb5ba197e81
I20251025 07:52:49.253108 2169 rpcz_store.cc:276] 1025 07:52:44.662401 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.662461 (+ 60us) service_pool.cc:225] Handling call
1025 07:52:49.252907 (+4590446us) ts_tablet_manager.cc:1151] Deleting tablet 7c7a9d7a4fb3449bb57eafa83230a839
1025 07:52:49.252919 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.252988 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4590201}
I20251025 07:52:49.254444 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 287) took 4591 ms. Trace:
I20251025 07:52:49.254536 2169 rpcz_store.cc:276] 1025 07:52:44.662669 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.662736 (+ 67us) service_pool.cc:225] Handling call
1025 07:52:49.254351 (+4591615us) ts_tablet_manager.cc:1151] Deleting tablet 70fe9aa2f58d4abea3b3b71066a29653
1025 07:52:49.254361 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.254427 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4591479}
W20251025 07:52:49.255158 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 7c7a9d7a4fb3449bb57eafa83230a839 because the tablet was not found. No further retry: Not found: Tablet not found: 7c7a9d7a4fb3449bb57eafa83230a839
W20251025 07:52:49.255435 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 70fe9aa2f58d4abea3b3b71066a29653 because the tablet was not found. No further retry: Not found: Tablet not found: 70fe9aa2f58d4abea3b3b71066a29653
I20251025 07:52:49.255757 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 288) took 4592 ms. Trace:
I20251025 07:52:49.255833 2169 rpcz_store.cc:276] 1025 07:52:44.662857 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.662953 (+ 96us) service_pool.cc:225] Handling call
1025 07:52:49.255690 (+4592737us) ts_tablet_manager.cc:1151] Deleting tablet f4578ce69a0d4322944e5ce7c8a59831
1025 07:52:49.255698 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.255744 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4592602}
I20251025 07:52:49.256656 2146 ts_tablet_manager.cc:1916] T fce535a0c0bd45c6bf17f81add297f40 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:49.257062 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 289) took 4593 ms. Trace:
I20251025 07:52:49.257143 2169 rpcz_store.cc:276] 1025 07:52:44.663117 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.663161 (+ 44us) service_pool.cc:225] Handling call
1025 07:52:49.256987 (+4593826us) ts_tablet_manager.cc:1151] Deleting tablet f0bdaa8cfd764f1b83f47881353d26b6
1025 07:52:49.256994 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.257047 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4593677}
I20251025 07:52:49.257766 1166 tablet_bootstrap.cc:492] T d0240d20f9904b4bbc8bf7f0f05e8850 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:49.257911 1166 ts_tablet_manager.cc:1403] T d0240d20f9904b4bbc8bf7f0f05e8850 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.001s
W20251025 07:52:49.258119 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet f4578ce69a0d4322944e5ce7c8a59831 because the tablet was not found. No further retry: Not found: Tablet not found: f4578ce69a0d4322944e5ce7c8a59831
I20251025 07:52:49.258342 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 290) took 4594 ms. Trace:
I20251025 07:52:49.258425 2169 rpcz_store.cc:276] 1025 07:52:44.663466 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.663537 (+ 71us) service_pool.cc:225] Handling call
1025 07:52:49.258244 (+4594707us) ts_tablet_manager.cc:1151] Deleting tablet a7ab3150a35b4caca9119a1374d3f2f7
1025 07:52:49.258252 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.258328 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4594541}
W20251025 07:52:49.258486 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet f0bdaa8cfd764f1b83f47881353d26b6 because the tablet was not found. No further retry: Not found: Tablet not found: f0bdaa8cfd764f1b83f47881353d26b6
I20251025 07:52:49.258687 1166 raft_consensus.cc:359] T d0240d20f9904b4bbc8bf7f0f05e8850 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:49.258800 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet a7ab3150a35b4caca9119a1374d3f2f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7ab3150a35b4caca9119a1374d3f2f7 already in progress: creating tablet
I20251025 07:52:49.258903 1166 raft_consensus.cc:385] T d0240d20f9904b4bbc8bf7f0f05e8850 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.258972 1166 raft_consensus.cc:740] T d0240d20f9904b4bbc8bf7f0f05e8850 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.242743 1167 tablet_bootstrap.cc:492] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:49.259280 1167 ts_tablet_manager.cc:1403] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.256s user 0.004s sys 0.000s
I20251025 07:52:49.259263 1166 consensus_queue.cc:260] T d0240d20f9904b4bbc8bf7f0f05e8850 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.259626 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 291) took 4595 ms. Trace:
I20251025 07:52:49.259712 2169 rpcz_store.cc:276] 1025 07:52:44.663720 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.663789 (+ 69us) service_pool.cc:225] Handling call
1025 07:52:49.259526 (+4595737us) ts_tablet_manager.cc:1151] Deleting tablet 8956ad28b35d45f28c683ac055bba441
1025 07:52:49.259534 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.259612 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4595597}
I20251025 07:52:49.259869 1166 ts_tablet_manager.cc:1434] T d0240d20f9904b4bbc8bf7f0f05e8850 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:49.260274 1166 tablet_bootstrap.cc:492] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:49.260210 1167 raft_consensus.cc:359] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.260470 1167 raft_consensus.cc:385] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.260553 1167 raft_consensus.cc:740] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.260803 1167 consensus_queue.cc:260] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.260998 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 292) took 4597 ms. Trace:
I20251025 07:52:49.261086 2169 rpcz_store.cc:276] 1025 07:52:44.663978 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.664215 (+ 237us) service_pool.cc:225] Handling call
1025 07:52:49.260915 (+4596700us) ts_tablet_manager.cc:1151] Deleting tablet 71432a6848944a0da06ef9b81f160dc4
1025 07:52:49.260925 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.260982 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4596541}
I20251025 07:52:49.262341 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 293) took 4598 ms. Trace:
I20251025 07:52:49.262476 1167 ts_tablet_manager.cc:1434] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251025 07:52:49.262418 2169 rpcz_store.cc:276] 1025 07:52:44.664251 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.664400 (+ 149us) service_pool.cc:225] Handling call
1025 07:52:49.262274 (+4597874us) ts_tablet_manager.cc:1151] Deleting tablet f3001314b56345f5a70314777cae0125
1025 07:52:49.262282 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.262328 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4597730}
I20251025 07:52:49.263504 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 294) took 4599 ms. Trace:
I20251025 07:52:49.263579 2169 rpcz_store.cc:276] 1025 07:52:44.664490 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.664617 (+ 127us) service_pool.cc:225] Handling call
1025 07:52:49.263437 (+4598820us) ts_tablet_manager.cc:1151] Deleting tablet f170424be2f643e7af499834955472b0
1025 07:52:49.263445 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.263490 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4598648}
W20251025 07:52:49.264019 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 8956ad28b35d45f28c683ac055bba441 because the tablet was not found. No further retry: Not found: Tablet not found: 8956ad28b35d45f28c683ac055bba441
W20251025 07:52:49.264302 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 71432a6848944a0da06ef9b81f160dc4 because the tablet was not found. No further retry: Not found: Tablet not found: 71432a6848944a0da06ef9b81f160dc4
W20251025 07:52:49.264518 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet f3001314b56345f5a70314777cae0125 because the tablet was not found. No further retry: Not found: Tablet not found: f3001314b56345f5a70314777cae0125
I20251025 07:52:49.264652 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 295) took 4599 ms. Trace:
I20251025 07:52:49.264725 2169 rpcz_store.cc:276] 1025 07:52:44.664780 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.664820 (+ 40us) service_pool.cc:225] Handling call
1025 07:52:49.264580 (+4599760us) ts_tablet_manager.cc:1151] Deleting tablet a13bf4d7eb1440f7a28b85cb1bf99667
1025 07:52:49.264588 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.264632 (+ 44us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4599640}
W20251025 07:52:49.264799 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet f170424be2f643e7af499834955472b0 because the tablet was not found. No further retry: Not found: Tablet not found: f170424be2f643e7af499834955472b0
I20251025 07:52:49.265200 2888 raft_consensus.cc:493] T 22fc90c704f64d2f9f4b8c54dee45576 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.265324 2888 raft_consensus.cc:515] T 22fc90c704f64d2f9f4b8c54dee45576 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:49.265765 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 296) took 4600 ms. Trace:
I20251025 07:52:49.265864 2888 leader_election.cc:290] T 22fc90c704f64d2f9f4b8c54dee45576 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:49.265906 1166 tablet_bootstrap.cc:654] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.267149 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22fc90c704f64d2f9f4b8c54dee45576" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:49.267575 712 leader_election.cc:343] T 22fc90c704f64d2f9f4b8c54dee45576 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.245833 2166 ts_tablet_manager.cc:1929] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:49.267920 2166 log.cc:1199] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/d9bfd7c0c3ad4e5b98b6708793d61d41
I20251025 07:52:49.268347 2166 ts_tablet_manager.cc:1950] T d9bfd7c0c3ad4e5b98b6708793d61d41 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
W20251025 07:52:49.269423 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet a13bf4d7eb1440f7a28b85cb1bf99667 because the tablet was not found. No further retry: Not found: Tablet not found: a13bf4d7eb1440f7a28b85cb1bf99667
I20251025 07:52:49.270079 2686 raft_consensus.cc:493] T 5dab439bc50d4e869548154e1d222ad6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.270231 2686 raft_consensus.cc:515] T 5dab439bc50d4e869548154e1d222ad6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.270780 2686 leader_election.cc:290] T 5dab439bc50d4e869548154e1d222ad6 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
W20251025 07:52:49.271745 850 consensus_peers.cc:597] T 570201e7bf05406a97c81dde071d894b P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 570201e7bf05406a97c81dde071d894b. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:49.273028 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5dab439bc50d4e869548154e1d222ad6" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:49.273422 850 leader_election.cc:343] T 5dab439bc50d4e869548154e1d222ad6 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.273690 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c40b2e06d894a06af9e58285892ea8d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:49.273897 772 raft_consensus.cc:3060] T 1c40b2e06d894a06af9e58285892ea8d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.276685 772 raft_consensus.cc:2468] T 1c40b2e06d894a06af9e58285892ea8d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:49.281900 771 raft_consensus.cc:2468] T 2d24e3e7a9e4431db609331ad1fd57c9 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:49.282899 1166 tablet_bootstrap.cc:492] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:49.283032 1166 ts_tablet_manager.cc:1403] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20251025 07:52:49.283715 1166 raft_consensus.cc:359] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.283874 1166 raft_consensus.cc:385] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.283914 1166 raft_consensus.cc:740] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.284078 1166 consensus_queue.cc:260] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.284493 1166 ts_tablet_manager.cc:1434] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251025 07:52:49.284749 1166 tablet_bootstrap.cc:492] T 72dc084f8efc45478771c247e07a16f4 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:49.286406 1166 tablet_bootstrap.cc:654] T 72dc084f8efc45478771c247e07a16f4 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.289496 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 182) took 7796 ms. Trace:
I20251025 07:52:49.289642 2166 rpcz_store.cc:276] 1025 07:52:41.492916 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.493136 (+ 220us) service_pool.cc:225] Handling call
1025 07:52:48.926317 (+7433181us) ts_tablet_manager.cc:1151] Deleting tablet d9bfd7c0c3ad4e5b98b6708793d61d41
1025 07:52:48.926330 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:48.976936 (+ 50606us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.245718 (+268782us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.289478 (+ 43760us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7433029}
I20251025 07:52:49.290207 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet d9bfd7c0c3ad4e5b98b6708793d61d41 (table default.Table_0c74279815ab4a5186497b68bb932523 [id=5202bb531f384527a167bf9efe00d9c6]) successfully deleted
I20251025 07:52:49.290814 2166 tablet_replica.cc:333] T a4e4b99be0434baaa75d988281797846 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:49.291054 2166 raft_consensus.cc:2243] T a4e4b99be0434baaa75d988281797846 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:49.291277 2166 raft_consensus.cc:2272] T a4e4b99be0434baaa75d988281797846 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:49.291862 2166 ts_tablet_manager.cc:1916] T a4e4b99be0434baaa75d988281797846 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:49.295609 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5dab439bc50d4e869548154e1d222ad6" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:49.295773 768 raft_consensus.cc:2468] T 5dab439bc50d4e869548154e1d222ad6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:49.296141 2886 raft_consensus.cc:493] T 062c0f3e51a3487880cea373f8cf1f54 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.296258 2886 raft_consensus.cc:515] T 062c0f3e51a3487880cea373f8cf1f54 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.297188 2886 leader_election.cc:290] T 062c0f3e51a3487880cea373f8cf1f54 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:49.297451 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "062c0f3e51a3487880cea373f8cf1f54" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:49.297926 850 leader_election.cc:343] T 062c0f3e51a3487880cea373f8cf1f54 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.299275 1166 tablet_bootstrap.cc:492] T 72dc084f8efc45478771c247e07a16f4 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:49.299407 1166 ts_tablet_manager.cc:1403] T 72dc084f8efc45478771c247e07a16f4 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.002s
I20251025 07:52:49.299782 2886 consensus_queue.cc:1048] T e9e8fffb8e984463b5fa2c7dd9e93227 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:49.300076 1166 raft_consensus.cc:359] T 72dc084f8efc45478771c247e07a16f4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.300215 1166 raft_consensus.cc:385] T 72dc084f8efc45478771c247e07a16f4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.300273 1166 raft_consensus.cc:740] T 72dc084f8efc45478771c247e07a16f4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.300454 1166 consensus_queue.cc:260] T 72dc084f8efc45478771c247e07a16f4 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.300946 1166 ts_tablet_manager.cc:1434] T 72dc084f8efc45478771c247e07a16f4 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251025 07:52:49.303683 2886 consensus_queue.cc:1048] T e8402fd33ba842e981c6c68787c7c548 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:49.308419 2686 raft_consensus.cc:493] T d46673a283fe42b78ddeb99fbd2678eb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.308553 2686 raft_consensus.cc:515] T d46673a283fe42b78ddeb99fbd2678eb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.309057 2686 leader_election.cc:290] T d46673a283fe42b78ddeb99fbd2678eb P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:49.310281 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d46673a283fe42b78ddeb99fbd2678eb" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:49.310750 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22fc90c704f64d2f9f4b8c54dee45576" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:49.310904 910 raft_consensus.cc:2468] T 22fc90c704f64d2f9f4b8c54dee45576 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:49.310956 1166 tablet_bootstrap.cc:492] T 2e098c9c4613459295b6043048cba587 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:49.310926 847 leader_election.cc:304] T 1c40b2e06d894a06af9e58285892ea8d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.263541 1167 tablet_bootstrap.cc:492] T 9903e80a26fd469a8744dc855534ba75 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:49.312440 1169 tablet_bootstrap.cc:492] T 1b5ac7e845a24b2eb5b2429d47361232 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:49.312629 1169 ts_tablet_manager.cc:1403] T 1b5ac7e845a24b2eb5b2429d47361232 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.140s user 0.017s sys 0.012s
I20251025 07:52:49.312749 1166 tablet_bootstrap.cc:654] T 2e098c9c4613459295b6043048cba587 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.314479 1169 raft_consensus.cc:359] T 1b5ac7e845a24b2eb5b2429d47361232 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:49.314709 1169 raft_consensus.cc:385] T 1b5ac7e845a24b2eb5b2429d47361232 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.314865 2886 raft_consensus.cc:2804] T 1c40b2e06d894a06af9e58285892ea8d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
W20251025 07:52:49.314945 850 leader_election.cc:343] T d46673a283fe42b78ddeb99fbd2678eb P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.315503 2886 raft_consensus.cc:697] T 1c40b2e06d894a06af9e58285892ea8d P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:49.315739 2886 consensus_queue.cc:237] T 1c40b2e06d894a06af9e58285892ea8d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.314800 1169 raft_consensus.cc:740] T 1b5ac7e845a24b2eb5b2429d47361232 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.317466 1169 consensus_queue.cc:260] T 1b5ac7e845a24b2eb5b2429d47361232 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:49.317981 1169 ts_tablet_manager.cc:1434] T 1b5ac7e845a24b2eb5b2429d47361232 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251025 07:52:49.318301 1169 tablet_bootstrap.cc:492] T c11f674385184ae6b235f1db8bf1fb91 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:49.320003 1169 tablet_bootstrap.cc:654] T c11f674385184ae6b235f1db8bf1fb91 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.324164 2888 raft_consensus.cc:493] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.324309 2888 raft_consensus.cc:515] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.324877 2888 leader_election.cc:290] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:49.326020 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72dc084f8efc45478771c247e07a16f4" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:49.326454 850 consensus_peers.cc:597] T 6a33cfb2f2de48d4a5e9e39730d4900e P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251025 07:52:49.326517 712 leader_election.cc:343] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.323894 2169 rpcz_store.cc:276] 1025 07:52:44.664955 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.664991 (+ 36us) service_pool.cc:225] Handling call
1025 07:52:49.265690 (+4600699us) ts_tablet_manager.cc:1151] Deleting tablet c59f6e6eb9594a8e83f109a09b034045
1025 07:52:49.265698 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.265749 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4600582}
I20251025 07:52:49.329003 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 297) took 4663 ms. Trace:
I20251025 07:52:49.329365 2169 rpcz_store.cc:276] 1025 07:52:44.665191 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.665234 (+ 43us) service_pool.cc:225] Handling call
1025 07:52:49.328918 (+4663684us) ts_tablet_manager.cc:1151] Deleting tablet c88c46e022ff41a58b02538f2151e260
1025 07:52:49.328928 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.328989 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4663553}
W20251025 07:52:49.329733 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet c59f6e6eb9594a8e83f109a09b034045 because the tablet was not found. No further retry: Not found: Tablet not found: c59f6e6eb9594a8e83f109a09b034045
W20251025 07:52:49.330044 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet c88c46e022ff41a58b02538f2151e260 because the tablet was not found. No further retry: Not found: Tablet not found: c88c46e022ff41a58b02538f2151e260
I20251025 07:52:49.330566 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 298) took 4638 ms. Trace:
I20251025 07:52:49.330794 2990 raft_consensus.cc:493] T 075f0a151e024220b51ba7fcaecdcc3d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.330660 2169 rpcz_store.cc:276] 1025 07:52:44.691924 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.691978 (+ 54us) service_pool.cc:225] Handling call
1025 07:52:49.330026 (+4638048us) ts_tablet_manager.cc:1151] Deleting tablet 4b278ce7768349e49dcd1131e404ab34
1025 07:52:49.330034 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.330116 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4622497}
I20251025 07:52:49.330917 2990 raft_consensus.cc:515] T 075f0a151e024220b51ba7fcaecdcc3d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.331028 1166 tablet_bootstrap.cc:492] T 2e098c9c4613459295b6043048cba587 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:49.331171 1166 ts_tablet_manager.cc:1403] T 2e098c9c4613459295b6043048cba587 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.001s
I20251025 07:52:49.331285 2169 tablet_replica.cc:333] T 9f039d9107ae493486d799ab6a1d731d P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:49.331445 2990 leader_election.cc:290] T 075f0a151e024220b51ba7fcaecdcc3d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:49.331472 2169 raft_consensus.cc:2243] T 9f039d9107ae493486d799ab6a1d731d P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
W20251025 07:52:49.331892 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 4b278ce7768349e49dcd1131e404ab34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b278ce7768349e49dcd1131e404ab34 already in progress: creating tablet
I20251025 07:52:49.331916 1166 raft_consensus.cc:359] T 2e098c9c4613459295b6043048cba587 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:49.331965 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "075f0a151e024220b51ba7fcaecdcc3d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:49.332067 1166 raft_consensus.cc:385] T 2e098c9c4613459295b6043048cba587 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.332118 1166 raft_consensus.cc:740] T 2e098c9c4613459295b6043048cba587 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
W20251025 07:52:49.332332 850 leader_election.cc:343] T 075f0a151e024220b51ba7fcaecdcc3d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.332556 847 leader_election.cc:304] T 2d24e3e7a9e4431db609331ad1fd57c9 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.332938 2169 raft_consensus.cc:2272] T 9f039d9107ae493486d799ab6a1d731d P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:49.333456 2686 raft_consensus.cc:2804] T 2d24e3e7a9e4431db609331ad1fd57c9 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:49.333570 2686 raft_consensus.cc:697] T 2d24e3e7a9e4431db609331ad1fd57c9 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:49.333771 2686 consensus_queue.cc:237] T 2d24e3e7a9e4431db609331ad1fd57c9 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.332314 1166 consensus_queue.cc:260] T 2e098c9c4613459295b6043048cba587 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:49.335003 1166 ts_tablet_manager.cc:1434] T 2e098c9c4613459295b6043048cba587 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251025 07:52:49.341246 1166 tablet_bootstrap.cc:492] T df9db9dd31d840e0b1394db04d662a62 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:49.343639 1166 tablet_bootstrap.cc:654] T df9db9dd31d840e0b1394db04d662a62 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.345934 847 leader_election.cc:304] T 5dab439bc50d4e869548154e1d222ad6 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.346577 2169 ts_tablet_manager.cc:1916] T 9f039d9107ae493486d799ab6a1d731d P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:49.347311 2991 raft_consensus.cc:2804] T 5dab439bc50d4e869548154e1d222ad6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:49.347383 2991 raft_consensus.cc:493] T 5dab439bc50d4e869548154e1d222ad6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:49.347450 2991 raft_consensus.cc:3060] T 5dab439bc50d4e869548154e1d222ad6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.350001 2991 raft_consensus.cc:515] T 5dab439bc50d4e869548154e1d222ad6 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.350833 2991 leader_election.cc:290] T 5dab439bc50d4e869548154e1d222ad6 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:49.351097 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5dab439bc50d4e869548154e1d222ad6" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:49.351444 1167 tablet_bootstrap.cc:654] T 9903e80a26fd469a8744dc855534ba75 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:49.351542 850 leader_election.cc:343] T 5dab439bc50d4e869548154e1d222ad6 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.352937 2169 ts_tablet_manager.cc:1929] T 9f039d9107ae493486d799ab6a1d731d P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:49.353034 2169 log.cc:1199] T 9f039d9107ae493486d799ab6a1d731d P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/9f039d9107ae493486d799ab6a1d731d
I20251025 07:52:49.353334 2169 ts_tablet_manager.cc:1950] T 9f039d9107ae493486d799ab6a1d731d P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:49.354065 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 299) took 4655 ms. Trace:
I20251025 07:52:49.354176 2169 rpcz_store.cc:276] 1025 07:52:44.698893 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.698954 (+ 61us) service_pool.cc:225] Handling call
1025 07:52:49.331246 (+4632292us) ts_tablet_manager.cc:1151] Deleting tablet 9f039d9107ae493486d799ab6a1d731d
1025 07:52:49.331254 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.349978 (+ 18724us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.352863 (+ 2885us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.354045 (+ 1182us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4622743}
I20251025 07:52:49.354629 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 302) took 4587 ms. Trace:
I20251025 07:52:49.354730 2169 rpcz_store.cc:276] 1025 07:52:44.767056 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.773699 (+ 6643us) service_pool.cc:225] Handling call
1025 07:52:49.354493 (+4580794us) ts_tablet_manager.cc:1151] Deleting tablet 29dbaf9149134f548a974a5779f7da67
1025 07:52:49.354506 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.354612 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4579781}
I20251025 07:52:49.354990 2169 tablet_replica.cc:333] T 11bfe0c87ca941a3846462c8a4512d83 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:49.355039 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 9f039d9107ae493486d799ab6a1d731d (table default.Table_3b10c71d2c1b4bd2849e0cad9ec767d7 [id=ee09c3f16a194bbf97dd2768b5231c06]) successfully deleted
I20251025 07:52:49.355203 2169 raft_consensus.cc:2243] T 11bfe0c87ca941a3846462c8a4512d83 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
W20251025 07:52:49.355737 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 29dbaf9149134f548a974a5779f7da67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29dbaf9149134f548a974a5779f7da67 already in progress: creating tablet
I20251025 07:52:49.356390 2169 raft_consensus.cc:2272] T 11bfe0c87ca941a3846462c8a4512d83 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:49.356920 2169 ts_tablet_manager.cc:1916] T 11bfe0c87ca941a3846462c8a4512d83 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:49.362592 1166 tablet_bootstrap.cc:492] T df9db9dd31d840e0b1394db04d662a62 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:49.362721 1166 ts_tablet_manager.cc:1403] T df9db9dd31d840e0b1394db04d662a62 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20251025 07:52:49.363695 1166 raft_consensus.cc:359] T df9db9dd31d840e0b1394db04d662a62 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.363902 1166 raft_consensus.cc:385] T df9db9dd31d840e0b1394db04d662a62 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.363951 1166 raft_consensus.cc:740] T df9db9dd31d840e0b1394db04d662a62 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.364161 1166 consensus_queue.cc:260] T df9db9dd31d840e0b1394db04d662a62 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.365770 1166 ts_tablet_manager.cc:1434] T df9db9dd31d840e0b1394db04d662a62 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:49.366112 1166 tablet_bootstrap.cc:492] T 792f32cbd66449ca98813c669b2a261d P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:49.366108 2169 ts_tablet_manager.cc:1929] T 11bfe0c87ca941a3846462c8a4512d83 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:49.366222 2169 log.cc:1199] T 11bfe0c87ca941a3846462c8a4512d83 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/11bfe0c87ca941a3846462c8a4512d83
I20251025 07:52:49.366555 2169 ts_tablet_manager.cc:1950] T 11bfe0c87ca941a3846462c8a4512d83 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:49.367447 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 304) took 4568 ms. Trace:
I20251025 07:52:49.367556 2169 rpcz_store.cc:276] 1025 07:52:44.798943 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.799008 (+ 65us) service_pool.cc:225] Handling call
1025 07:52:49.354953 (+4555945us) ts_tablet_manager.cc:1151] Deleting tablet 11bfe0c87ca941a3846462c8a4512d83
1025 07:52:49.354962 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.362892 (+ 7930us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.366036 (+ 3144us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.367425 (+ 1389us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4552700}
I20251025 07:52:49.367990 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 307) took 4482 ms. Trace:
I20251025 07:52:49.368113 1166 tablet_bootstrap.cc:654] T 792f32cbd66449ca98813c669b2a261d P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.371044 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 11bfe0c87ca941a3846462c8a4512d83 (table default.Table_00b3b394e5ed46faaac0202afe54b437 [id=304ef5bba5ed4e198622b8c8dc17f43a]) successfully deleted
I20251025 07:52:49.374022 771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d46673a283fe42b78ddeb99fbd2678eb" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:49.374225 771 raft_consensus.cc:2468] T d46673a283fe42b78ddeb99fbd2678eb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:49.378990 710 leader_election.cc:304] T 22fc90c704f64d2f9f4b8c54dee45576 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.379400 2888 raft_consensus.cc:2804] T 22fc90c704f64d2f9f4b8c54dee45576 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:49.379472 2888 raft_consensus.cc:493] T 22fc90c704f64d2f9f4b8c54dee45576 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:49.379539 2888 raft_consensus.cc:3060] T 22fc90c704f64d2f9f4b8c54dee45576 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:49.379773 850 consensus_peers.cc:597] T e8402fd33ba842e981c6c68787c7c548 P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:49.380044 2991 raft_consensus.cc:493] T d0240d20f9904b4bbc8bf7f0f05e8850 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.380151 2991 raft_consensus.cc:515] T d0240d20f9904b4bbc8bf7f0f05e8850 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.380648 2991 leader_election.cc:290] T d0240d20f9904b4bbc8bf7f0f05e8850 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:49.381377 2169 rpcz_store.cc:276] 1025 07:52:44.885884 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.885936 (+ 52us) service_pool.cc:225] Handling call
1025 07:52:49.367894 (+4481958us) ts_tablet_manager.cc:1151] Deleting tablet 9d6a9937d10a4cc2813671dd1bfa1664
1025 07:52:49.367906 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.367973 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4479634}
I20251025 07:52:49.381735 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 308) took 4494 ms. Trace:
I20251025 07:52:49.381827 2169 rpcz_store.cc:276] 1025 07:52:44.887421 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.887470 (+ 49us) service_pool.cc:225] Handling call
1025 07:52:49.381649 (+4494179us) ts_tablet_manager.cc:1151] Deleting tablet d5e5e5bbcf9f43118eea502b69880af5
1025 07:52:49.381659 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.381722 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4493295}
I20251025 07:52:49.381880 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0240d20f9904b4bbc8bf7f0f05e8850" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:49.382145 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 309) took 4494 ms. Trace:
I20251025 07:52:49.382272 2169 rpcz_store.cc:276] 1025 07:52:44.888022 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.888072 (+ 50us) service_pool.cc:225] Handling call
1025 07:52:49.382067 (+4493995us) ts_tablet_manager.cc:1151] Deleting tablet 53198283718d4b45b30df65b5bde96a6
1025 07:52:49.382077 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.382130 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":113,"tablet-delete.queue_time_us":4493810}
I20251025 07:52:49.382310 2888 raft_consensus.cc:515] T 22fc90c704f64d2f9f4b8c54dee45576 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:49.382570 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 311) took 3927 ms. Trace:
I20251025 07:52:49.382649 2169 rpcz_store.cc:276] 1025 07:52:45.455385 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.455582 (+ 197us) service_pool.cc:225] Handling call
1025 07:52:49.382482 (+3926900us) ts_tablet_manager.cc:1151] Deleting tablet f3c87a83082b421cb4be546ac4ce7ad8
1025 07:52:49.382490 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.382535 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3926692}
I20251025 07:52:49.382902 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 312) took 3927 ms. Trace:
I20251025 07:52:49.382972 2169 rpcz_store.cc:276] 1025 07:52:45.455507 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.455674 (+ 167us) service_pool.cc:225] Handling call
1025 07:52:49.382836 (+3927162us) ts_tablet_manager.cc:1151] Deleting tablet 08da9d2687704725aacd11cd92c6a9f9
1025 07:52:49.382844 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.382890 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3926936}
I20251025 07:52:49.383220 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 313) took 3927 ms. Trace:
I20251025 07:52:49.383291 2169 rpcz_store.cc:276] 1025 07:52:45.455661 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.455945 (+ 284us) service_pool.cc:225] Handling call
1025 07:52:49.383150 (+3927205us) ts_tablet_manager.cc:1151] Deleting tablet 551d07a9242740dcb9366bb2d707a4a8
1025 07:52:49.383159 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.383207 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3927070}
I20251025 07:52:49.383520 2888 leader_election.cc:290] T 22fc90c704f64d2f9f4b8c54dee45576 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:49.383560 2169 tablet_replica.cc:333] T 0551e5d694894e8fbeb66b5e0c1cd75b P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:49.383744 2169 raft_consensus.cc:2243] T 0551e5d694894e8fbeb66b5e0c1cd75b P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:49.383809 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22fc90c704f64d2f9f4b8c54dee45576" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:49.383912 2169 raft_consensus.cc:2272] T 0551e5d694894e8fbeb66b5e0c1cd75b P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:49.383955 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 9d6a9937d10a4cc2813671dd1bfa1664 because the tablet was not found. No further retry: Not found: Tablet not found: 9d6a9937d10a4cc2813671dd1bfa1664
W20251025 07:52:49.384214 712 leader_election.cc:343] T 22fc90c704f64d2f9f4b8c54dee45576 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.384524 2169 ts_tablet_manager.cc:1916] T 0551e5d694894e8fbeb66b5e0c1cd75b P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:49.386502 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet d5e5e5bbcf9f43118eea502b69880af5 because the tablet was not found. No further retry: Not found: Tablet not found: d5e5e5bbcf9f43118eea502b69880af5
W20251025 07:52:49.386914 850 leader_election.cc:343] T d0240d20f9904b4bbc8bf7f0f05e8850 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:49.386940 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 53198283718d4b45b30df65b5bde96a6 because the tablet was not found. No further retry: Not found: Tablet not found: 53198283718d4b45b30df65b5bde96a6
W20251025 07:52:49.387202 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet f3c87a83082b421cb4be546ac4ce7ad8 because the tablet was not found. No further retry: Not found: Tablet not found: f3c87a83082b421cb4be546ac4ce7ad8
W20251025 07:52:49.387439 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 08da9d2687704725aacd11cd92c6a9f9 because the tablet was not found. No further retry: Not found: Tablet not found: 08da9d2687704725aacd11cd92c6a9f9
W20251025 07:52:49.387775 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 551d07a9242740dcb9366bb2d707a4a8 because the tablet was not found. No further retry: Not found: Tablet not found: 551d07a9242740dcb9366bb2d707a4a8
I20251025 07:52:49.389732 1166 tablet_bootstrap.cc:492] T 792f32cbd66449ca98813c669b2a261d P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:49.389860 1166 ts_tablet_manager.cc:1403] T 792f32cbd66449ca98813c669b2a261d P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20251025 07:52:49.390595 1166 raft_consensus.cc:359] T 792f32cbd66449ca98813c669b2a261d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.390798 1166 raft_consensus.cc:385] T 792f32cbd66449ca98813c669b2a261d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.390854 1166 raft_consensus.cc:740] T 792f32cbd66449ca98813c669b2a261d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.391099 1166 consensus_queue.cc:260] T 792f32cbd66449ca98813c669b2a261d P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.391677 1166 ts_tablet_manager.cc:1434] T 792f32cbd66449ca98813c669b2a261d P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:49.391958 1166 tablet_bootstrap.cc:492] T 6ce62847e80f48e8807fdbbbdc50803c P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:49.392032 2169 ts_tablet_manager.cc:1929] T 0551e5d694894e8fbeb66b5e0c1cd75b P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:49.392143 2169 log.cc:1199] T 0551e5d694894e8fbeb66b5e0c1cd75b P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/0551e5d694894e8fbeb66b5e0c1cd75b
I20251025 07:52:49.392508 2169 ts_tablet_manager.cc:1950] T 0551e5d694894e8fbeb66b5e0c1cd75b P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:49.393261 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 314) took 3763 ms. Trace:
I20251025 07:52:49.393342 2169 rpcz_store.cc:276] 1025 07:52:45.629657 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.629716 (+ 59us) service_pool.cc:225] Handling call
1025 07:52:49.383519 (+3753803us) ts_tablet_manager.cc:1151] Deleting tablet 0551e5d694894e8fbeb66b5e0c1cd75b
1025 07:52:49.383527 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.388931 (+ 5404us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.391943 (+ 3012us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.393240 (+ 1297us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3753609}
I20251025 07:52:49.393486 1166 tablet_bootstrap.cc:654] T 6ce62847e80f48e8807fdbbbdc50803c P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.394397 2888 raft_consensus.cc:493] T 3e79d69d5d004cc395b8e35569e25440 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.394697 2888 raft_consensus.cc:515] T 3e79d69d5d004cc395b8e35569e25440 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.395787 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e79d69d5d004cc395b8e35569e25440" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:49.395972 2888 leader_election.cc:290] T 3e79d69d5d004cc395b8e35569e25440 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:49.396251 712 leader_election.cc:343] T 3e79d69d5d004cc395b8e35569e25440 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.398751 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 315) took 3749 ms. Trace:
I20251025 07:52:49.398861 2169 rpcz_store.cc:276] 1025 07:52:45.649483 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.649542 (+ 59us) service_pool.cc:225] Handling call
1025 07:52:49.398589 (+3749047us) ts_tablet_manager.cc:1151] Deleting tablet 110a8e9e4c6c434284c0ae14dd6987f7
1025 07:52:49.398603 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.398729 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3748796}
I20251025 07:52:49.401312 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 0551e5d694894e8fbeb66b5e0c1cd75b (table default.Table_2cd70245aa3d4809a0647e7fa499f62f [id=a003268ec3da425989fa4041699de4d3]) successfully deleted
W20251025 07:52:49.401957 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 110a8e9e4c6c434284c0ae14dd6987f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 110a8e9e4c6c434284c0ae14dd6987f7 already in progress: creating tablet
I20251025 07:52:49.402889 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 316) took 3737 ms. Trace:
I20251025 07:52:49.402971 2169 rpcz_store.cc:276] 1025 07:52:45.665731 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.665784 (+ 53us) service_pool.cc:225] Handling call
1025 07:52:49.402764 (+3736980us) ts_tablet_manager.cc:1151] Deleting tablet 623dec4eab1d4666baf2334a01eeb36d
1025 07:52:49.402776 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.402873 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3736807}
I20251025 07:52:49.403363 2169 tablet_replica.cc:333] T cb829649f73d4d3f8862d268c953bd86 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:49.403558 2169 raft_consensus.cc:2243] T cb829649f73d4d3f8862d268c953bd86 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
W20251025 07:52:49.403566 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 623dec4eab1d4666baf2334a01eeb36d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 623dec4eab1d4666baf2334a01eeb36d already in progress: creating tablet
I20251025 07:52:49.403760 2169 raft_consensus.cc:2272] T cb829649f73d4d3f8862d268c953bd86 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:49.404309 2169 ts_tablet_manager.cc:1916] T cb829649f73d4d3f8862d268c953bd86 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:49.404754 765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "075f0a151e024220b51ba7fcaecdcc3d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:49.404933 765 raft_consensus.cc:2468] T 075f0a151e024220b51ba7fcaecdcc3d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:49.408114 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72dc084f8efc45478771c247e07a16f4" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:49.408335 910 raft_consensus.cc:2468] T 72dc084f8efc45478771c247e07a16f4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:49.408601 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e79d69d5d004cc395b8e35569e25440" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:49.408701 765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0240d20f9904b4bbc8bf7f0f05e8850" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:49.408838 765 raft_consensus.cc:2468] T d0240d20f9904b4bbc8bf7f0f05e8850 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:49.408877 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22fc90c704f64d2f9f4b8c54dee45576" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:49.409061 909 raft_consensus.cc:3060] T 22fc90c704f64d2f9f4b8c54dee45576 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.409137 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5dab439bc50d4e869548154e1d222ad6" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:49.409256 768 raft_consensus.cc:3060] T 5dab439bc50d4e869548154e1d222ad6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.411870 909 raft_consensus.cc:2468] T 22fc90c704f64d2f9f4b8c54dee45576 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:49.412187 768 raft_consensus.cc:2468] T 5dab439bc50d4e869548154e1d222ad6 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:49.414008 1166 tablet_bootstrap.cc:492] T 6ce62847e80f48e8807fdbbbdc50803c P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:49.414201 1166 ts_tablet_manager.cc:1403] T 6ce62847e80f48e8807fdbbbdc50803c P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.002s
I20251025 07:52:49.414283 2169 ts_tablet_manager.cc:1929] T cb829649f73d4d3f8862d268c953bd86 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 07:52:49.414392 2169 log.cc:1199] T cb829649f73d4d3f8862d268c953bd86 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/cb829649f73d4d3f8862d268c953bd86
I20251025 07:52:49.414744 2169 ts_tablet_manager.cc:1950] T cb829649f73d4d3f8862d268c953bd86 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:49.415062 1166 raft_consensus.cc:359] T 6ce62847e80f48e8807fdbbbdc50803c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.415231 1166 raft_consensus.cc:385] T 6ce62847e80f48e8807fdbbbdc50803c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.415307 1166 raft_consensus.cc:740] T 6ce62847e80f48e8807fdbbbdc50803c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.415537 1166 consensus_queue.cc:260] T 6ce62847e80f48e8807fdbbbdc50803c P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.416425 1166 ts_tablet_manager.cc:1434] T 6ce62847e80f48e8807fdbbbdc50803c P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:49.417274 1166 tablet_bootstrap.cc:492] T 8917a289bd9b4428a4603429edd94581 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:49.418927 1166 tablet_bootstrap.cc:654] T 8917a289bd9b4428a4603429edd94581 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.419312 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 317) took 3708 ms. Trace:
I20251025 07:52:49.419407 2169 rpcz_store.cc:276] 1025 07:52:45.710606 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.710718 (+ 112us) service_pool.cc:225] Handling call
1025 07:52:49.403294 (+3692576us) ts_tablet_manager.cc:1151] Deleting tablet cb829649f73d4d3f8862d268c953bd86
1025 07:52:49.403302 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.407482 (+ 4180us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.414174 (+ 6692us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.419292 (+ 5118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3692380}
I20251025 07:52:49.420856 2169 tablet_replica.cc:333] T ebf3a2a926754305b5bd95fcebb25d29 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:49.421056 2169 raft_consensus.cc:2243] T ebf3a2a926754305b5bd95fcebb25d29 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:49.421207 2169 raft_consensus.cc:2272] T ebf3a2a926754305b5bd95fcebb25d29 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:49.421728 2169 ts_tablet_manager.cc:1916] T ebf3a2a926754305b5bd95fcebb25d29 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:49.422020 2146 ts_tablet_manager.cc:1929] T fce535a0c0bd45c6bf17f81add297f40 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:49.422297 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet cb829649f73d4d3f8862d268c953bd86 (table default.Table_a9026f5e095c4063b86d4c544c686e22 [id=7944225d3ce7439b881bfb9289d9f040]) successfully deleted
I20251025 07:52:49.436403 1167 tablet_bootstrap.cc:492] T 9903e80a26fd469a8744dc855534ba75 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:49.436707 1167 ts_tablet_manager.cc:1403] T 9903e80a26fd469a8744dc855534ba75 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.173s user 0.004s sys 0.000s
I20251025 07:52:49.437821 1167 raft_consensus.cc:359] T 9903e80a26fd469a8744dc855534ba75 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.438090 1167 raft_consensus.cc:385] T 9903e80a26fd469a8744dc855534ba75 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.438275 1167 raft_consensus.cc:740] T 9903e80a26fd469a8744dc855534ba75 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.438575 1167 consensus_queue.cc:260] T 9903e80a26fd469a8744dc855534ba75 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.447757 2169 ts_tablet_manager.cc:1929] T ebf3a2a926754305b5bd95fcebb25d29 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:49.447889 2169 log.cc:1199] T ebf3a2a926754305b5bd95fcebb25d29 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/ebf3a2a926754305b5bd95fcebb25d29
I20251025 07:52:49.448274 2169 ts_tablet_manager.cc:1950] T ebf3a2a926754305b5bd95fcebb25d29 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:49.449056 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 318) took 3726 ms. Trace:
I20251025 07:52:49.449157 2169 rpcz_store.cc:276] 1025 07:52:45.723096 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.723177 (+ 81us) service_pool.cc:225] Handling call
1025 07:52:49.420810 (+3697633us) ts_tablet_manager.cc:1151] Deleting tablet ebf3a2a926754305b5bd95fcebb25d29
1025 07:52:49.420820 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.437712 (+ 16892us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.447621 (+ 9909us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.449035 (+ 1414us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3697439}
I20251025 07:52:49.450101 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet ebf3a2a926754305b5bd95fcebb25d29 (table default.Table_ac4beca0c3bd4cf29565faf5323d9ff7 [id=857bcd1dbe2e477bbc0d77f79a58f7f7]) successfully deleted
I20251025 07:52:49.451251 1169 tablet_bootstrap.cc:492] T c11f674385184ae6b235f1db8bf1fb91 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:49.451483 1169 ts_tablet_manager.cc:1403] T c11f674385184ae6b235f1db8bf1fb91 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.133s user 0.023s sys 0.043s
I20251025 07:52:49.452234 1169 raft_consensus.cc:359] T c11f674385184ae6b235f1db8bf1fb91 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:49.452468 1169 raft_consensus.cc:385] T c11f674385184ae6b235f1db8bf1fb91 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.452575 1169 raft_consensus.cc:740] T c11f674385184ae6b235f1db8bf1fb91 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.452860 1169 consensus_queue.cc:260] T c11f674385184ae6b235f1db8bf1fb91 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:49.454073 3011 raft_consensus.cc:493] T fa826b85f0c94150a17d4e417dd3d62f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.454195 3011 raft_consensus.cc:515] T fa826b85f0c94150a17d4e417dd3d62f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:49.454670 3011 leader_election.cc:290] T fa826b85f0c94150a17d4e417dd3d62f P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:49.455871 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa826b85f0c94150a17d4e417dd3d62f" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:49.456437 2169 tablet_replica.cc:333] T ef1854d35a3047e3a9fbe0166ab1295a P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:49.456629 2169 raft_consensus.cc:2243] T ef1854d35a3047e3a9fbe0166ab1295a P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:49.456935 2169 raft_consensus.cc:2272] T ef1854d35a3047e3a9fbe0166ab1295a P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:49.457482 2169 ts_tablet_manager.cc:1916] T ef1854d35a3047e3a9fbe0166ab1295a P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:49.458789 1169 ts_tablet_manager.cc:1434] T c11f674385184ae6b235f1db8bf1fb91 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251025 07:52:49.460136 1169 tablet_bootstrap.cc:492] T 6a33cfb2f2de48d4a5e9e39730d4900e P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:49.461748 1169 tablet_bootstrap.cc:654] T 6a33cfb2f2de48d4a5e9e39730d4900e P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.464010 3126 raft_consensus.cc:493] T 0582ee42ab564c07ae3facfc5935617c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.464161 3126 raft_consensus.cc:515] T 0582ee42ab564c07ae3facfc5935617c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.464795 3126 leader_election.cc:290] T 0582ee42ab564c07ae3facfc5935617c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:49.465286 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0582ee42ab564c07ae3facfc5935617c" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:49.430320 1166 tablet_bootstrap.cc:492] T 8917a289bd9b4428a4603429edd94581 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:49.465660 1166 ts_tablet_manager.cc:1403] T 8917a289bd9b4428a4603429edd94581 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.048s user 0.006s sys 0.000s
I20251025 07:52:49.466610 1166 raft_consensus.cc:359] T 8917a289bd9b4428a4603429edd94581 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.466850 1166 raft_consensus.cc:385] T 8917a289bd9b4428a4603429edd94581 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.466938 1166 raft_consensus.cc:740] T 8917a289bd9b4428a4603429edd94581 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.467199 1166 consensus_queue.cc:260] T 8917a289bd9b4428a4603429edd94581 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.467703 1166 ts_tablet_manager.cc:1434] T 8917a289bd9b4428a4603429edd94581 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:49.468091 1166 tablet_bootstrap.cc:492] T 48b73e0c0dce4be686d05f991a21bd19 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:49.469938 1166 tablet_bootstrap.cc:654] T 48b73e0c0dce4be686d05f991a21bd19 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:49.470329 712 leader_election.cc:343] T 0582ee42ab564c07ae3facfc5935617c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:49.452225 847 leader_election.cc:343] T 062c0f3e51a3487880cea373f8cf1f54 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: 062c0f3e51a3487880cea373f8cf1f54
I20251025 07:52:49.455258 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa826b85f0c94150a17d4e417dd3d62f" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:49.471307 847 leader_election.cc:304] T 062c0f3e51a3487880cea373f8cf1f54 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.463445 2686 raft_consensus.cc:493] T 3b940fa5ae26427dafbcfd26372754b3 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.471544 2990 raft_consensus.cc:2749] T 062c0f3e51a3487880cea373f8cf1f54 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:49.471684 2686 raft_consensus.cc:515] T 3b940fa5ae26427dafbcfd26372754b3 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.471798 847 leader_election.cc:304] T d46673a283fe42b78ddeb99fbd2678eb P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.472064 2990 raft_consensus.cc:2804] T d46673a283fe42b78ddeb99fbd2678eb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:49.472162 2990 raft_consensus.cc:493] T d46673a283fe42b78ddeb99fbd2678eb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:49.472271 2990 raft_consensus.cc:3060] T d46673a283fe42b78ddeb99fbd2678eb P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.472344 2686 leader_election.cc:290] T 3b940fa5ae26427dafbcfd26372754b3 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:49.472666 983 leader_election.cc:304] T fa826b85f0c94150a17d4e417dd3d62f P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:49.472705 2886 consensus_queue.cc:1048] T dff7f86ae8924685b26011e504e2247b P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:49.473052 3011 raft_consensus.cc:2749] T fa826b85f0c94150a17d4e417dd3d62f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:49.473449 2886 consensus_queue.cc:1048] T e9e8fffb8e984463b5fa2c7dd9e93227 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:49.475945 2990 raft_consensus.cc:515] T d46673a283fe42b78ddeb99fbd2678eb P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.476547 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b940fa5ae26427dafbcfd26372754b3" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:49.477181 850 leader_election.cc:343] T 3b940fa5ae26427dafbcfd26372754b3 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.478552 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d46673a283fe42b78ddeb99fbd2678eb" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:49.479152 850 leader_election.cc:343] T d46673a283fe42b78ddeb99fbd2678eb P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.479461 2990 leader_election.cc:290] T d46673a283fe42b78ddeb99fbd2678eb P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:49.479566 1166 tablet_bootstrap.cc:492] T 48b73e0c0dce4be686d05f991a21bd19 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:49.479763 1166 ts_tablet_manager.cc:1403] T 48b73e0c0dce4be686d05f991a21bd19 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20251025 07:52:49.480791 1166 raft_consensus.cc:359] T 48b73e0c0dce4be686d05f991a21bd19 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:49.480912 2886 raft_consensus.cc:493] T df0d86ad301245c1a6bcb44805936d25 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.480981 1166 raft_consensus.cc:385] T 48b73e0c0dce4be686d05f991a21bd19 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.481072 1166 raft_consensus.cc:740] T 48b73e0c0dce4be686d05f991a21bd19 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.481050 2886 raft_consensus.cc:515] T df0d86ad301245c1a6bcb44805936d25 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.481585 2886 leader_election.cc:290] T df0d86ad301245c1a6bcb44805936d25 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:49.482211 2146 log.cc:1199] T fce535a0c0bd45c6bf17f81add297f40 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/fce535a0c0bd45c6bf17f81add297f40
I20251025 07:52:49.482100 1166 consensus_queue.cc:260] T 48b73e0c0dce4be686d05f991a21bd19 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:49.482225 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df0d86ad301245c1a6bcb44805936d25" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:49.482074 847 leader_election.cc:304] T 075f0a151e024220b51ba7fcaecdcc3d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.482702 1166 ts_tablet_manager.cc:1434] T 48b73e0c0dce4be686d05f991a21bd19 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251025 07:52:49.483039 2146 ts_tablet_manager.cc:1950] T fce535a0c0bd45c6bf17f81add297f40 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:49.483440 2990 raft_consensus.cc:2804] T 075f0a151e024220b51ba7fcaecdcc3d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20251025 07:52:49.483502 850 leader_election.cc:343] T df0d86ad301245c1a6bcb44805936d25 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.483605 2990 raft_consensus.cc:493] T 075f0a151e024220b51ba7fcaecdcc3d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:49.483773 2990 raft_consensus.cc:3060] T 075f0a151e024220b51ba7fcaecdcc3d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.486500 2990 raft_consensus.cc:515] T 075f0a151e024220b51ba7fcaecdcc3d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.487412 2990 leader_election.cc:290] T 075f0a151e024220b51ba7fcaecdcc3d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:49.487493 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "075f0a151e024220b51ba7fcaecdcc3d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:49.487955 847 leader_election.cc:304] T d0240d20f9904b4bbc8bf7f0f05e8850 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:49.488085 850 leader_election.cc:343] T 075f0a151e024220b51ba7fcaecdcc3d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.488881 2166 ts_tablet_manager.cc:1929] T a4e4b99be0434baaa75d988281797846 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:49.489940 3122 raft_consensus.cc:2804] T d0240d20f9904b4bbc8bf7f0f05e8850 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:49.490037 3122 raft_consensus.cc:493] T d0240d20f9904b4bbc8bf7f0f05e8850 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:49.490147 3122 raft_consensus.cc:3060] T d0240d20f9904b4bbc8bf7f0f05e8850 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.490444 1166 tablet_bootstrap.cc:492] T 4b278ce7768349e49dcd1131e404ab34 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:49.492805 3122 raft_consensus.cc:515] T d0240d20f9904b4bbc8bf7f0f05e8850 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.493558 3122 leader_election.cc:290] T d0240d20f9904b4bbc8bf7f0f05e8850 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:49.493794 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0240d20f9904b4bbc8bf7f0f05e8850" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:49.494393 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 260) took 5721 ms. Trace:
W20251025 07:52:49.494499 850 leader_election.cc:343] T d0240d20f9904b4bbc8bf7f0f05e8850 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.494534 2146 rpcz_store.cc:276] 1025 07:52:43.773398 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:43.773448 (+ 50us) service_pool.cc:225] Handling call
1025 07:52:49.150255 (+5376807us) ts_tablet_manager.cc:1151] Deleting tablet fce535a0c0bd45c6bf17f81add297f40
1025 07:52:49.150266 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.307253 (+156987us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.421884 (+114631us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.494339 (+ 72455us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":581,"tablet-delete.queue_time_us":5376612}
I20251025 07:52:49.495115 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet fce535a0c0bd45c6bf17f81add297f40 (table default.Table_1de0b151cf4c4c10aca35e45623caf2c [id=a947f21398254a53bc6668c4f8d39e53]) successfully deleted
I20251025 07:52:49.495669 847 leader_election.cc:304] T 5dab439bc50d4e869548154e1d222ad6 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.496110 2166 log.cc:1199] T a4e4b99be0434baaa75d988281797846 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/a4e4b99be0434baaa75d988281797846
I20251025 07:52:49.496734 2166 ts_tablet_manager.cc:1950] T a4e4b99be0434baaa75d988281797846 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:49.497555 3122 raft_consensus.cc:2804] T 5dab439bc50d4e869548154e1d222ad6 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:49.497769 3122 raft_consensus.cc:697] T 5dab439bc50d4e869548154e1d222ad6 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:49.498096 3122 consensus_queue.cc:237] T 5dab439bc50d4e869548154e1d222ad6 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.499653 1166 tablet_bootstrap.cc:654] T 4b278ce7768349e49dcd1131e404ab34 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:49.499867 439 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:42203, user_credentials={real_user=slave}} blocked reactor thread for 46766.7us
I20251025 07:52:49.500344 2169 ts_tablet_manager.cc:1929] T ef1854d35a3047e3a9fbe0166ab1295a P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 07:52:49.500465 2169 log.cc:1199] T ef1854d35a3047e3a9fbe0166ab1295a P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/ef1854d35a3047e3a9fbe0166ab1295a
I20251025 07:52:49.500805 2169 ts_tablet_manager.cc:1950] T ef1854d35a3047e3a9fbe0166ab1295a P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:49.501328 1169 tablet_bootstrap.cc:492] T 6a33cfb2f2de48d4a5e9e39730d4900e P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:49.501456 1169 ts_tablet_manager.cc:1403] T 6a33cfb2f2de48d4a5e9e39730d4900e P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.041s user 0.018s sys 0.018s
I20251025 07:52:49.501538 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 319) took 3774 ms. Trace:
I20251025 07:52:49.501612 2169 rpcz_store.cc:276] 1025 07:52:45.727357 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.727421 (+ 64us) service_pool.cc:225] Handling call
1025 07:52:49.456386 (+3728965us) ts_tablet_manager.cc:1151] Deleting tablet ef1854d35a3047e3a9fbe0166ab1295a
1025 07:52:49.456396 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.485089 (+ 28693us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.500227 (+ 15138us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.501522 (+ 1295us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3728772}
I20251025 07:52:49.501984 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 320) took 3622 ms. Trace:
I20251025 07:52:49.502074 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet ef1854d35a3047e3a9fbe0166ab1295a (table default.Table_088aad2a46b041bfa29ed8e6929760b1 [id=834669c54dae449790818684fd8659bc]) successfully deleted
I20251025 07:52:49.502066 2169 rpcz_store.cc:276] 1025 07:52:45.879506 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.879756 (+ 250us) service_pool.cc:225] Handling call
1025 07:52:49.501869 (+3622113us) ts_tablet_manager.cc:1151] Deleting tablet f69cdaa7ee094283890036c0189e2c81
1025 07:52:49.501879 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.501969 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3621549}
I20251025 07:52:49.502350 2169 tablet_replica.cc:333] T df1248f6dbd94af68c587da7d149e279 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:49.502579 2169 raft_consensus.cc:2243] T df1248f6dbd94af68c587da7d149e279 P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
W20251025 07:52:49.502841 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet f69cdaa7ee094283890036c0189e2c81 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f69cdaa7ee094283890036c0189e2c81 already in progress: creating tablet
I20251025 07:52:49.502882 2169 raft_consensus.cc:2272] T df1248f6dbd94af68c587da7d149e279 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:49.503391 2169 ts_tablet_manager.cc:1916] T df1248f6dbd94af68c587da7d149e279 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:49.503337 1169 raft_consensus.cc:359] T 6a33cfb2f2de48d4a5e9e39730d4900e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:49.503491 1169 raft_consensus.cc:385] T 6a33cfb2f2de48d4a5e9e39730d4900e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.503546 1169 raft_consensus.cc:740] T 6a33cfb2f2de48d4a5e9e39730d4900e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.503773 1169 consensus_queue.cc:260] T 6a33cfb2f2de48d4a5e9e39730d4900e P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:49.507390 1169 ts_tablet_manager.cc:1434] T 6a33cfb2f2de48d4a5e9e39730d4900e P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251025 07:52:49.508091 2886 consensus_queue.cc:1048] T 645263a61b654f04985c40d04506c087 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:49.508379 1169 tablet_bootstrap.cc:492] T e8402fd33ba842e981c6c68787c7c548 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:49.512547 1169 tablet_bootstrap.cc:654] T e8402fd33ba842e981c6c68787c7c548 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.489452 1167 ts_tablet_manager.cc:1434] T 9903e80a26fd469a8744dc855534ba75 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.053s user 0.002s sys 0.000s
I20251025 07:52:49.509889 710 leader_election.cc:304] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.516659 1167 tablet_bootstrap.cc:492] T aea8fcf0154a4bd4a05b19e27ab78b79 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
W20251025 07:52:49.516917 710 leader_election.cc:343] T 3e79d69d5d004cc395b8e35569e25440 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.517061 710 leader_election.cc:304] T 3e79d69d5d004cc395b8e35569e25440 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.518306 1167 tablet_bootstrap.cc:654] T aea8fcf0154a4bd4a05b19e27ab78b79 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.519934 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 261) took 5607 ms. Trace:
I20251025 07:52:49.520093 2146 rpcz_store.cc:276] 1025 07:52:43.912447 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:43.912514 (+ 67us) service_pool.cc:225] Handling call
1025 07:52:49.519736 (+5607222us) ts_tablet_manager.cc:1151] Deleting tablet 32a143a715cb485cb4501f4e28c68252
1025 07:52:49.519747 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.519874 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5607020}
I20251025 07:52:49.520689 1169 tablet_bootstrap.cc:492] T e8402fd33ba842e981c6c68787c7c548 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:49.520802 1169 ts_tablet_manager.cc:1403] T e8402fd33ba842e981c6c68787c7c548 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.007s
W20251025 07:52:49.520725 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 32a143a715cb485cb4501f4e28c68252 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32a143a715cb485cb4501f4e28c68252 already in progress: creating tablet
I20251025 07:52:49.521250 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 262) took 5534 ms. Trace:
I20251025 07:52:49.521332 3011 raft_consensus.cc:493] T 6a33cfb2f2de48d4a5e9e39730d4900e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.521442 3011 raft_consensus.cc:515] T 6a33cfb2f2de48d4a5e9e39730d4900e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:49.521391 2146 rpcz_store.cc:276] 1025 07:52:43.987147 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:43.987218 (+ 71us) service_pool.cc:225] Handling call
1025 07:52:49.521080 (+5533862us) ts_tablet_manager.cc:1151] Deleting tablet 0540d6a10cf04fa89756623e9aaaeb68
1025 07:52:49.521094 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.521198 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5530158}
I20251025 07:52:49.521708 2888 raft_consensus.cc:2804] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:49.521778 2888 raft_consensus.cc:493] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:49.521837 2888 raft_consensus.cc:3060] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:49.521934 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 0540d6a10cf04fa89756623e9aaaeb68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0540d6a10cf04fa89756623e9aaaeb68 already in progress: creating tablet
I20251025 07:52:49.522010 3011 leader_election.cc:290] T 6a33cfb2f2de48d4a5e9e39730d4900e P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:49.522751 1169 raft_consensus.cc:359] T e8402fd33ba842e981c6c68787c7c548 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:49.522884 1169 raft_consensus.cc:385] T e8402fd33ba842e981c6c68787c7c548 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.522926 1169 raft_consensus.cc:740] T e8402fd33ba842e981c6c68787c7c548 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.523115 1169 consensus_queue.cc:260] T e8402fd33ba842e981c6c68787c7c548 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:49.524458 2888 raft_consensus.cc:515] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.524556 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 183) took 8031 ms. Trace:
I20251025 07:52:49.524637 2166 rpcz_store.cc:276] 1025 07:52:41.493000 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.493320 (+ 320us) service_pool.cc:225] Handling call
1025 07:52:49.290766 (+7797446us) ts_tablet_manager.cc:1151] Deleting tablet a4e4b99be0434baaa75d988281797846
1025 07:52:49.290776 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.422639 (+131863us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.488778 (+ 66139us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.524536 (+ 35758us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7797325}
I20251025 07:52:49.524959 2888 leader_election.cc:290] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:49.525152 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet a4e4b99be0434baaa75d988281797846 (table default.Table_dac528b081844b7c86027169c6234c27 [id=b0caa78ee8c84aa8bea4818cf9feb70c]) successfully deleted
I20251025 07:52:49.525219 2888 raft_consensus.cc:2749] T 3e79d69d5d004cc395b8e35569e25440 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:49.525832 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72dc084f8efc45478771c247e07a16f4" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:49.526312 712 leader_election.cc:343] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.526294 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a33cfb2f2de48d4a5e9e39730d4900e" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:49.526645 1169 ts_tablet_manager.cc:1434] T e8402fd33ba842e981c6c68787c7c548 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20251025 07:52:49.527645 1169 tablet_bootstrap.cc:492] T d77704b4fe6c4456a2ba40ffec0c60fc P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:49.528470 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "075f0a151e024220b51ba7fcaecdcc3d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:49.528611 773 raft_consensus.cc:3060] T 075f0a151e024220b51ba7fcaecdcc3d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.529004 1169 tablet_bootstrap.cc:654] T d77704b4fe6c4456a2ba40ffec0c60fc P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.529152 765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d46673a283fe42b78ddeb99fbd2678eb" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:49.529299 765 raft_consensus.cc:3060] T d46673a283fe42b78ddeb99fbd2678eb P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.530078 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a33cfb2f2de48d4a5e9e39730d4900e" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:49.530295 774 raft_consensus.cc:2393] T 6a33cfb2f2de48d4a5e9e39730d4900e P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:49.531540 773 raft_consensus.cc:2468] T 075f0a151e024220b51ba7fcaecdcc3d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:49.532145 765 raft_consensus.cc:2468] T d46673a283fe42b78ddeb99fbd2678eb P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:49.532387 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df0d86ad301245c1a6bcb44805936d25" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:49.532516 770 raft_consensus.cc:2468] T df0d86ad301245c1a6bcb44805936d25 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:49.532770 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0240d20f9904b4bbc8bf7f0f05e8850" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:49.532883 775 raft_consensus.cc:3060] T d0240d20f9904b4bbc8bf7f0f05e8850 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.536144 775 raft_consensus.cc:2468] T d0240d20f9904b4bbc8bf7f0f05e8850 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:49.545431 983 leader_election.cc:304] T 6a33cfb2f2de48d4a5e9e39730d4900e P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:49.545753 3011 raft_consensus.cc:3060] T 6a33cfb2f2de48d4a5e9e39730d4900e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.546427 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 268) took 5036 ms. Trace:
I20251025 07:52:49.547179 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 184) took 8053 ms. Trace:
I20251025 07:52:49.547299 2166 rpcz_store.cc:276] 1025 07:52:41.493578 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.493616 (+ 38us) service_pool.cc:225] Handling call
1025 07:52:49.547084 (+8053468us) ts_tablet_manager.cc:1151] Deleting tablet 80ee75935d7e461d9d4066fac9819bef
1025 07:52:49.547094 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.547160 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8053342}
W20251025 07:52:49.547948 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 80ee75935d7e461d9d4066fac9819bef because the tablet was not found. No further retry: Not found: Tablet not found: 80ee75935d7e461d9d4066fac9819bef
I20251025 07:52:49.548391 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 185) took 8054 ms. Trace:
I20251025 07:52:49.548710 2166 rpcz_store.cc:276] 1025 07:52:41.493866 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.493906 (+ 40us) service_pool.cc:225] Handling call
1025 07:52:49.548301 (+8054395us) ts_tablet_manager.cc:1151] Deleting tablet b94634ac5077439fadfbbce569f60e8b
1025 07:52:49.548309 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.548364 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8054268}
I20251025 07:52:49.549069 3011 raft_consensus.cc:2749] T 6a33cfb2f2de48d4a5e9e39730d4900e P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:49.549255 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet b94634ac5077439fadfbbce569f60e8b because the tablet was not found. No further retry: Not found: Tablet not found: b94634ac5077439fadfbbce569f60e8b
I20251025 07:52:49.549612 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 186) took 8055 ms. Trace:
I20251025 07:52:49.549675 2166 rpcz_store.cc:276] 1025 07:52:41.494195 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.494236 (+ 41us) service_pool.cc:225] Handling call
1025 07:52:49.549545 (+8055309us) ts_tablet_manager.cc:1151] Deleting tablet 92aa605fabbc4257a6a204c087e7cc7f
1025 07:52:49.549553 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.549600 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8055184}
W20251025 07:52:49.550015 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 92aa605fabbc4257a6a204c087e7cc7f because the tablet was not found. No further retry: Not found: Tablet not found: 92aa605fabbc4257a6a204c087e7cc7f
I20251025 07:52:49.551347 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0582ee42ab564c07ae3facfc5935617c" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:49.553345 1167 tablet_bootstrap.cc:492] T aea8fcf0154a4bd4a05b19e27ab78b79 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:49.553496 1167 ts_tablet_manager.cc:1403] T aea8fcf0154a4bd4a05b19e27ab78b79 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.001s
I20251025 07:52:49.553989 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 187) took 8059 ms. Trace:
I20251025 07:52:49.554069 2166 rpcz_store.cc:276] 1025 07:52:41.494505 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.494544 (+ 39us) service_pool.cc:225] Handling call
1025 07:52:49.553908 (+8059364us) ts_tablet_manager.cc:1151] Deleting tablet 50ae9bca6bed4f749d54191ac1543410
1025 07:52:49.553916 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.553974 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8059235}
W20251025 07:52:49.554540 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 50ae9bca6bed4f749d54191ac1543410 because the tablet was not found. No further retry: Not found: Tablet not found: 50ae9bca6bed4f749d54191ac1543410
I20251025 07:52:49.555248 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 188) took 8060 ms. Trace:
I20251025 07:52:49.555334 2166 rpcz_store.cc:276] 1025 07:52:41.494819 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.494864 (+ 45us) service_pool.cc:225] Handling call
1025 07:52:49.555180 (+8060316us) ts_tablet_manager.cc:1151] Deleting tablet 3abfd52327a04ba88625f391a8658fbb
1025 07:52:49.555187 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.555234 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8060187}
W20251025 07:52:49.555701 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 3abfd52327a04ba88625f391a8658fbb because the tablet was not found. No further retry: Not found: Tablet not found: 3abfd52327a04ba88625f391a8658fbb
I20251025 07:52:49.556607 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 189) took 8061 ms. Trace:
I20251025 07:52:49.556712 2166 rpcz_store.cc:276] 1025 07:52:41.495154 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.495204 (+ 50us) service_pool.cc:225] Handling call
1025 07:52:49.556320 (+8061116us) ts_tablet_manager.cc:1151] Deleting tablet 34315f64b89148efbf1f20a43d5c5efb
1025 07:52:49.556327 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.556593 (+ 266us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8060991}
W20251025 07:52:49.557031 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 34315f64b89148efbf1f20a43d5c5efb because the tablet was not found. No further retry: Not found: Tablet not found: 34315f64b89148efbf1f20a43d5c5efb
I20251025 07:52:49.434031 454 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_fa83d0528b4b481989bfc0ca35c6de44 [id=bf77b5ff6f1048dd996d948437f9ff5e]). Expected version 3 got 0
I20251025 07:52:49.557606 2166 tablet_replica.cc:333] T 88eb51447e0f4080baa49dd902deb24f P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:49.557543 454 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3]). Expected version 1 got 0
I20251025 07:52:49.557796 2166 raft_consensus.cc:2243] T 88eb51447e0f4080baa49dd902deb24f P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:49.557971 2166 raft_consensus.cc:2272] T 88eb51447e0f4080baa49dd902deb24f P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:49.558539 2166 ts_tablet_manager.cc:1916] T 88eb51447e0f4080baa49dd902deb24f P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:49.563175 1166 tablet_bootstrap.cc:492] T 4b278ce7768349e49dcd1131e404ab34 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:49.563333 1166 ts_tablet_manager.cc:1403] T 4b278ce7768349e49dcd1131e404ab34 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.073s user 0.007s sys 0.007s
W20251025 07:52:49.563889 850 consensus_peers.cc:597] T ebf3a2a926754305b5bd95fcebb25d29 P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ebf3a2a926754305b5bd95fcebb25d29. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:49.564117 1166 raft_consensus.cc:359] T 4b278ce7768349e49dcd1131e404ab34 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.564293 1166 raft_consensus.cc:385] T 4b278ce7768349e49dcd1131e404ab34 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.564344 1166 raft_consensus.cc:740] T 4b278ce7768349e49dcd1131e404ab34 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.564551 1166 consensus_queue.cc:260] T 4b278ce7768349e49dcd1131e404ab34 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.564585 454 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5]). Expected version 1 got 0
I20251025 07:52:49.565114 1166 ts_tablet_manager.cc:1434] T 4b278ce7768349e49dcd1131e404ab34 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:49.565465 1166 tablet_bootstrap.cc:492] T 3e79d69d5d004cc395b8e35569e25440 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:49.567131 1166 tablet_bootstrap.cc:654] T 3e79d69d5d004cc395b8e35569e25440 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.570811 847 leader_election.cc:304] T 075f0a151e024220b51ba7fcaecdcc3d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.572723 454 catalog_manager.cc:5649] T 143aa3fce9ba4850b000a94054e5dc3a P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } }
I20251025 07:52:49.573423 1167 raft_consensus.cc:359] T aea8fcf0154a4bd4a05b19e27ab78b79 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.573928 3122 raft_consensus.cc:2804] T 075f0a151e024220b51ba7fcaecdcc3d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:49.574164 3122 raft_consensus.cc:697] T 075f0a151e024220b51ba7fcaecdcc3d P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:49.574431 3122 consensus_queue.cc:237] T 075f0a151e024220b51ba7fcaecdcc3d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.580282 2886 raft_consensus.cc:493] T 72618756f756484e895e885ef12545c8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.580394 2886 raft_consensus.cc:515] T 72618756f756484e895e885ef12545c8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.580704 2686 raft_consensus.cc:493] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.580907 2886 leader_election.cc:290] T 72618756f756484e895e885ef12545c8 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:49.580862 2686 raft_consensus.cc:515] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.581525 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72618756f756484e895e885ef12545c8" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:49.582219 850 leader_election.cc:343] T 72618756f756484e895e885ef12545c8 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.582526 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4da89f863cc43f3bf6df908bb9dc6bf" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:49.583133 850 leader_election.cc:343] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.583415 2686 leader_election.cc:290] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:49.585036 847 leader_election.cc:304] T d46673a283fe42b78ddeb99fbd2678eb P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.585299 2686 raft_consensus.cc:2804] T d46673a283fe42b78ddeb99fbd2678eb P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:49.585397 847 leader_election.cc:304] T df0d86ad301245c1a6bcb44805936d25 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.585666 2686 raft_consensus.cc:697] T d46673a283fe42b78ddeb99fbd2678eb P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:49.586036 2686 consensus_queue.cc:237] T d46673a283fe42b78ddeb99fbd2678eb P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.586967 1166 tablet_bootstrap.cc:492] T 3e79d69d5d004cc395b8e35569e25440 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:49.587103 1166 ts_tablet_manager.cc:1403] T 3e79d69d5d004cc395b8e35569e25440 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20251025 07:52:49.587466 1167 raft_consensus.cc:385] T aea8fcf0154a4bd4a05b19e27ab78b79 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.587570 1167 raft_consensus.cc:740] T aea8fcf0154a4bd4a05b19e27ab78b79 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.587762 1167 consensus_queue.cc:260] T aea8fcf0154a4bd4a05b19e27ab78b79 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.587961 1166 raft_consensus.cc:359] T 3e79d69d5d004cc395b8e35569e25440 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.588146 1166 raft_consensus.cc:385] T 3e79d69d5d004cc395b8e35569e25440 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.588204 1166 raft_consensus.cc:740] T 3e79d69d5d004cc395b8e35569e25440 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.588413 1166 consensus_queue.cc:260] T 3e79d69d5d004cc395b8e35569e25440 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.588929 1166 ts_tablet_manager.cc:1434] T 3e79d69d5d004cc395b8e35569e25440 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 07:52:49.589123 1167 ts_tablet_manager.cc:1434] T aea8fcf0154a4bd4a05b19e27ab78b79 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.036s user 0.002s sys 0.000s
I20251025 07:52:49.589188 1166 tablet_bootstrap.cc:492] T 3aeaa0e9dc594e21a9ddd923e314017d P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:49.589953 1167 tablet_bootstrap.cc:492] T dbb254115a574bf4976aff0ea56f3bd9 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:49.590624 454 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 783487b60c144f719ed2eb3c67b5ccab (table default.Table_09095c0829ed4d1cb8b763fc695e7fa8 [id=e372ca4384644779a6b37f389f549dec]). Expected version 1 got 0
I20251025 07:52:49.590813 1166 tablet_bootstrap.cc:654] T 3aeaa0e9dc594e21a9ddd923e314017d P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.591007 454 catalog_manager.cc:5649] T 627a9cfd398346f2aa34c8932a11b5a8 P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } }
I20251025 07:52:49.591728 454 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f]). Expected version 1 got 0
I20251025 07:52:49.592053 454 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7]). Expected version 1 got 0
I20251025 07:52:49.592437 454 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 872cb7512d104827b1f1f186d33a4a44 (table default.Table_d5c65dce8b1b4214ba519d964012bfc0 [id=d4be274b36a845a99a4b96b8846248fd]). Expected version 1 got 0
I20251025 07:52:49.592824 847 leader_election.cc:304] T d0240d20f9904b4bbc8bf7f0f05e8850 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.593107 2886 raft_consensus.cc:2804] T df0d86ad301245c1a6bcb44805936d25 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:49.593176 2886 raft_consensus.cc:493] T df0d86ad301245c1a6bcb44805936d25 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:49.593235 2886 raft_consensus.cc:3060] T df0d86ad301245c1a6bcb44805936d25 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.596009 2886 raft_consensus.cc:515] T df0d86ad301245c1a6bcb44805936d25 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.596719 2169 ts_tablet_manager.cc:1929] T df1248f6dbd94af68c587da7d149e279 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:49.596843 2169 log.cc:1199] T df1248f6dbd94af68c587da7d149e279 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/df1248f6dbd94af68c587da7d149e279
I20251025 07:52:49.597213 2169 ts_tablet_manager.cc:1950] T df1248f6dbd94af68c587da7d149e279 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:49.597168 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df0d86ad301245c1a6bcb44805936d25" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:49.597769 850 leader_election.cc:343] T df0d86ad301245c1a6bcb44805936d25 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.597942 2886 leader_election.cc:290] T df0d86ad301245c1a6bcb44805936d25 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:49.598117 2886 raft_consensus.cc:2804] T d0240d20f9904b4bbc8bf7f0f05e8850 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:49.598313 2886 raft_consensus.cc:697] T d0240d20f9904b4bbc8bf7f0f05e8850 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:49.598431 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 321) took 3680 ms. Trace:
I20251025 07:52:49.598610 2886 consensus_queue.cc:237] T d0240d20f9904b4bbc8bf7f0f05e8850 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.598543 2169 rpcz_store.cc:276] 1025 07:52:45.918472 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.918537 (+ 65us) service_pool.cc:225] Handling call
1025 07:52:49.502309 (+3583772us) ts_tablet_manager.cc:1151] Deleting tablet df1248f6dbd94af68c587da7d149e279
1025 07:52:49.502317 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.506447 (+ 4130us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.596585 (+ 90138us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.598410 (+ 1825us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3583511}
W20251025 07:52:49.600988 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:43563, user_credentials={real_user=slave}} blocked reactor thread for 84080.8us
W20251025 07:52:49.601308 847 leader_election.cc:343] T 3b940fa5ae26427dafbcfd26372754b3 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: 3b940fa5ae26427dafbcfd26372754b3
I20251025 07:52:49.601419 847 leader_election.cc:304] T 3b940fa5ae26427dafbcfd26372754b3 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.601322 710 leader_election.cc:304] T 22fc90c704f64d2f9f4b8c54dee45576 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.603070 3137 raft_consensus.cc:2804] T 22fc90c704f64d2f9f4b8c54dee45576 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:49.603263 3137 raft_consensus.cc:697] T 22fc90c704f64d2f9f4b8c54dee45576 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:49.603628 3137 consensus_queue.cc:237] T 22fc90c704f64d2f9f4b8c54dee45576 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:49.609217 1167 tablet_bootstrap.cc:654] T dbb254115a574bf4976aff0ea56f3bd9 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.609874 2990 raft_consensus.cc:2749] T 3b940fa5ae26427dafbcfd26372754b3 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:49.610562 2990 consensus_queue.cc:1048] T 6a33cfb2f2de48d4a5e9e39730d4900e P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:49.599063 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet df1248f6dbd94af68c587da7d149e279 (table default.Table_73279bdee234454e9def34441da9579a [id=6430d82b82c042469a77ea9f4dad15fc]) successfully deleted
I20251025 07:52:49.615536 454 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 04d27d54fdb7483b869408af65ff85c9 (table default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2]). Expected version 2 got 0
I20251025 07:52:49.616674 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72618756f756484e895e885ef12545c8" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:49.616752 1166 tablet_bootstrap.cc:492] T 3aeaa0e9dc594e21a9ddd923e314017d P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:49.616878 1166 ts_tablet_manager.cc:1403] T 3aeaa0e9dc594e21a9ddd923e314017d P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.008s
I20251025 07:52:49.616966 772 raft_consensus.cc:2468] T 72618756f756484e895e885ef12545c8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
W20251025 07:52:49.617424 710 leader_election.cc:343] T 0582ee42ab564c07ae3facfc5935617c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.617558 1166 raft_consensus.cc:359] T 3aeaa0e9dc594e21a9ddd923e314017d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.617722 1166 raft_consensus.cc:385] T 3aeaa0e9dc594e21a9ddd923e314017d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.617770 1166 raft_consensus.cc:740] T 3aeaa0e9dc594e21a9ddd923e314017d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.617700 710 leader_election.cc:304] T 0582ee42ab564c07ae3facfc5935617c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.617925 1166 consensus_queue.cc:260] T 3aeaa0e9dc594e21a9ddd923e314017d P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.618221 3142 raft_consensus.cc:2749] T 0582ee42ab564c07ae3facfc5935617c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:49.618417 1166 ts_tablet_manager.cc:1434] T 3aeaa0e9dc594e21a9ddd923e314017d P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251025 07:52:49.618683 1166 tablet_bootstrap.cc:492] T aea8fcf0154a4bd4a05b19e27ab78b79 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:49.618896 1045 raft_consensus.cc:3060] T dff7f86ae8924685b26011e504e2247b P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.619283 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4da89f863cc43f3bf6df908bb9dc6bf" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:49.619421 772 raft_consensus.cc:2468] T e4da89f863cc43f3bf6df908bb9dc6bf P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:49.620081 1166 tablet_bootstrap.cc:654] T aea8fcf0154a4bd4a05b19e27ab78b79 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.622888 2686 consensus_queue.cc:1048] T dff7f86ae8924685b26011e504e2247b P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:49.623704 454 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 22f7db660616457b83b24e947a0b0bae (table default.Table_4a86191163f2408f9bcc795cd8d1bbf6 [id=bb316a7e000748ca902a9bfb494802d7]). Expected version 1 got 0
I20251025 07:52:49.624662 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df0d86ad301245c1a6bcb44805936d25" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:49.624872 772 raft_consensus.cc:3060] T df0d86ad301245c1a6bcb44805936d25 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.627600 772 raft_consensus.cc:2468] T df0d86ad301245c1a6bcb44805936d25 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:49.644006 2169 tablet_replica.cc:333] T ab42909afc524f2fa3c1041495d13a59 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:49.644291 2169 raft_consensus.cc:2243] T ab42909afc524f2fa3c1041495d13a59 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:49.644497 2169 raft_consensus.cc:2272] T ab42909afc524f2fa3c1041495d13a59 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:49.646435 1169 tablet_bootstrap.cc:492] T d77704b4fe6c4456a2ba40ffec0c60fc P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:49.646576 1169 ts_tablet_manager.cc:1403] T d77704b4fe6c4456a2ba40ffec0c60fc P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.119s user 0.026s sys 0.045s
I20251025 07:52:49.647256 1169 raft_consensus.cc:359] T d77704b4fe6c4456a2ba40ffec0c60fc P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:49.647418 1169 raft_consensus.cc:385] T d77704b4fe6c4456a2ba40ffec0c60fc P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.647460 1169 raft_consensus.cc:740] T d77704b4fe6c4456a2ba40ffec0c60fc P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.647665 1169 consensus_queue.cc:260] T d77704b4fe6c4456a2ba40ffec0c60fc P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:49.648938 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72dc084f8efc45478771c247e07a16f4" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:49.649116 909 raft_consensus.cc:3060] T 72dc084f8efc45478771c247e07a16f4 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.651018 847 leader_election.cc:304] T 72618756f756484e895e885ef12545c8 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.651293 2169 ts_tablet_manager.cc:1916] T ab42909afc524f2fa3c1041495d13a59 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:49.651837 3122 raft_consensus.cc:493] T 67cf96a7dd3f42e19b86746c87e689f6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.651863 909 raft_consensus.cc:2468] T 72dc084f8efc45478771c247e07a16f4 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:49.651949 3122 raft_consensus.cc:515] T 67cf96a7dd3f42e19b86746c87e689f6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.652892 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67cf96a7dd3f42e19b86746c87e689f6" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:49.653384 850 leader_election.cc:343] T 67cf96a7dd3f42e19b86746c87e689f6 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.653517 3122 leader_election.cc:290] T 67cf96a7dd3f42e19b86746c87e689f6 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:49.653630 3122 raft_consensus.cc:2804] T 72618756f756484e895e885ef12545c8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:49.653681 3122 raft_consensus.cc:493] T 72618756f756484e895e885ef12545c8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:49.653730 3122 raft_consensus.cc:3060] T 72618756f756484e895e885ef12545c8 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.654718 2146 rpcz_store.cc:276] 1025 07:52:44.510018 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.510077 (+ 59us) service_pool.cc:225] Handling call
1025 07:52:49.546270 (+5036193us) ts_tablet_manager.cc:1151] Deleting tablet 60c60a9ee37f43978910bd067a098bb7
1025 07:52:49.546282 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.546382 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5036001}
I20251025 07:52:49.655489 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 272) took 5069 ms. Trace:
I20251025 07:52:49.655566 2146 rpcz_store.cc:276] 1025 07:52:44.585931 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.586362 (+ 431us) service_pool.cc:225] Handling call
1025 07:52:49.655397 (+5069035us) ts_tablet_manager.cc:1151] Deleting tablet 74e2d78c024849b0af5c0bde1f48c36f
1025 07:52:49.655408 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.655473 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5068860}
I20251025 07:52:49.655995 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 274) took 5069 ms. Trace:
I20251025 07:52:49.656055 2146 rpcz_store.cc:276] 1025 07:52:44.586793 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.587030 (+ 237us) service_pool.cc:225] Handling call
1025 07:52:49.655924 (+5068894us) ts_tablet_manager.cc:1151] Deleting tablet c4610b0c62e741eca28c21f63cc3de81
1025 07:52:49.655932 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.655983 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5068757}
I20251025 07:52:49.656424 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 271) took 5070 ms. Trace:
I20251025 07:52:49.656606 3122 raft_consensus.cc:515] T 72618756f756484e895e885ef12545c8 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.657101 3122 leader_election.cc:290] T 72618756f756484e895e885ef12545c8 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:49.657182 3143 raft_consensus.cc:493] T 1068f566477e4c6db7dde566fff4c2fb P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.656508 2146 rpcz_store.cc:276] 1025 07:52:44.585851 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.585964 (+ 113us) service_pool.cc:225] Handling call
1025 07:52:49.656339 (+5070375us) ts_tablet_manager.cc:1151] Deleting tablet 943ec703e65f4fd3b0c690e08c0bd226
1025 07:52:49.656347 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.656409 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5064291}
I20251025 07:52:49.657392 3143 raft_consensus.cc:515] T 1068f566477e4c6db7dde566fff4c2fb P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.657507 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72618756f756484e895e885ef12545c8" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:49.627911 2166 ts_tablet_manager.cc:1929] T 88eb51447e0f4080baa49dd902deb24f P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:49.657840 2166 log.cc:1199] T 88eb51447e0f4080baa49dd902deb24f P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/88eb51447e0f4080baa49dd902deb24f
I20251025 07:52:49.657974 3143 leader_election.cc:290] T 1068f566477e4c6db7dde566fff4c2fb P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:49.658269 2166 ts_tablet_manager.cc:1950] T 88eb51447e0f4080baa49dd902deb24f P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:49.659011 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 190) took 8163 ms. Trace:
I20251025 07:52:49.659094 2166 rpcz_store.cc:276] 1025 07:52:41.495508 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.495550 (+ 42us) service_pool.cc:225] Handling call
1025 07:52:49.557558 (+8062008us) ts_tablet_manager.cc:1151] Deleting tablet 88eb51447e0f4080baa49dd902deb24f
1025 07:52:49.557568 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.624980 (+ 67412us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.627821 (+ 2841us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.658994 (+ 31173us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8061881}
I20251025 07:52:49.659432 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 191) took 8163 ms. Trace:
I20251025 07:52:49.659510 2166 rpcz_store.cc:276] 1025 07:52:41.495918 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.495959 (+ 41us) service_pool.cc:225] Handling call
1025 07:52:49.659342 (+8163383us) ts_tablet_manager.cc:1151] Deleting tablet 08da9d2687704725aacd11cd92c6a9f9
1025 07:52:49.659351 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.659418 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8163246}
I20251025 07:52:49.659818 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 192) took 8163 ms. Trace:
I20251025 07:52:49.659931 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1068f566477e4c6db7dde566fff4c2fb" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:49.659891 2166 rpcz_store.cc:276] 1025 07:52:41.496286 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.496327 (+ 41us) service_pool.cc:225] Handling call
1025 07:52:49.659729 (+8163402us) ts_tablet_manager.cc:1151] Deleting tablet d66ad049d5484228a268b0facb00ecf1
1025 07:52:49.659740 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.659802 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8163266}
I20251025 07:52:49.660853 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 193) took 8164 ms. Trace:
I20251025 07:52:49.660929 2166 rpcz_store.cc:276] 1025 07:52:41.496687 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.496727 (+ 40us) service_pool.cc:225] Handling call
1025 07:52:49.660765 (+8164038us) ts_tablet_manager.cc:1151] Deleting tablet 4b28215f61574cf48c447dd01955ac77
1025 07:52:49.660777 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.660837 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8163888}
I20251025 07:52:49.661177 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 194) took 8161 ms. Trace:
I20251025 07:52:49.661245 2166 rpcz_store.cc:276] 1025 07:52:41.499334 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.499752 (+ 418us) service_pool.cc:225] Handling call
1025 07:52:49.661111 (+8161359us) ts_tablet_manager.cc:1151] Deleting tablet 7fbae8be73bf4e18b5a5c4ec4cdebc12
1025 07:52:49.661119 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.661164 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8161218}
I20251025 07:52:49.661499 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 195) took 8157 ms. Trace:
I20251025 07:52:49.661564 2166 rpcz_store.cc:276] 1025 07:52:41.504336 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.504384 (+ 48us) service_pool.cc:225] Handling call
1025 07:52:49.661436 (+8157052us) ts_tablet_manager.cc:1151] Deleting tablet 551d07a9242740dcb9366bb2d707a4a8
1025 07:52:49.661443 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.661487 (+ 44us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8156887}
I20251025 07:52:49.661809 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 196) took 8157 ms. Trace:
I20251025 07:52:49.661877 2166 rpcz_store.cc:276] 1025 07:52:41.504727 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.504767 (+ 40us) service_pool.cc:225] Handling call
1025 07:52:49.661746 (+8156979us) ts_tablet_manager.cc:1151] Deleting tablet 6ec421eb820b4f40afcff5a5bf80c508
1025 07:52:49.661754 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.661796 (+ 42us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8156848}
I20251025 07:52:49.662174 2166 tablet_replica.cc:333] T af743ee344e543a0aa6da3cd7a30e467 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:49.662426 2166 raft_consensus.cc:2243] T af743ee344e543a0aa6da3cd7a30e467 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:49.662586 2166 raft_consensus.cc:2272] T af743ee344e543a0aa6da3cd7a30e467 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:49.663182 2166 ts_tablet_manager.cc:1916] T af743ee344e543a0aa6da3cd7a30e467 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:49.655216 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 60c60a9ee37f43978910bd067a098bb7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60c60a9ee37f43978910bd067a098bb7 already in progress: creating tablet
W20251025 07:52:49.663756 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 74e2d78c024849b0af5c0bde1f48c36f because the tablet was not found. No further retry: Not found: Tablet not found: 74e2d78c024849b0af5c0bde1f48c36f
I20251025 07:52:49.663952 454 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet f70091024c034c118fb86578db19a31c (table default.Table_b7da40b96d6c4359bed6f898e8cf6275 [id=e041946c093b4c789ce175c736cc230b]). Expected version 2 got 0
I20251025 07:52:49.664173 454 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 5dab439bc50d4e869548154e1d222ad6 (table default.Table_fb9def2302784c999bf0f5707c60eb4c [id=45325c188eac4d34bb016d01de091f87]). Expected version 3 got 0
I20251025 07:52:49.660256 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 88eb51447e0f4080baa49dd902deb24f (table default.Table_b210463b66db4877b9a791f7cd741492 [id=438936db6ab24eb09a48b31ca013031b]) successfully deleted
I20251025 07:52:49.664533 454 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]). Expected version 3 got 0
W20251025 07:52:49.664567 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 08da9d2687704725aacd11cd92c6a9f9 because the tablet was not found. No further retry: Not found: Tablet not found: 08da9d2687704725aacd11cd92c6a9f9
W20251025 07:52:49.664790 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet d66ad049d5484228a268b0facb00ecf1 because the tablet was not found. No further retry: Not found: Tablet not found: d66ad049d5484228a268b0facb00ecf1
W20251025 07:52:49.664980 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 4b28215f61574cf48c447dd01955ac77 because the tablet was not found. No further retry: Not found: Tablet not found: 4b28215f61574cf48c447dd01955ac77
I20251025 07:52:49.665066 454 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 5fc092c2bb2d41ac8c5b4fc07460c8ec (table default.Table_cd20e202397645f8821dcde13b5c0ce3 [id=630c274339664b9db18ab4ef7c29c16f]). Expected version 2 got 0
W20251025 07:52:49.665277 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 551d07a9242740dcb9366bb2d707a4a8 because the tablet was not found. No further retry: Not found: Tablet not found: 551d07a9242740dcb9366bb2d707a4a8
W20251025 07:52:49.665467 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 6ec421eb820b4f40afcff5a5bf80c508 because the tablet was not found. No further retry: Not found: Tablet not found: 6ec421eb820b4f40afcff5a5bf80c508
W20251025 07:52:49.663992 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet c4610b0c62e741eca28c21f63cc3de81 because the tablet was not found. No further retry: Not found: Tablet not found: c4610b0c62e741eca28c21f63cc3de81
W20251025 07:52:49.665885 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 943ec703e65f4fd3b0c690e08c0bd226 because the tablet was not found. No further retry: Not found: Tablet not found: 943ec703e65f4fd3b0c690e08c0bd226
I20251025 07:52:49.666028 454 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 0325d462441349ef81841f1b0cab2a20 (table default.Table_750ecb905839434fa9a1aa72f87394be [id=9ce3ba77c56a4ef09cb3b40ea0e488f6]). Expected version 1 got 0
I20251025 07:52:49.667325 1169 ts_tablet_manager.cc:1434] T d77704b4fe6c4456a2ba40ffec0c60fc P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20251025 07:52:49.657750 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 273) took 5071 ms. Trace:
W20251025 07:52:49.657863 850 leader_election.cc:343] T 72618756f756484e895e885ef12545c8 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.670192 2146 rpcz_store.cc:276] 1025 07:52:44.586665 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.586719 (+ 54us) service_pool.cc:225] Handling call
1025 07:52:49.657664 (+5070945us) ts_tablet_manager.cc:1151] Deleting tablet 5fca357d7e24494fb6af4bd73d98833c
1025 07:52:49.657676 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.657735 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5060681}
W20251025 07:52:49.670850 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 5fca357d7e24494fb6af4bd73d98833c because the tablet was not found. No further retry: Not found: Tablet not found: 5fca357d7e24494fb6af4bd73d98833c
I20251025 07:52:49.671684 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 278) took 5074 ms. Trace:
I20251025 07:52:49.659575 847 leader_election.cc:304] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.666661 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1068f566477e4c6db7dde566fff4c2fb" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:49.673101 2990 raft_consensus.cc:2804] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:49.673190 2990 raft_consensus.cc:493] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:49.673267 2990 raft_consensus.cc:3060] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.673430 983 leader_election.cc:304] T 1068f566477e4c6db7dde566fff4c2fb P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:49.673879 2146 rpcz_store.cc:276] 1025 07:52:44.597449 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.597496 (+ 47us) service_pool.cc:225] Handling call
1025 07:52:49.671198 (+5073702us) ts_tablet_manager.cc:1151] Deleting tablet 7c7a9d7a4fb3449bb57eafa83230a839
1025 07:52:49.671214 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.671664 (+ 450us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5073540}
I20251025 07:52:49.674011 3143 raft_consensus.cc:2749] T 1068f566477e4c6db7dde566fff4c2fb P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:49.674393 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 7c7a9d7a4fb3449bb57eafa83230a839 because the tablet was not found. No further retry: Not found: Tablet not found: 7c7a9d7a4fb3449bb57eafa83230a839
I20251025 07:52:49.674470 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 277) took 5077 ms. Trace:
I20251025 07:52:49.674542 2146 rpcz_store.cc:276] 1025 07:52:44.597334 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.597604 (+ 270us) service_pool.cc:225] Handling call
1025 07:52:49.674377 (+5076773us) ts_tablet_manager.cc:1151] Deleting tablet 02e4c4d6ffb34cecaf17beb5ba197e81
1025 07:52:49.674389 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.674455 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5076624}
W20251025 07:52:49.674908 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 02e4c4d6ffb34cecaf17beb5ba197e81 because the tablet was not found. No further retry: Not found: Tablet not found: 02e4c4d6ffb34cecaf17beb5ba197e81
I20251025 07:52:49.674959 1169 tablet_bootstrap.cc:492] T fce535a0c0bd45c6bf17f81add297f40 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:49.675261 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 280) took 5076 ms. Trace:
I20251025 07:52:49.675407 2146 rpcz_store.cc:276] 1025 07:52:44.598906 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.599011 (+ 105us) service_pool.cc:225] Handling call
1025 07:52:49.675184 (+5076173us) ts_tablet_manager.cc:1151] Deleting tablet 70fe9aa2f58d4abea3b3b71066a29653
1025 07:52:49.675193 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.675244 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5076007}
W20251025 07:52:49.675853 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 70fe9aa2f58d4abea3b3b71066a29653 because the tablet was not found. No further retry: Not found: Tablet not found: 70fe9aa2f58d4abea3b3b71066a29653
I20251025 07:52:49.675875 2990 raft_consensus.cc:515] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.676389 3011 raft_consensus.cc:493] T 1b5ac7e845a24b2eb5b2429d47361232 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.676506 3011 raft_consensus.cc:515] T 1b5ac7e845a24b2eb5b2429d47361232 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:49.676628 1169 tablet_bootstrap.cc:654] T fce535a0c0bd45c6bf17f81add297f40 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.677006 3011 leader_election.cc:290] T 1b5ac7e845a24b2eb5b2429d47361232 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:49.676978 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4da89f863cc43f3bf6df908bb9dc6bf" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:49.677047 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 282) took 5077 ms. Trace:
I20251025 07:52:49.677106 2146 rpcz_store.cc:276] 1025 07:52:44.599081 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.599128 (+ 47us) service_pool.cc:225] Handling call
1025 07:52:49.676966 (+5077838us) ts_tablet_manager.cc:1151] Deleting tablet c59f6e6eb9594a8e83f109a09b034045
1025 07:52:49.676975 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.677031 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5077669}
W20251025 07:52:49.677467 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet c59f6e6eb9594a8e83f109a09b034045 because the tablet was not found. No further retry: Not found: Tablet not found: c59f6e6eb9594a8e83f109a09b034045
W20251025 07:52:49.677561 850 leader_election.cc:343] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.677748 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 281) took 5078 ms. Trace:
I20251025 07:52:49.677819 2146 rpcz_store.cc:276] 1025 07:52:44.599005 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.599146 (+ 141us) service_pool.cc:225] Handling call
1025 07:52:49.677672 (+5078526us) ts_tablet_manager.cc:1151] Deleting tablet f4578ce69a0d4322944e5ce7c8a59831
1025 07:52:49.677680 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.677734 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5078195}
W20251025 07:52:49.678197 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet f4578ce69a0d4322944e5ce7c8a59831 because the tablet was not found. No further retry: Not found: Tablet not found: f4578ce69a0d4322944e5ce7c8a59831
I20251025 07:52:49.678285 2990 leader_election.cc:290] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:49.678439 771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b5ac7e845a24b2eb5b2429d47361232" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:49.679209 1166 tablet_bootstrap.cc:492] T aea8fcf0154a4bd4a05b19e27ab78b79 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:49.679319 1166 ts_tablet_manager.cc:1403] T aea8fcf0154a4bd4a05b19e27ab78b79 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.061s user 0.010s sys 0.003s
W20251025 07:52:49.679432 983 leader_election.cc:343] T 1b5ac7e845a24b2eb5b2429d47361232 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: 1b5ac7e845a24b2eb5b2429d47361232
I20251025 07:52:49.679529 983 leader_election.cc:304] T 1b5ac7e845a24b2eb5b2429d47361232 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:49.679857 3143 raft_consensus.cc:2749] T 1b5ac7e845a24b2eb5b2429d47361232 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:49.679999 1166 raft_consensus.cc:359] T aea8fcf0154a4bd4a05b19e27ab78b79 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.680106 2146 tablet_replica.cc:333] T 2b594d4718344aafbdaec34040695589 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:49.680140 1166 raft_consensus.cc:385] T aea8fcf0154a4bd4a05b19e27ab78b79 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.680192 1166 raft_consensus.cc:740] T aea8fcf0154a4bd4a05b19e27ab78b79 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.680220 847 leader_election.cc:304] T df0d86ad301245c1a6bcb44805936d25 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.680280 2146 raft_consensus.cc:2243] T 2b594d4718344aafbdaec34040695589 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:49.680405 1166 consensus_queue.cc:260] T aea8fcf0154a4bd4a05b19e27ab78b79 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.681806 2169 ts_tablet_manager.cc:1929] T ab42909afc524f2fa3c1041495d13a59 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:49.681919 2169 log.cc:1199] T ab42909afc524f2fa3c1041495d13a59 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/ab42909afc524f2fa3c1041495d13a59
I20251025 07:52:49.681960 2686 raft_consensus.cc:2804] T df0d86ad301245c1a6bcb44805936d25 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:49.682277 2990 raft_consensus.cc:493] T af743ee344e543a0aa6da3cd7a30e467 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.682314 2169 ts_tablet_manager.cc:1950] T ab42909afc524f2fa3c1041495d13a59 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:49.682403 2990 raft_consensus.cc:515] T af743ee344e543a0aa6da3cd7a30e467 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.682500 1166 ts_tablet_manager.cc:1434] T aea8fcf0154a4bd4a05b19e27ab78b79 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251025 07:52:49.682749 1166 tablet_bootstrap.cc:492] T ab68e5a89b234064b27aa7cae9467e14 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:49.683014 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 344) took 1811 ms. Trace:
I20251025 07:52:49.683081 2169 rpcz_store.cc:276] 1025 07:52:47.871656 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:47.871877 (+ 221us) service_pool.cc:225] Handling call
1025 07:52:49.643929 (+1772052us) ts_tablet_manager.cc:1151] Deleting tablet ab42909afc524f2fa3c1041495d13a59
1025 07:52:49.643945 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.654478 (+ 10533us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.680486 (+ 26008us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.682997 (+ 2511us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1749448}
I20251025 07:52:49.683329 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af743ee344e543a0aa6da3cd7a30e467" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:49.683481 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet ab42909afc524f2fa3c1041495d13a59 (table default.Table_7eb8424d5192485da1a2d67bfdb2bbbf [id=40c29df99590408cbd5c583d4fc741eb]) successfully deleted
W20251025 07:52:49.683764 850 leader_election.cc:343] T af743ee344e543a0aa6da3cd7a30e467 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.683811 2686 raft_consensus.cc:697] T df0d86ad301245c1a6bcb44805936d25 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:49.683769 2990 leader_election.cc:290] T af743ee344e543a0aa6da3cd7a30e467 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:49.683961 2146 raft_consensus.cc:2272] T 2b594d4718344aafbdaec34040695589 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:49.684201 1166 tablet_bootstrap.cc:654] T ab68e5a89b234064b27aa7cae9467e14 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.684185 2686 consensus_queue.cc:237] T df0d86ad301245c1a6bcb44805936d25 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.684724 2146 ts_tablet_manager.cc:1916] T 2b594d4718344aafbdaec34040695589 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:49.685935 2166 ts_tablet_manager.cc:1929] T af743ee344e543a0aa6da3cd7a30e467 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:49.686045 2166 log.cc:1199] T af743ee344e543a0aa6da3cd7a30e467 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/af743ee344e543a0aa6da3cd7a30e467
I20251025 07:52:49.686336 2166 ts_tablet_manager.cc:1950] T af743ee344e543a0aa6da3cd7a30e467 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:49.686352 710 leader_election.cc:304] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.686820 3137 raft_consensus.cc:2804] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:49.686949 3137 raft_consensus.cc:697] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:49.687011 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 197) took 8181 ms. Trace:
I20251025 07:52:49.687083 2166 rpcz_store.cc:276] 1025 07:52:41.505087 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.505141 (+ 54us) service_pool.cc:225] Handling call
1025 07:52:49.662104 (+8156963us) ts_tablet_manager.cc:1151] Deleting tablet af743ee344e543a0aa6da3cd7a30e467
1025 07:52:49.662115 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.669746 (+ 7631us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.685835 (+ 16089us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.686992 (+ 1157us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8156816}
I20251025 07:52:49.687215 3137 consensus_queue.cc:237] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.687520 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet af743ee344e543a0aa6da3cd7a30e467 (table default.Table_843881ca867542619b4fff840d2879ca [id=d324f310a01a4ae5abbff6ddafffca7d]) successfully deleted
I20251025 07:52:49.691975 1167 tablet_bootstrap.cc:492] T dbb254115a574bf4976aff0ea56f3bd9 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:49.692103 1167 ts_tablet_manager.cc:1403] T dbb254115a574bf4976aff0ea56f3bd9 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.102s user 0.003s sys 0.002s
I20251025 07:52:49.692150 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 198) took 8185 ms. Trace:
I20251025 07:52:49.692214 2166 rpcz_store.cc:276] 1025 07:52:41.506327 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.506373 (+ 46us) service_pool.cc:225] Handling call
1025 07:52:49.692057 (+8185684us) ts_tablet_manager.cc:1151] Deleting tablet 467324e2432f4caca148951329caa292
1025 07:52:49.692065 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.692136 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8185533}
W20251025 07:52:49.692638 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 467324e2432f4caca148951329caa292 because the tablet was not found. No further retry: Not found: Tablet not found: 467324e2432f4caca148951329caa292
I20251025 07:52:49.692786 1167 raft_consensus.cc:359] T dbb254115a574bf4976aff0ea56f3bd9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.693004 1167 raft_consensus.cc:385] T dbb254115a574bf4976aff0ea56f3bd9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.693071 1167 raft_consensus.cc:740] T dbb254115a574bf4976aff0ea56f3bd9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.693205 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 199) took 8157 ms. Trace:
I20251025 07:52:49.694816 765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4da89f863cc43f3bf6df908bb9dc6bf" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:49.695055 765 raft_consensus.cc:3060] T e4da89f863cc43f3bf6df908bb9dc6bf P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.694994 2166 rpcz_store.cc:276] 1025 07:52:41.536144 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.536211 (+ 67us) service_pool.cc:225] Handling call
1025 07:52:49.692939 (+8156728us) ts_tablet_manager.cc:1151] Deleting tablet 1d261e432cf5403ea162129706bc4d4d
1025 07:52:49.692948 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.693188 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8138571}
I20251025 07:52:49.695425 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72618756f756484e895e885ef12545c8" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:49.695567 772 raft_consensus.cc:3060] T 72618756f756484e895e885ef12545c8 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:49.695571 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 1d261e432cf5403ea162129706bc4d4d because the tablet was not found. No further retry: Not found: Tablet not found: 1d261e432cf5403ea162129706bc4d4d
I20251025 07:52:49.696007 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 200) took 7790 ms. Trace:
I20251025 07:52:49.696071 2166 rpcz_store.cc:276] 1025 07:52:41.905238 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.906188 (+ 950us) service_pool.cc:225] Handling call
1025 07:52:49.695925 (+7789737us) ts_tablet_manager.cc:1151] Deleting tablet 2784bbfb97d5466f9ec856add7d0b29a
1025 07:52:49.695933 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.695993 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7789538}
W20251025 07:52:49.696429 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 2784bbfb97d5466f9ec856add7d0b29a because the tablet was not found. No further retry: Not found: Tablet not found: 2784bbfb97d5466f9ec856add7d0b29a
I20251025 07:52:49.696646 761 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67cf96a7dd3f42e19b86746c87e689f6" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:49.696832 761 raft_consensus.cc:2468] T 67cf96a7dd3f42e19b86746c87e689f6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:49.697032 2166 tablet_replica.cc:333] T fd265cdbf6a8440393a2b5d8240d1872 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:49.697194 2166 raft_consensus.cc:2243] T fd265cdbf6a8440393a2b5d8240d1872 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:49.697307 2166 raft_consensus.cc:2272] T fd265cdbf6a8440393a2b5d8240d1872 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:49.697768 2166 ts_tablet_manager.cc:1916] T fd265cdbf6a8440393a2b5d8240d1872 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:49.698856 772 raft_consensus.cc:2468] T 72618756f756484e895e885ef12545c8 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:49.699215 454 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]). Expected version 1 got 0
I20251025 07:52:49.702376 454 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 3a0f50c8000144a4acb5219484b01800 (table default.Table_c62e0dc1ae504f9bb912013506ff3031 [id=6f395892b09240fea02482b3199cfb5e]). Expected version 1 got 0
I20251025 07:52:49.706754 1166 tablet_bootstrap.cc:492] T ab68e5a89b234064b27aa7cae9467e14 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:49.706903 1166 ts_tablet_manager.cc:1403] T ab68e5a89b234064b27aa7cae9467e14 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.006s
I20251025 07:52:49.707593 1166 raft_consensus.cc:359] T ab68e5a89b234064b27aa7cae9467e14 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.707777 1166 raft_consensus.cc:385] T ab68e5a89b234064b27aa7cae9467e14 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.707839 1166 raft_consensus.cc:740] T ab68e5a89b234064b27aa7cae9467e14 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.693840 1167 consensus_queue.cc:260] T dbb254115a574bf4976aff0ea56f3bd9 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.702430 765 raft_consensus.cc:2468] T e4da89f863cc43f3bf6df908bb9dc6bf P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:49.708024 1166 consensus_queue.cc:260] T ab68e5a89b234064b27aa7cae9467e14 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.708514 1166 ts_tablet_manager.cc:1434] T ab68e5a89b234064b27aa7cae9467e14 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 07:52:49.708665 1167 ts_tablet_manager.cc:1434] T dbb254115a574bf4976aff0ea56f3bd9 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20251025 07:52:49.708793 1166 tablet_bootstrap.cc:492] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:49.709074 3142 raft_consensus.cc:493] T 8917a289bd9b4428a4603429edd94581 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.709167 454 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525]). Expected version 1 got 0
I20251025 07:52:49.709192 3142 raft_consensus.cc:515] T 8917a289bd9b4428a4603429edd94581 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.709349 454 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_67b99749f3bf43228b40d8b9807e69b4 [id=fc8180c09d7b4e74b2e161f7acb8fe0e]). Expected version 2 got 0
I20251025 07:52:49.709579 454 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c]). Expected version 1 got 0
I20251025 07:52:49.709786 3142 leader_election.cc:290] T 8917a289bd9b4428a4603429edd94581 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:49.709860 454 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 755f0d6c87044e20a8020f8410940123 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 4 got 0
I20251025 07:52:49.710191 454 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 830fdbc6812e416690d0db42a73af503 (table default.Table_92824b3f2a774564b3aea04aaa25d204 [id=fb21a45b86ab446e8c3dae4512a9598f]). Expected version 1 got 0
I20251025 07:52:49.710242 1166 tablet_bootstrap.cc:654] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.710394 2166 ts_tablet_manager.cc:1929] T fd265cdbf6a8440393a2b5d8240d1872 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:49.710489 2166 log.cc:1199] T fd265cdbf6a8440393a2b5d8240d1872 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/fd265cdbf6a8440393a2b5d8240d1872
I20251025 07:52:49.710453 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8917a289bd9b4428a4603429edd94581" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:49.710779 2166 ts_tablet_manager.cc:1950] T fd265cdbf6a8440393a2b5d8240d1872 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
W20251025 07:52:49.711261 712 leader_election.cc:343] T 8917a289bd9b4428a4603429edd94581 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.711673 1167 tablet_bootstrap.cc:492] T 66134661be834f7b811d7927ab629812 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:49.714031 1167 tablet_bootstrap.cc:654] T 66134661be834f7b811d7927ab629812 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.716984 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 201) took 7811 ms. Trace:
I20251025 07:52:49.717082 2166 rpcz_store.cc:276] 1025 07:52:41.905352 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.906426 (+ 1074us) service_pool.cc:225] Handling call
1025 07:52:49.696988 (+7790562us) ts_tablet_manager.cc:1151] Deleting tablet fd265cdbf6a8440393a2b5d8240d1872
1025 07:52:49.696997 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.700651 (+ 3654us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.710275 (+ 9624us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.716962 (+ 6687us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7790427}
I20251025 07:52:49.717188 454 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236]). Expected version 1 got 0
I20251025 07:52:49.717372 454 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 8707b93d491e4e4386d148087a2f0870 (table default.Table_799dcb1cef2f4fe6b9b8f17f9f1026c5 [id=59840a567e5244e8a9024cbad42471e4]). Expected version 1 got 0
I20251025 07:52:49.717424 2166 tablet_replica.cc:333] T d77704b4fe6c4456a2ba40ffec0c60fc P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:49.717595 2166 raft_consensus.cc:2243] T d77704b4fe6c4456a2ba40ffec0c60fc P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:49.717634 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet fd265cdbf6a8440393a2b5d8240d1872 (table default.Table_434ef1ade0a14af987864563993afae8 [id=3a3af35e11944a6f97fc0144cb9d08d4]) successfully deleted
I20251025 07:52:49.717730 2166 raft_consensus.cc:2272] T d77704b4fe6c4456a2ba40ffec0c60fc P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:49.718236 454 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet b56209e4934e4407a8b4518ca4af0d46 (table default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de]). Expected version 1 got 0
I20251025 07:52:49.718281 2166 ts_tablet_manager.cc:1916] T d77704b4fe6c4456a2ba40ffec0c60fc P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:49.718677 454 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_d11bba62f65740319701b4337a5c255a [id=ad812862698d421184ebf501ce79b7be]). Expected version 3 got 0
I20251025 07:52:49.718878 454 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84]). Expected version 1 got 0
I20251025 07:52:49.719286 1166 tablet_bootstrap.cc:492] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:49.719424 1166 ts_tablet_manager.cc:1403] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s
I20251025 07:52:49.719411 454 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3]). Expected version 2 got 0
I20251025 07:52:49.719970 454 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 288e543bb279410c91c01dd1623e110f (table default.Table_6e242ba55b194c23a6701f4b064fdbe5 [id=4c34b64b299f4162a53866cf01c5298f]). Expected version 1 got 0
I20251025 07:52:49.720065 1166 raft_consensus.cc:359] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.720239 1166 raft_consensus.cc:385] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.720296 1166 raft_consensus.cc:740] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.720482 1166 consensus_queue.cc:260] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.720651 454 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e]). Expected version 1 got 0
I20251025 07:52:49.720948 454 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet cd23b8094d7e472eba6bd170248858a9 (table default.Table_cea16af97a2242968d72900d3a84a86d [id=d82b3e1317434cf08a0d51e8a9427224]). Expected version 4 got 3
I20251025 07:52:49.721117 454 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]). Expected version 1 got 0
I20251025 07:52:49.721342 454 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]). Expected version 1 got 0
I20251025 07:52:49.722252 453 catalog_manager.cc:6194] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: default.Table_0c31f60926cd4ae7b0e8ebb7b28b9f95 [id=bf3a3f8dae184d999f2159dfcc7803b2] alter complete (version 2)
I20251025 07:52:49.722888 1166 ts_tablet_manager.cc:1434] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251025 07:52:49.723203 1166 tablet_bootstrap.cc:492] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:49.724444 2166 ts_tablet_manager.cc:1929] T d77704b4fe6c4456a2ba40ffec0c60fc P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:49.724556 2166 log.cc:1199] T d77704b4fe6c4456a2ba40ffec0c60fc P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/d77704b4fe6c4456a2ba40ffec0c60fc
I20251025 07:52:49.724813 2166 ts_tablet_manager.cc:1950] T d77704b4fe6c4456a2ba40ffec0c60fc P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:49.725136 3143 raft_consensus.cc:493] T c11f674385184ae6b235f1db8bf1fb91 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.725308 3143 raft_consensus.cc:515] T c11f674385184ae6b235f1db8bf1fb91 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:49.725467 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 202) took 7820 ms. Trace:
I20251025 07:52:49.725540 2166 rpcz_store.cc:276] 1025 07:52:41.905438 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.906588 (+ 1150us) service_pool.cc:225] Handling call
1025 07:52:49.717377 (+7810789us) ts_tablet_manager.cc:1151] Deleting tablet d77704b4fe6c4456a2ba40ffec0c60fc
1025 07:52:49.717388 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.721516 (+ 4128us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.724329 (+ 2813us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.725444 (+ 1115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7810664}
I20251025 07:52:49.726084 3143 leader_election.cc:290] T c11f674385184ae6b235f1db8bf1fb91 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:49.726869 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet d77704b4fe6c4456a2ba40ffec0c60fc (table default.Table_e42a1f7287a34753b7e4daf36893d9c2 [id=ff115adf26ec4e499d44caa9b0366a49]) successfully deleted
I20251025 07:52:49.727598 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c11f674385184ae6b235f1db8bf1fb91" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:49.727881 772 raft_consensus.cc:2393] T c11f674385184ae6b235f1db8bf1fb91 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
W20251025 07:52:49.727906 983 leader_election.cc:343] T c11f674385184ae6b235f1db8bf1fb91 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: c11f674385184ae6b235f1db8bf1fb91
I20251025 07:52:49.728439 982 leader_election.cc:304] T c11f674385184ae6b235f1db8bf1fb91 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:49.728806 3143 raft_consensus.cc:3060] T c11f674385184ae6b235f1db8bf1fb91 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.731549 3143 raft_consensus.cc:2749] T c11f674385184ae6b235f1db8bf1fb91 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:49.734099 1167 tablet_bootstrap.cc:492] T 66134661be834f7b811d7927ab629812 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:49.734284 1167 ts_tablet_manager.cc:1403] T 66134661be834f7b811d7927ab629812 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20251025 07:52:49.734709 2169 tablet_replica.cc:333] T 4a1ec8c2ed7f41abb588f592f62097d1 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:49.734951 2169 raft_consensus.cc:2243] T 4a1ec8c2ed7f41abb588f592f62097d1 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:49.735208 1167 raft_consensus.cc:359] T 66134661be834f7b811d7927ab629812 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.735411 1167 raft_consensus.cc:385] T 66134661be834f7b811d7927ab629812 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.735481 1167 raft_consensus.cc:740] T 66134661be834f7b811d7927ab629812 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.735652 2169 raft_consensus.cc:2272] T 4a1ec8c2ed7f41abb588f592f62097d1 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:49.735852 1167 consensus_queue.cc:260] T 66134661be834f7b811d7927ab629812 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:49.736469 455 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6fe43ccecf7047319090ebd035fe24a5 consensus state has no leader
I20251025 07:52:49.736721 455 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:47610 (request call id 118) took 3973 ms. Trace:
I20251025 07:52:49.737134 2169 ts_tablet_manager.cc:1916] T 4a1ec8c2ed7f41abb588f592f62097d1 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:49.736853 455 rpcz_store.cc:276] 1025 07:52:45.763355 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:47.922946 (+2159591us) service_pool.cc:225] Handling call
1025 07:52:47.923436 (+ 490us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:47.923669 (+ 233us) catalog_manager.cc:3708] Apply alter schema
1025 07:52:47.923883 (+ 214us) catalog_manager.cc:3740] Acquired catalog manager lock
1025 07:52:47.923961 (+ 78us) catalog_manager.cc:3898] Updating metadata on disk
1025 07:52:47.941655 (+ 17694us) catalog_manager.cc:3929] Committing alterations to in-memory state
1025 07:52:49.736669 (+1795014us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:47.924981 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:47.925084 (+ 103us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:47.925091 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:47.925093 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:47.925196 (+ 103us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:47.925212 (+ 16us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:47.925216 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:47.925294 (+ 78us) tablet.cc:699] Row locks acquired
1025 07:52:47.925297 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:47.925330 (+ 33us) write_op.cc:270] Start()
1025 07:52:47.925364 (+ 34us) write_op.cc:276] Timestamp: P: 1761378767925315 usec, L: 0
1025 07:52:47.925368 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:47.925551 (+ 183us) log.cc:844] Serialized 405 byte log entry
1025 07:52:47.938691 (+ 13140us) op_driver.cc:464] REPLICATION: finished
1025 07:52:47.938952 (+ 261us) write_op.cc:301] APPLY: starting
1025 07:52:47.938973 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:47.939021 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:47.939065 (+ 44us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:47.939122 (+ 57us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:47.939158 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:47.939165 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:47.939246 (+ 81us) log.cc:844] Serialized 23 byte log entry
1025 07:52:47.939317 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:47.939338 (+ 21us) write_op.cc:454] Released schema lock
1025 07:52:47.939346 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:47.939505 (+ 159us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:47.939511 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":200,"apply.run_cpu_time_us":495,"apply.run_wall_time_us":494,"num_ops":1,"prepare.queue_time_us":473,"prepare.run_cpu_time_us":672,"prepare.run_wall_time_us":672,"raft.queue_time_us":656,"raft.run_cpu_time_us":326,"raft.run_wall_time_us":325,"replication_time_us":13292,"spinlock_wait_cycles":56576}]]}
W20251025 07:52:49.737927 455 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.0.104.1:35777 (request call id 199) took 1743 ms (client timeout 1000 ms). Trace:
W20251025 07:52:49.738054 455 rpcz_store.cc:269] 1025 07:52:47.994250 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.737846 (+1743596us) service_pool.cc:212] Skipping call since client already timed out
1025 07:52:49.737886 (+ 40us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20251025 07:52:49.738340 455 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.0.104.2:42387 (request call id 219) took 1713 ms (client timeout 1000 ms). Trace:
W20251025 07:52:49.738447 455 rpcz_store.cc:269] 1025 07:52:48.024517 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.738267 (+1713750us) service_pool.cc:212] Skipping call since client already timed out
1025 07:52:49.738301 (+ 34us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20251025 07:52:49.738683 455 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.0.104.3:38361 (request call id 217) took 1129 ms (client timeout 1000 ms). Trace:
W20251025 07:52:49.738806 455 rpcz_store.cc:269] 1025 07:52:48.608901 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.738590 (+1129689us) service_pool.cc:212] Skipping call since client already timed out
1025 07:52:49.738640 (+ 50us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20251025 07:52:49.739238 822 heartbeater.cc:344] Connected to a master server at 127.0.104.62:37421
I20251025 07:52:49.739388 822 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
I20251025 07:52:49.739425 455 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_438c3dc0bb6a45fa8e469ee9ab9138ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:49.740085 455 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_438c3dc0bb6a45fa8e469ee9ab9138ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:49.742967 1095 heartbeater.cc:344] Connected to a master server at 127.0.104.62:37421
I20251025 07:52:49.743121 1095 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
I20251025 07:52:49.747833 1167 ts_tablet_manager.cc:1434] T 66134661be834f7b811d7927ab629812 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20251025 07:52:49.754354 1167 tablet_bootstrap.cc:492] T 294d7d20c2f545b8813e632aa058a60c P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:49.756016 1167 tablet_bootstrap.cc:654] T 294d7d20c2f545b8813e632aa058a60c P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.761456 2169 ts_tablet_manager.cc:1929] T 4a1ec8c2ed7f41abb588f592f62097d1 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:49.761595 2169 log.cc:1199] T 4a1ec8c2ed7f41abb588f592f62097d1 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/4a1ec8c2ed7f41abb588f592f62097d1
I20251025 07:52:49.761917 2169 ts_tablet_manager.cc:1950] T 4a1ec8c2ed7f41abb588f592f62097d1 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:49.724922 1166 tablet_bootstrap.cc:654] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.732856 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8917a289bd9b4428a4603429edd94581" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:49.733938 847 leader_election.cc:304] T 67cf96a7dd3f42e19b86746c87e689f6 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.766494 910 raft_consensus.cc:2468] T 8917a289bd9b4428a4603429edd94581 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:49.766940 1169 tablet_bootstrap.cc:492] T fce535a0c0bd45c6bf17f81add297f40 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:49.766875 847 leader_election.cc:304] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.767072 1169 ts_tablet_manager.cc:1403] T fce535a0c0bd45c6bf17f81add297f40 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.092s user 0.004s sys 0.014s
I20251025 07:52:49.767169 2988 raft_consensus.cc:2804] T 67cf96a7dd3f42e19b86746c87e689f6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:49.767241 2988 raft_consensus.cc:493] T 67cf96a7dd3f42e19b86746c87e689f6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:49.767298 2988 raft_consensus.cc:3060] T 67cf96a7dd3f42e19b86746c87e689f6 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.756836 3137 raft_consensus.cc:493] T ee1c8ea694284d80a30f9c05be4cbf49 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.766347 2166 tablet_replica.cc:333] T e4da89f863cc43f3bf6df908bb9dc6bf P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:49.767460 3137 raft_consensus.cc:515] T ee1c8ea694284d80a30f9c05be4cbf49 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.767567 2166 raft_consensus.cc:2243] T e4da89f863cc43f3bf6df908bb9dc6bf P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:49.767705 2166 raft_consensus.cc:2272] T e4da89f863cc43f3bf6df908bb9dc6bf P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:49.767768 1169 raft_consensus.cc:359] T fce535a0c0bd45c6bf17f81add297f40 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:49.767905 1169 raft_consensus.cc:385] T fce535a0c0bd45c6bf17f81add297f40 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.767944 1169 raft_consensus.cc:740] T fce535a0c0bd45c6bf17f81add297f40 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.768096 3137 leader_election.cc:290] T ee1c8ea694284d80a30f9c05be4cbf49 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:49.768162 1169 consensus_queue.cc:260] T fce535a0c0bd45c6bf17f81add297f40 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:49.768476 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee1c8ea694284d80a30f9c05be4cbf49" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:49.768605 1169 ts_tablet_manager.cc:1434] T fce535a0c0bd45c6bf17f81add297f40 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
W20251025 07:52:49.768977 712 leader_election.cc:343] T ee1c8ea694284d80a30f9c05be4cbf49 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.769054 1169 tablet_bootstrap.cc:492] T 645263a61b654f04985c40d04506c087 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:49.769317 2166 ts_tablet_manager.cc:1916] T e4da89f863cc43f3bf6df908bb9dc6bf P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:49.769899 2988 raft_consensus.cc:515] T 67cf96a7dd3f42e19b86746c87e689f6 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.770478 2886 raft_consensus.cc:2804] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:49.770610 2886 raft_consensus.cc:697] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:49.770905 2886 consensus_queue.cc:237] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.773002 448 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:47610 (request call id 119) took 3972 ms. Trace:
I20251025 07:52:49.773056 2988 leader_election.cc:290] T 67cf96a7dd3f42e19b86746c87e689f6 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:49.773409 847 leader_election.cc:304] T 72618756f756484e895e885ef12545c8 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.773128 448 rpcz_store.cc:276] 1025 07:52:45.800749 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:47.923043 (+2122294us) service_pool.cc:225] Handling call
1025 07:52:47.923520 (+ 477us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:47.923790 (+ 270us) catalog_manager.cc:3708] Apply alter schema
1025 07:52:47.923984 (+ 194us) catalog_manager.cc:3740] Acquired catalog manager lock
1025 07:52:47.924094 (+ 110us) catalog_manager.cc:3898] Updating metadata on disk
1025 07:52:47.943912 (+ 19818us) catalog_manager.cc:3929] Committing alterations to in-memory state
1025 07:52:49.772918 (+1829006us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:47.925679 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:47.925772 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:47.925777 (+ 5us) write_op.cc:435] Acquired schema lock
1025 07:52:47.925779 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:47.925865 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:47.925877 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:47.925883 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:47.925937 (+ 54us) tablet.cc:699] Row locks acquired
1025 07:52:47.925941 (+ 4us) write_op.cc:260] PREPARE: finished
1025 07:52:47.925972 (+ 31us) write_op.cc:270] Start()
1025 07:52:47.926002 (+ 30us) write_op.cc:276] Timestamp: P: 1761378767925957 usec, L: 0
1025 07:52:47.926005 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:47.926167 (+ 162us) log.cc:844] Serialized 405 byte log entry
1025 07:52:47.938757 (+ 12590us) op_driver.cc:464] REPLICATION: finished
1025 07:52:47.939603 (+ 846us) write_op.cc:301] APPLY: starting
1025 07:52:47.939619 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:47.939659 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:47.939705 (+ 46us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:47.939758 (+ 53us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:47.939794 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:47.939805 (+ 11us) write_op.cc:312] APPLY: finished
1025 07:52:47.939873 (+ 68us) log.cc:844] Serialized 23 byte log entry
1025 07:52:47.939920 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:47.939938 (+ 18us) write_op.cc:454] Released schema lock
1025 07:52:47.939945 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":40,"spinlock_wait_cycles":22784,"child_traces":[["op",{"apply.queue_time_us":808,"apply.run_cpu_time_us":407,"apply.run_wall_time_us":408,"num_ops":1,"prepare.queue_time_us":1159,"prepare.run_cpu_time_us":561,"prepare.run_wall_time_us":560,"raft.queue_time_us":422,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":12738,"thread_start_us":17114,"threads_started":1}]]}
W20251025 07:52:49.773766 847 leader_election.cc:343] T af743ee344e543a0aa6da3cd7a30e467 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: af743ee344e543a0aa6da3cd7a30e467
I20251025 07:52:49.773857 847 leader_election.cc:304] T af743ee344e543a0aa6da3cd7a30e467 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.773965 448 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_2c8bad53187a49858df0c2bde3485e61" } modify_external_catalogs: true
I20251025 07:52:49.774047 2991 raft_consensus.cc:2804] T 72618756f756484e895e885ef12545c8 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:49.774179 2991 raft_consensus.cc:697] T 72618756f756484e895e885ef12545c8 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:49.774173 448 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_2c8bad53187a49858df0c2bde3485e61" } modify_external_catalogs: true
I20251025 07:52:49.774411 2991 consensus_queue.cc:237] T 72618756f756484e895e885ef12545c8 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.775020 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67cf96a7dd3f42e19b86746c87e689f6" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:49.775538 850 leader_election.cc:343] T 67cf96a7dd3f42e19b86746c87e689f6 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.779194 1167 tablet_bootstrap.cc:492] T 294d7d20c2f545b8813e632aa058a60c P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:49.779361 1167 ts_tablet_manager.cc:1403] T 294d7d20c2f545b8813e632aa058a60c P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.000s
I20251025 07:52:49.780318 1167 raft_consensus.cc:359] T 294d7d20c2f545b8813e632aa058a60c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.780524 1167 raft_consensus.cc:385] T 294d7d20c2f545b8813e632aa058a60c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.780588 1167 raft_consensus.cc:740] T 294d7d20c2f545b8813e632aa058a60c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.781229 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f0c4f0e8f014e59be9e46d2afe0be9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.781847 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 80ee75935d7e461d9d4066fac9819bef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.782819 2990 raft_consensus.cc:2749] T af743ee344e543a0aa6da3cd7a30e467 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:49.782917 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e6544202d4f489d82f22d6073ef240d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.783802 894 tablet_service.cc:1552] Processing DeleteTablet for tablet ffe7c8e1d6c44872b7b65e512764a56a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.784029 745 tablet_service.cc:1552] Processing DeleteTablet for tablet ce1ebb1ec14b4801930a72765bca1a7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:49.784375 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b6383c99e9e4c90a374da9ccd830a6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:49.784534 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 80fe3255ed614270bd8352a2853fbc43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:49.784690 749 tablet_service.cc:1552] Processing DeleteTablet for tablet ebc824ec4c60411dab1a2ea4a385148e with delete_type TABLET_DATA_DELETED (replaced by tablet 29dbaf9149134f548a974a5779f7da67) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:49.784833 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 91728987303b41fd840eb56f3a311c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:49.784955 756 tablet_service.cc:1552] Processing DeleteTablet for tablet a77eb82f26ae43b48de1e85693372d34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:49.785120 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 0f5f6ca6c3cd4d369690e425387a4344 with delete_type TABLET_DATA_DELETED (replaced by tablet 22fc90c704f64d2f9f4b8c54dee45576) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:49.785300 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 36c8fcafd5464b4e999276a8062a812a with delete_type TABLET_DATA_DELETED (replaced by tablet 181059cc79d54999bf58504c85b739e9) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:49.785342 894 tablet_service.cc:1552] Processing DeleteTablet for tablet dda0b26e4f1a44928bc4b84f333901c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.785475 741 tablet_service.cc:1552] Processing DeleteTablet for tablet aaef24ccdfba45fe8204327723df834e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:49.785882 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 80fe3255ed614270bd8352a2853fbc43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.786479 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b6383c99e9e4c90a374da9ccd830a6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.786989 1167 consensus_queue.cc:260] T 294d7d20c2f545b8813e632aa058a60c P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.787071 894 tablet_service.cc:1552] Processing DeleteTablet for tablet fce535a0c0bd45c6bf17f81add297f40 with delete_type TABLET_DATA_DELETED (replaced by tablet 079f91be4217493d8ab865f5ccce0cb1) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.787693 894 tablet_service.cc:1552] Processing DeleteTablet for tablet ce1ebb1ec14b4801930a72765bca1a7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.788404 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 91728987303b41fd840eb56f3a311c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.788493 1167 ts_tablet_manager.cc:1434] T 294d7d20c2f545b8813e632aa058a60c P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251025 07:52:49.789131 894 tablet_service.cc:1552] Processing DeleteTablet for tablet ebc824ec4c60411dab1a2ea4a385148e with delete_type TABLET_DATA_DELETED (replaced by tablet 29dbaf9149134f548a974a5779f7da67) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.789395 1167 tablet_bootstrap.cc:492] T 19f1b56cdbd24ebe8688cc05fb795d07 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:49.789970 894 tablet_service.cc:1552] Processing DeleteTablet for tablet a77eb82f26ae43b48de1e85693372d34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.790601 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 050a5d3b5f934dd19bf2e221f2fe0780 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.791071 1167 tablet_bootstrap.cc:654] T 19f1b56cdbd24ebe8688cc05fb795d07 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.791755 741 tablet_service.cc:1552] Processing DeleteTablet for tablet 050a5d3b5f934dd19bf2e221f2fe0780 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:49.791927 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 062c0f3e51a3487880cea373f8cf1f54 with delete_type TABLET_DATA_DELETED (replaced by tablet b56209e4934e4407a8b4518ca4af0d46) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:49.792069 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 04449cc97f404a19b6a1b20dea074f95 with delete_type TABLET_DATA_DELETED (replaced by tablet f70091024c034c118fb86578db19a31c) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:49.792518 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 36c8fcafd5464b4e999276a8062a812a with delete_type TABLET_DATA_DELETED (replaced by tablet 181059cc79d54999bf58504c85b739e9) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.792970 3137 consensus_queue.cc:1048] T ed243540f2d746b980c25687fbca7414 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251025 07:52:49.793946 850 consensus_peers.cc:597] T 288e543bb279410c91c01dd1623e110f P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:49.795032 755 tablet_service.cc:1552] Processing DeleteTablet for tablet f43ef51ee45d47a99ec732b49d50d3b4 with delete_type TABLET_DATA_DELETED (replaced by tablet e429b94326cc4827a03968fd92de38e0) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:49.796267 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b940fa5ae26427dafbcfd26372754b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:49.798034 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 04449cc97f404a19b6a1b20dea074f95 with delete_type TABLET_DATA_DELETED (replaced by tablet f70091024c034c118fb86578db19a31c) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.798740 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 716fb61312e94a4d9eb8ab8336ca140c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:49.798748 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet dda0b26e4f1a44928bc4b84f333901c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:49.799278 755 tablet_service.cc:1552] Processing DeleteTablet for tablet f97acc9a638b4d5a968dcc3412a384bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:49.799633 894 tablet_service.cc:1552] Processing DeleteTablet for tablet f43ef51ee45d47a99ec732b49d50d3b4 with delete_type TABLET_DATA_DELETED (replaced by tablet e429b94326cc4827a03968fd92de38e0) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.799705 1045 raft_consensus.cc:3060] T fa826b85f0c94150a17d4e417dd3d62f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.800010 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 88eb51447e0f4080baa49dd902deb24f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:49.800436 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b940fa5ae26427dafbcfd26372754b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.800834 710 leader_election.cc:304] T 8917a289bd9b4428a4603429edd94581 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.801811 3142 raft_consensus.cc:2804] T 8917a289bd9b4428a4603429edd94581 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:49.801888 3142 raft_consensus.cc:493] T 8917a289bd9b4428a4603429edd94581 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:49.801954 3142 raft_consensus.cc:3060] T 8917a289bd9b4428a4603429edd94581 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.802228 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 0f5f6ca6c3cd4d369690e425387a4344 with delete_type TABLET_DATA_DELETED (replaced by tablet 22fc90c704f64d2f9f4b8c54dee45576) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.802947 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 4c155aa7fa5b4d68bbab1918b8432372 with delete_type TABLET_DATA_DELETED (replaced by tablet 110a8e9e4c6c434284c0ae14dd6987f7) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.804203 2988 raft_consensus.cc:493] T 6ce62847e80f48e8807fdbbbdc50803c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.804359 2988 raft_consensus.cc:515] T 6ce62847e80f48e8807fdbbbdc50803c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.804788 3142 raft_consensus.cc:515] T 8917a289bd9b4428a4603429edd94581 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.805312 2988 leader_election.cc:290] T 6ce62847e80f48e8807fdbbbdc50803c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:49.805377 3142 leader_election.cc:290] T 8917a289bd9b4428a4603429edd94581 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:49.805370 894 tablet_service.cc:1552] Processing DeleteTablet for tablet d9bfd7c0c3ad4e5b98b6708793d61d41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.805402 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ce62847e80f48e8807fdbbbdc50803c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:49.805960 3137 raft_consensus.cc:493] T dbb254115a574bf4976aff0ea56f3bd9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.806066 3137 raft_consensus.cc:515] T dbb254115a574bf4976aff0ea56f3bd9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.806392 894 tablet_service.cc:1552] Processing DeleteTablet for tablet aaef24ccdfba45fe8204327723df834e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.806653 3137 leader_election.cc:290] T dbb254115a574bf4976aff0ea56f3bd9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:49.807089 850 leader_election.cc:343] T 6ce62847e80f48e8807fdbbbdc50803c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.807194 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 60c60a9ee37f43978910bd067a098bb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:43 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.808079 894 tablet_service.cc:1552] Processing DeleteTablet for tablet f97acc9a638b4d5a968dcc3412a384bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.808881 894 tablet_service.cc:1552] Processing DeleteTablet for tablet b3a0af893a8947848e12093d0a7f0229 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.809298 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbb254115a574bf4976aff0ea56f3bd9" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:49.809435 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8917a289bd9b4428a4603429edd94581" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:49.810184 894 tablet_service.cc:1552] Processing DeleteTablet for tablet f661d5909657455581dcb5e6eadd3c0b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:59616
W20251025 07:52:49.810761 712 leader_election.cc:343] T dbb254115a574bf4976aff0ea56f3bd9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.810865 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 48ff4ce7846b4e56a37eee9bcb17bd5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:58412
W20251025 07:52:49.810963 712 leader_election.cc:343] T 8917a289bd9b4428a4603429edd94581 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.811139 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 26c849d376d143c6bef0c8e0e1d7d03a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:49.811177 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 88eb51447e0f4080baa49dd902deb24f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.812949 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 062c0f3e51a3487880cea373f8cf1f54 with delete_type TABLET_DATA_DELETED (replaced by tablet b56209e4934e4407a8b4518ca4af0d46) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.813869 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 26c849d376d143c6bef0c8e0e1d7d03a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.814824 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67cf96a7dd3f42e19b86746c87e689f6" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:49.814884 894 tablet_service.cc:1552] Processing DeleteTablet for tablet f656945454fe4921ab819c772a089eb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.815009 772 raft_consensus.cc:3060] T 67cf96a7dd3f42e19b86746c87e689f6 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.817844 772 raft_consensus.cc:2468] T 67cf96a7dd3f42e19b86746c87e689f6 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:49.818938 894 tablet_service.cc:1552] Processing DeleteTablet for tablet e54fe7cbcffa40c4b04144b47bee2b48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.819986 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 48ff4ce7846b4e56a37eee9bcb17bd5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.820158 1167 tablet_bootstrap.cc:492] T 19f1b56cdbd24ebe8688cc05fb795d07 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:49.820289 1167 ts_tablet_manager.cc:1403] T 19f1b56cdbd24ebe8688cc05fb795d07 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.004s
I20251025 07:52:49.820911 1167 raft_consensus.cc:359] T 19f1b56cdbd24ebe8688cc05fb795d07 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.821067 1167 raft_consensus.cc:385] T 19f1b56cdbd24ebe8688cc05fb795d07 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.821118 1167 raft_consensus.cc:740] T 19f1b56cdbd24ebe8688cc05fb795d07 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.821317 1167 consensus_queue.cc:260] T 19f1b56cdbd24ebe8688cc05fb795d07 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.821542 3137 raft_consensus.cc:493] T 9903e80a26fd469a8744dc855534ba75 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.821684 3137 raft_consensus.cc:515] T 9903e80a26fd469a8744dc855534ba75 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.821770 1167 ts_tablet_manager.cc:1434] T 19f1b56cdbd24ebe8688cc05fb795d07 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251025 07:52:49.822014 1167 tablet_bootstrap.cc:492] T d40519ed9ee746348b17be2786d60e95 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:49.822284 3137 leader_election.cc:290] T 9903e80a26fd469a8744dc855534ba75 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:49.822952 894 tablet_service.cc:1552] Processing DeleteTablet for tablet d77704b4fe6c4456a2ba40ffec0c60fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.823343 2166 ts_tablet_manager.cc:1929] T e4da89f863cc43f3bf6df908bb9dc6bf P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:49.823433 2166 log.cc:1199] T e4da89f863cc43f3bf6df908bb9dc6bf P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/e4da89f863cc43f3bf6df908bb9dc6bf
I20251025 07:52:49.823731 1167 tablet_bootstrap.cc:654] T d40519ed9ee746348b17be2786d60e95 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.823761 2166 ts_tablet_manager.cc:1950] T e4da89f863cc43f3bf6df908bb9dc6bf P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:49.824491 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 203) took 7918 ms. Trace:
I20251025 07:52:49.824576 2166 rpcz_store.cc:276] 1025 07:52:41.905524 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.906726 (+ 1202us) service_pool.cc:225] Handling call
1025 07:52:49.766293 (+7859567us) ts_tablet_manager.cc:1151] Deleting tablet e4da89f863cc43f3bf6df908bb9dc6bf
1025 07:52:49.766304 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.788746 (+ 22442us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.823223 (+ 34477us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.824471 (+ 1248us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7859425}
I20251025 07:52:49.825007 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet e4da89f863cc43f3bf6df908bb9dc6bf (table default.Table_3d995c9c75174000b6cb98779cc0bb2f [id=6c1ffab0043545c2b7cce93a641eaa1d]) successfully deleted
I20251025 07:52:49.826797 894 tablet_service.cc:1552] Processing DeleteTablet for tablet e4da89f863cc43f3bf6df908bb9dc6bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.828348 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9903e80a26fd469a8744dc855534ba75" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:49.828920 2988 raft_consensus.cc:493] T 243b8828fc344bacab46d7bab6bd2dec P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 07:52:49.828917 712 leader_election.cc:343] T 9903e80a26fd469a8744dc855534ba75 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.829046 2988 raft_consensus.cc:515] T 243b8828fc344bacab46d7bab6bd2dec P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.829553 2988 leader_election.cc:290] T 243b8828fc344bacab46d7bab6bd2dec P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:49.830328 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "243b8828fc344bacab46d7bab6bd2dec" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:49.830778 850 leader_election.cc:343] T 243b8828fc344bacab46d7bab6bd2dec P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.832877 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f7c15529be54a11961f385a953b0b03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.833602 894 tablet_service.cc:1552] Processing DeleteTablet for tablet e47c4f741f354a19b49777bd06e54764 with delete_type TABLET_DATA_DELETED (replaced by tablet 755f0d6c87044e20a8020f8410940123) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.834291 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 95efd553493e4966bc48a01cac6e1e10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.834697 2169 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44572 (request call id 345) took 1867 ms. Trace:
I20251025 07:52:49.834836 2169 rpcz_store.cc:276] 1025 07:52:47.894921 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:47.898205 (+ 3284us) service_pool.cc:225] Handling call
1025 07:52:49.734664 (+1836459us) ts_tablet_manager.cc:1151] Deleting tablet 4a1ec8c2ed7f41abb588f592f62097d1
1025 07:52:49.734672 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.741921 (+ 7249us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.761329 (+ 19408us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.762657 (+ 1328us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1836222}
I20251025 07:52:49.835565 905 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee1c8ea694284d80a30f9c05be4cbf49" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:49.835825 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbb254115a574bf4976aff0ea56f3bd9" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:49.836409 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8917a289bd9b4428a4603429edd94581" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:49.836541 909 raft_consensus.cc:3060] T 8917a289bd9b4428a4603429edd94581 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.837013 907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9903e80a26fd469a8744dc855534ba75" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:49.837311 765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ce62847e80f48e8807fdbbbdc50803c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:49.837491 765 raft_consensus.cc:2468] T 6ce62847e80f48e8807fdbbbdc50803c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
W20251025 07:52:49.837845 850 consensus_peers.cc:597] T 437da5ded9b648abb668a8cb4896452c P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:49.838670 765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "243b8828fc344bacab46d7bab6bd2dec" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:49.838815 765 raft_consensus.cc:2468] T 243b8828fc344bacab46d7bab6bd2dec P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:49.840951 2988 raft_consensus.cc:493] T a6ae1315b28940d49b667e3c80347021 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.841061 2988 raft_consensus.cc:515] T a6ae1315b28940d49b667e3c80347021 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.841533 2988 leader_election.cc:290] T a6ae1315b28940d49b667e3c80347021 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:49.841813 2988 raft_consensus.cc:493] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.841913 2988 raft_consensus.cc:515] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.842479 2988 leader_election.cc:290] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:49.843111 2988 consensus_queue.cc:1048] T 288e543bb279410c91c01dd1623e110f P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:49.845125 755 tablet_service.cc:1552] Processing DeleteTablet for tablet d9bfd7c0c3ad4e5b98b6708793d61d41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:49.845564 755 tablet_service.cc:1552] Processing DeleteTablet for tablet a4e4b99be0434baaa75d988281797846 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:49.846009 909 raft_consensus.cc:2468] T 8917a289bd9b4428a4603429edd94581 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:49.846079 755 tablet_service.cc:1552] Processing DeleteTablet for tablet af743ee344e543a0aa6da3cd7a30e467 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:49.847257 755 tablet_service.cc:1552] Processing DeleteTablet for tablet f3c87a83082b421cb4be546ac4ce7ad8 with delete_type TABLET_DATA_DELETED (replaced by tablet b17d2a9ad6be49d29f2413666fba7e69) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:49.847782 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d6a9937d10a4cc2813671dd1bfa1664 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:49.848466 755 tablet_service.cc:1552] Processing DeleteTablet for tablet e4da89f863cc43f3bf6df908bb9dc6bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
W20251025 07:52:49.848624 850 consensus_peers.cc:597] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:49.848902 755 tablet_service.cc:1552] Processing DeleteTablet for tablet d77704b4fe6c4456a2ba40ffec0c60fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:49.850076 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 6897405bbc1c47a4ab0aa3af80e60e3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:58412
W20251025 07:52:49.853104 850 consensus_peers.cc:597] T 4a1ec8c2ed7f41abb588f592f62097d1 P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4a1ec8c2ed7f41abb588f592f62097d1. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:49.858944 847 leader_election.cc:304] T 67cf96a7dd3f42e19b86746c87e689f6 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.859329 2991 raft_consensus.cc:2804] T 67cf96a7dd3f42e19b86746c87e689f6 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:49.859458 2991 raft_consensus.cc:697] T 67cf96a7dd3f42e19b86746c87e689f6 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:49.859769 847 leader_election.cc:304] T 6ce62847e80f48e8807fdbbbdc50803c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.859716 2991 consensus_queue.cc:237] T 67cf96a7dd3f42e19b86746c87e689f6 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.859956 2990 raft_consensus.cc:2804] T 6ce62847e80f48e8807fdbbbdc50803c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:49.860020 2990 raft_consensus.cc:493] T 6ce62847e80f48e8807fdbbbdc50803c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:49.860077 2990 raft_consensus.cc:3060] T 6ce62847e80f48e8807fdbbbdc50803c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.861243 455 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 120) took 4060 ms. Trace:
I20251025 07:52:49.861377 455 rpcz_store.cc:276] 1025 07:52:45.800879 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.739233 (+3938354us) service_pool.cc:225] Handling call
1025 07:52:49.740178 (+ 945us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:49.740690 (+ 512us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:49.860926 (+120236us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:49.861100 (+ 174us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:49.861202 (+ 102us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:49.776428 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:49.776553 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:49.776560 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:49.776562 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:49.791562 (+ 15000us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:49.791580 (+ 18us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:49.791585 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:49.791674 (+ 89us) tablet.cc:699] Row locks acquired
1025 07:52:49.791677 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:49.791722 (+ 45us) write_op.cc:270] Start()
1025 07:52:49.791760 (+ 38us) write_op.cc:276] Timestamp: P: 1761378769791703 usec, L: 0
1025 07:52:49.791763 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:49.792004 (+ 241us) log.cc:844] Serialized 433 byte log entry
1025 07:52:49.853257 (+ 61253us) op_driver.cc:464] REPLICATION: finished
1025 07:52:49.860006 (+ 6749us) write_op.cc:301] APPLY: starting
1025 07:52:49.860024 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:49.860067 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:49.860130 (+ 63us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:49.860242 (+ 112us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:49.860279 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:49.860289 (+ 10us) write_op.cc:312] APPLY: finished
1025 07:52:49.860393 (+ 104us) log.cc:844] Serialized 29 byte log entry
1025 07:52:49.860824 (+ 431us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:49.860842 (+ 18us) write_op.cc:454] Released schema lock
1025 07:52:49.860848 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":6083,"num_ops":2,"prepare.queue_time_us":35217,"prepare.run_cpu_time_us":952,"prepare.run_wall_time_us":32519,"raft.queue_time_us":2552,"raft.run_cpu_time_us":309,"raft.run_wall_time_us":309,"replication_time_us":61450,"thread_start_us":36996,"threads_started":2}]]}
I20251025 07:52:49.864501 2990 raft_consensus.cc:515] T 6ce62847e80f48e8807fdbbbdc50803c P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.865249 2990 leader_election.cc:290] T 6ce62847e80f48e8807fdbbbdc50803c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:49.865422 2988 raft_consensus.cc:493] T 792f32cbd66449ca98813c669b2a261d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.865577 2988 raft_consensus.cc:515] T 792f32cbd66449ca98813c669b2a261d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.865840 3142 raft_consensus.cc:493] T 66134661be834f7b811d7927ab629812 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.865990 3142 raft_consensus.cc:515] T 66134661be834f7b811d7927ab629812 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.866292 2988 leader_election.cc:290] T 792f32cbd66449ca98813c669b2a261d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:49.866647 3142 leader_election.cc:290] T 66134661be834f7b811d7927ab629812 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:49.870751 1167 tablet_bootstrap.cc:492] T d40519ed9ee746348b17be2786d60e95 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:49.870911 1167 ts_tablet_manager.cc:1403] T d40519ed9ee746348b17be2786d60e95 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.007s
I20251025 07:52:49.871318 2166 tablet_replica.cc:333] T e47c4f741f354a19b49777bd06e54764 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:49.871505 2166 raft_consensus.cc:2243] T e47c4f741f354a19b49777bd06e54764 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:49.871632 1167 raft_consensus.cc:359] T d40519ed9ee746348b17be2786d60e95 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.871803 1167 raft_consensus.cc:385] T d40519ed9ee746348b17be2786d60e95 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.871856 1167 raft_consensus.cc:740] T d40519ed9ee746348b17be2786d60e95 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
W20251025 07:52:49.872598 850 consensus_peers.cc:597] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:49.873005 2166 raft_consensus.cc:2272] T e47c4f741f354a19b49777bd06e54764 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:49.873026 1167 consensus_queue.cc:260] T d40519ed9ee746348b17be2786d60e95 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.873533 1167 ts_tablet_manager.cc:1434] T d40519ed9ee746348b17be2786d60e95 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251025 07:52:49.873893 1167 tablet_bootstrap.cc:492] T 8707b93d491e4e4386d148087a2f0870 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:49.875906 1167 tablet_bootstrap.cc:654] T 8707b93d491e4e4386d148087a2f0870 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.878043 847 leader_election.cc:304] T 243b8828fc344bacab46d7bab6bd2dec P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.878402 2988 raft_consensus.cc:2804] T 243b8828fc344bacab46d7bab6bd2dec P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:49.878521 2988 raft_consensus.cc:493] T 243b8828fc344bacab46d7bab6bd2dec P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:49.878638 2988 raft_consensus.cc:3060] T 243b8828fc344bacab46d7bab6bd2dec P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.841899 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6ae1315b28940d49b667e3c80347021" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:49.842732 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5fc092c2bb2d41ac8c5b4fc07460c8ec" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:49.851068 3011 raft_consensus.cc:493] T fd265cdbf6a8440393a2b5d8240d1872 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.879364 3011 raft_consensus.cc:515] T fd265cdbf6a8440393a2b5d8240d1872 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:49.879844 453 catalog_manager.cc:6194] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: default.Table_4791672658f043f6b97b9320a531ebb5 [id=dfd20cf9c3e541c4abe23d59e0aa7f2a] alter complete (version 1)
I20251025 07:52:49.879905 3011 leader_election.cc:290] T fd265cdbf6a8440393a2b5d8240d1872 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:49.854332 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 80ee75935d7e461d9d4066fac9819bef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:49.880108 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b940fa5ae26427dafbcfd26372754b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:49.880216 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet b3a0af893a8947848e12093d0a7f0229 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:49.880303 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet f661d5909657455581dcb5e6eadd3c0b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:49.880415 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 88eb51447e0f4080baa49dd902deb24f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:49.880506 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 60c60a9ee37f43978910bd067a098bb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:43 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:49.880623 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet f97acc9a638b4d5a968dcc3412a384bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:49.880714 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 26c849d376d143c6bef0c8e0e1d7d03a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:49.880795 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 48ff4ce7846b4e56a37eee9bcb17bd5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:49.880880 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 062c0f3e51a3487880cea373f8cf1f54 with delete_type TABLET_DATA_DELETED (replaced by tablet b56209e4934e4407a8b4518ca4af0d46) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:49.854487 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e6544202d4f489d82f22d6073ef240d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:49.854588 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet ffe7c8e1d6c44872b7b65e512764a56a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:49.855008 1010 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f0c4f0e8f014e59be9e46d2afe0be9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:49.855108 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 80fe3255ed614270bd8352a2853fbc43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:49.855188 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b6383c99e9e4c90a374da9ccd830a6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:49.855266 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet fce535a0c0bd45c6bf17f81add297f40 with delete_type TABLET_DATA_DELETED (replaced by tablet 079f91be4217493d8ab865f5ccce0cb1) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:49.855417 1011 tablet_service.cc:1552] Processing DeleteTablet for tablet ebc824ec4c60411dab1a2ea4a385148e with delete_type TABLET_DATA_DELETED (replaced by tablet 29dbaf9149134f548a974a5779f7da67) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:49.855885 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet a77eb82f26ae43b48de1e85693372d34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:49.865902 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ce62847e80f48e8807fdbbbdc50803c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:49.866914 1043 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "792f32cbd66449ca98813c669b2a261d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:49.867246 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66134661be834f7b811d7927ab629812" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:49.871989 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet ce1ebb1ec14b4801930a72765bca1a7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:49.872144 1012 tablet_service.cc:1552] Processing DeleteTablet for tablet 11bfe0c87ca941a3846462c8a4512d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:44 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:49.872253 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet 91728987303b41fd840eb56f3a311c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:49.872370 1014 tablet_service.cc:1552] Processing DeleteTablet for tablet 050a5d3b5f934dd19bf2e221f2fe0780 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:49.872493 1015 tablet_service.cc:1552] Processing DeleteTablet for tablet 36c8fcafd5464b4e999276a8062a812a with delete_type TABLET_DATA_DELETED (replaced by tablet 181059cc79d54999bf58504c85b739e9) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:49.872944 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet aaef24ccdfba45fe8204327723df834e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:49.873891 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 0f5f6ca6c3cd4d369690e425387a4344 with delete_type TABLET_DATA_DELETED (replaced by tablet 22fc90c704f64d2f9f4b8c54dee45576) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:49.874200 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet 4c155aa7fa5b4d68bbab1918b8432372 with delete_type TABLET_DATA_DELETED (replaced by tablet 110a8e9e4c6c434284c0ae14dd6987f7) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:49.874444 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet d9bfd7c0c3ad4e5b98b6708793d61d41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:49.874698 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet 04449cc97f404a19b6a1b20dea074f95 with delete_type TABLET_DATA_DELETED (replaced by tablet f70091024c034c118fb86578db19a31c) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:49.876236 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet f43ef51ee45d47a99ec732b49d50d3b4 with delete_type TABLET_DATA_DELETED (replaced by tablet e429b94326cc4827a03968fd92de38e0) from {username='slave'} at 127.0.0.1:44572
W20251025 07:52:49.882372 712 leader_election.cc:343] T 66134661be834f7b811d7927ab629812 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.882686 2988 raft_consensus.cc:515] T 243b8828fc344bacab46d7bab6bd2dec P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:49.883210 982 leader_election.cc:343] T fd265cdbf6a8440393a2b5d8240d1872 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: fd265cdbf6a8440393a2b5d8240d1872
I20251025 07:52:49.883455 2988 leader_election.cc:290] T 243b8828fc344bacab46d7bab6bd2dec P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
W20251025 07:52:49.883551 850 leader_election.cc:343] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:49.884078 850 leader_election.cc:343] T 6ce62847e80f48e8807fdbbbdc50803c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:49.884232 850 leader_election.cc:343] T 792f32cbd66449ca98813c669b2a261d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:49.884382 850 leader_election.cc:343] T a6ae1315b28940d49b667e3c80347021 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.884589 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "243b8828fc344bacab46d7bab6bd2dec" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:49.884671 983 leader_election.cc:343] T fd265cdbf6a8440393a2b5d8240d1872 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: fd265cdbf6a8440393a2b5d8240d1872
I20251025 07:52:49.884783 983 leader_election.cc:304] T fd265cdbf6a8440393a2b5d8240d1872 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:49.885260 2597 raft_consensus.cc:2749] T fd265cdbf6a8440393a2b5d8240d1872 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:49.886190 850 leader_election.cc:343] T 243b8828fc344bacab46d7bab6bd2dec P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.886672 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 4a1ec8c2ed7f41abb588f592f62097d1 (table default.Table_91898f2a551144d18df0309c827855bf [id=aaa734f6549741768b0156bdf5cc25af]) successfully deleted
I20251025 07:52:49.887681 2990 consensus_queue.cc:1048] T 437da5ded9b648abb668a8cb4896452c P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251025 07:52:49.888530 875 tablet_service.cc:1552] Processing DeleteTablet for tablet c11f674385184ae6b235f1db8bf1fb91 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.888883 875 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e098c9c4613459295b6043048cba587 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.889245 875 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c4970c8a491460e8b4b1c29fde83e4c with delete_type TABLET_DATA_DELETED (replaced by tablet 783487b60c144f719ed2eb3c67b5ccab) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.889616 875 tablet_service.cc:1552] Processing DeleteTablet for tablet 32a143a715cb485cb4501f4e28c68252 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c389d798cb94d8a972683c57b1f9f1f) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.889958 875 tablet_service.cc:1552] Processing DeleteTablet for tablet cc407c7eb12e4eadb41fb21385611c94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.890435 875 tablet_service.cc:1552] Processing DeleteTablet for tablet 6897405bbc1c47a4ab0aa3af80e60e3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.891175 875 tablet_service.cc:1552] Processing DeleteTablet for tablet 1912e59548874e0e8e989050ec0064c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.891515 875 tablet_service.cc:1552] Processing DeleteTablet for tablet 0540d6a10cf04fa89756623e9aaaeb68 with delete_type TABLET_DATA_DELETED (replaced by tablet 55a4a524676b4b9aa9464e934b662f30) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.891855 875 tablet_service.cc:1552] Processing DeleteTablet for tablet 716fb61312e94a4d9eb8ab8336ca140c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.892174 875 tablet_service.cc:1552] Processing DeleteTablet for tablet b02f5c0ab0b24288bb50a3ba0e6bd78a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.892498 875 tablet_service.cc:1552] Processing DeleteTablet for tablet ee1c8ea694284d80a30f9c05be4cbf49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.892812 875 tablet_service.cc:1552] Processing DeleteTablet for tablet 6dcab53a020847a1aa27930265ba0205 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.893121 875 tablet_service.cc:1552] Processing DeleteTablet for tablet 72dc084f8efc45478771c247e07a16f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.893273 2169 tablet_replica.cc:333] T 716fb61312e94a4d9eb8ab8336ca140c P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:49.893484 875 tablet_service.cc:1552] Processing DeleteTablet for tablet 50ae9bca6bed4f749d54191ac1543410 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.893473 2169 raft_consensus.cc:2243] T 716fb61312e94a4d9eb8ab8336ca140c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:49.893644 2169 raft_consensus.cc:2272] T 716fb61312e94a4d9eb8ab8336ca140c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:49.893841 875 tablet_service.cc:1552] Processing DeleteTablet for tablet fd265cdbf6a8440393a2b5d8240d1872 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.894176 875 tablet_service.cc:1552] Processing DeleteTablet for tablet 1b5ac7e845a24b2eb5b2429d47361232 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.894490 2169 ts_tablet_manager.cc:1916] T 716fb61312e94a4d9eb8ab8336ca140c P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:49.894522 875 tablet_service.cc:1552] Processing DeleteTablet for tablet a7ab3150a35b4caca9119a1374d3f2f7 with delete_type TABLET_DATA_DELETED (replaced by tablet c19c97f881cb45218d4a6f6f231aa96e) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.896373 755 tablet_service.cc:1552] Processing DeleteTablet for tablet c59f6e6eb9594a8e83f109a09b034045 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:49.897058 755 tablet_service.cc:1552] Processing DeleteTablet for tablet b02f5c0ab0b24288bb50a3ba0e6bd78a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:49.898226 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c4970c8a491460e8b4b1c29fde83e4c with delete_type TABLET_DATA_DELETED (replaced by tablet 783487b60c144f719ed2eb3c67b5ccab) from {username='slave'} at 127.0.0.1:58412
W20251025 07:52:49.898676 440 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 8707b93d491e4e4386d148087a2f0870 (table default.Table_799dcb1cef2f4fe6b9b8f17f9f1026c5 [id=59840a567e5244e8a9024cbad42471e4], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8707b93d491e4e4386d148087a2f0870 consensus state has no leader
I20251025 07:52:49.899366 755 tablet_service.cc:1552] Processing DeleteTablet for tablet fd265cdbf6a8440393a2b5d8240d1872 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:49.900494 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c7a9d7a4fb3449bb57eafa83230a839 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:33 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:49.900490 3142 raft_consensus.cc:493] T aea8fcf0154a4bd4a05b19e27ab78b79 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.900820 3142 raft_consensus.cc:515] T aea8fcf0154a4bd4a05b19e27ab78b79 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.901691 3142 leader_election.cc:290] T aea8fcf0154a4bd4a05b19e27ab78b79 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:49.759150 2146 ts_tablet_manager.cc:1929] T 2b594d4718344aafbdaec34040695589 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:49.742630 960 heartbeater.cc:344] Connected to a master server at 127.0.104.62:37421
I20251025 07:52:49.902330 2146 log.cc:1199] T 2b594d4718344aafbdaec34040695589 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/2b594d4718344aafbdaec34040695589
I20251025 07:52:49.902482 960 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
I20251025 07:52:49.902944 2146 ts_tablet_manager.cc:1950] T 2b594d4718344aafbdaec34040695589 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:49.903465 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 275) took 5316 ms. Trace:
I20251025 07:52:49.903610 2146 rpcz_store.cc:276] 1025 07:52:44.586875 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.587217 (+ 342us) service_pool.cc:225] Handling call
1025 07:52:49.680063 (+5092846us) ts_tablet_manager.cc:1151] Deleting tablet 2b594d4718344aafbdaec34040695589
1025 07:52:49.680071 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.747119 (+ 67048us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.759023 (+ 11904us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.903451 (+144428us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3061,"tablet-delete.queue_time_us":5075355}
I20251025 07:52:49.904203 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 285) took 5299 ms. Trace:
I20251025 07:52:49.904317 2146 rpcz_store.cc:276] 1025 07:52:44.605126 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.605185 (+ 59us) service_pool.cc:225] Handling call
1025 07:52:49.904116 (+5298931us) ts_tablet_manager.cc:1151] Deleting tablet f170424be2f643e7af499834955472b0
1025 07:52:49.904125 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.904192 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5298756}
I20251025 07:52:49.905748 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aea8fcf0154a4bd4a05b19e27ab78b79" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:49.906186 3011 raft_consensus.cc:493] T fce535a0c0bd45c6bf17f81add297f40 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 07:52:49.906258 712 leader_election.cc:343] T aea8fcf0154a4bd4a05b19e27ab78b79 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.906333 3011 raft_consensus.cc:515] T fce535a0c0bd45c6bf17f81add297f40 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:49.906647 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 2b594d4718344aafbdaec34040695589 (table default.Table_e08a3ff70f3a411e82b4bb292d768961 [id=8d075a6652b14a37a2bfbfddc6139eb9]) successfully deleted
I20251025 07:52:49.906932 3011 leader_election.cc:290] T fce535a0c0bd45c6bf17f81add297f40 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
W20251025 07:52:49.906970 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet f170424be2f643e7af499834955472b0 because the tablet was not found. No further retry: Not found: Tablet not found: f170424be2f643e7af499834955472b0
W20251025 07:52:49.907449 983 leader_election.cc:343] T fce535a0c0bd45c6bf17f81add297f40 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: fce535a0c0bd45c6bf17f81add297f40
I20251025 07:52:49.907557 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fce535a0c0bd45c6bf17f81add297f40" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:49.907800 772 raft_consensus.cc:2393] T fce535a0c0bd45c6bf17f81add297f40 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:49.908210 982 leader_election.cc:304] T fce535a0c0bd45c6bf17f81add297f40 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:49.908654 3011 raft_consensus.cc:3060] T fce535a0c0bd45c6bf17f81add297f40 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:49.909637 710 leader_election.cc:343] T dbb254115a574bf4976aff0ea56f3bd9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.909757 710 leader_election.cc:304] T dbb254115a574bf4976aff0ea56f3bd9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:49.910174 710 leader_election.cc:343] T 9903e80a26fd469a8744dc855534ba75 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.910266 710 leader_election.cc:304] T 9903e80a26fd469a8744dc855534ba75 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.910368 765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5fc092c2bb2d41ac8c5b4fc07460c8ec" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
W20251025 07:52:49.910640 710 leader_election.cc:343] T ee1c8ea694284d80a30f9c05be4cbf49 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.910702 3137 raft_consensus.cc:2749] T dbb254115a574bf4976aff0ea56f3bd9 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:49.910733 710 leader_election.cc:304] T ee1c8ea694284d80a30f9c05be4cbf49 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.910832 3137 raft_consensus.cc:2749] T 9903e80a26fd469a8744dc855534ba75 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:49.910913 3137 raft_consensus.cc:2749] T ee1c8ea694284d80a30f9c05be4cbf49 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:49.911623 3011 raft_consensus.cc:2749] T fce535a0c0bd45c6bf17f81add297f40 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:49.911967 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6ae1315b28940d49b667e3c80347021" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:49.912106 772 raft_consensus.cc:2468] T a6ae1315b28940d49b667e3c80347021 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:49.873538 2166 ts_tablet_manager.cc:1916] T e47c4f741f354a19b49777bd06e54764 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:49.922880 3011 raft_consensus.cc:493] T e8402fd33ba842e981c6c68787c7c548 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.923158 3011 raft_consensus.cc:515] T e8402fd33ba842e981c6c68787c7c548 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:49.924501 3011 leader_election.cc:290] T e8402fd33ba842e981c6c68787c7c548 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:49.924839 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8402fd33ba842e981c6c68787c7c548" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:49.925827 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8402fd33ba842e981c6c68787c7c548" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:49.926376 983 leader_election.cc:304] T e8402fd33ba842e981c6c68787c7c548 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:49.926965 3011 raft_consensus.cc:2749] T e8402fd33ba842e981c6c68787c7c548 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:49.933589 712 consensus_peers.cc:597] T ed243540f2d746b980c25687fbca7414 P 23eeffd3de254ce18db5febfde971c6b -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:49.934155 710 leader_election.cc:304] T 8917a289bd9b4428a4603429edd94581 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:49.934427 2888 raft_consensus.cc:2804] T 8917a289bd9b4428a4603429edd94581 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:49.934530 2888 raft_consensus.cc:697] T 8917a289bd9b4428a4603429edd94581 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:49.934739 2888 consensus_queue.cc:237] T 8917a289bd9b4428a4603429edd94581 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.938668 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 943ec703e65f4fd3b0c690e08c0bd226 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:49.940304 755 tablet_service.cc:1552] Processing DeleteTablet for tablet c88c46e022ff41a58b02538f2151e260 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:49.940995 755 tablet_service.cc:1552] Processing DeleteTablet for tablet fa998b9ca6724c4cb1eb68a3eecb0b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:49.941596 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 70fe9aa2f58d4abea3b3b71066a29653 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:49.942234 755 tablet_service.cc:1552] Processing DeleteTablet for tablet f4578ce69a0d4322944e5ce7c8a59831 with delete_type TABLET_DATA_DELETED (replaced by tablet 48b73e0c0dce4be686d05f991a21bd19) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:49.942770 755 tablet_service.cc:1552] Processing DeleteTablet for tablet ff3daed6c9af47e3976b82b4b9a41086 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:49.943320 1167 tablet_bootstrap.cc:492] T 8707b93d491e4e4386d148087a2f0870 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:49.943835 1167 ts_tablet_manager.cc:1403] T 8707b93d491e4e4386d148087a2f0870 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.070s user 0.003s sys 0.000s
I20251025 07:52:49.944123 875 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b594d4718344aafbdaec34040695589 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
W20251025 07:52:49.944882 850 consensus_peers.cc:597] T 11bfe0c87ca941a3846462c8a4512d83 P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11bfe0c87ca941a3846462c8a4512d83. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:49.944823 1167 raft_consensus.cc:359] T 8707b93d491e4e4386d148087a2f0870 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.944973 1167 raft_consensus.cc:385] T 8707b93d491e4e4386d148087a2f0870 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.945022 1167 raft_consensus.cc:740] T 8707b93d491e4e4386d148087a2f0870 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.945188 1167 consensus_queue.cc:260] T 8707b93d491e4e4386d148087a2f0870 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.945420 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ce62847e80f48e8807fdbbbdc50803c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:49.945580 772 raft_consensus.cc:3060] T 6ce62847e80f48e8807fdbbbdc50803c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.946313 875 tablet_service.cc:1552] Processing DeleteTablet for tablet fa998b9ca6724c4cb1eb68a3eecb0b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.949127 772 raft_consensus.cc:2468] T 6ce62847e80f48e8807fdbbbdc50803c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:49.950973 771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "243b8828fc344bacab46d7bab6bd2dec" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:49.951128 771 raft_consensus.cc:3060] T 243b8828fc344bacab46d7bab6bd2dec P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:49.952229 1167 ts_tablet_manager.cc:1434] T 8707b93d491e4e4386d148087a2f0870 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251025 07:52:49.952499 1167 tablet_bootstrap.cc:492] T b71bdfd09a27449d98509bb220680147 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:49.953799 771 raft_consensus.cc:2468] T 243b8828fc344bacab46d7bab6bd2dec P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:49.954094 1167 tablet_bootstrap.cc:654] T b71bdfd09a27449d98509bb220680147 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.956724 894 tablet_service.cc:1552] Processing DeleteTablet for tablet d66ad049d5484228a268b0facb00ecf1 with delete_type TABLET_DATA_DELETED (replaced by tablet 2e098c9c4613459295b6043048cba587) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.957176 875 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b28215f61574cf48c447dd01955ac77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.963140 2166 ts_tablet_manager.cc:1929] T e47c4f741f354a19b49777bd06e54764 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:49.963263 2166 log.cc:1199] T e47c4f741f354a19b49777bd06e54764 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/e47c4f741f354a19b49777bd06e54764
I20251025 07:52:49.963564 2166 ts_tablet_manager.cc:1950] T e47c4f741f354a19b49777bd06e54764 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:49.963515 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "792f32cbd66449ca98813c669b2a261d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:49.964035 772 raft_consensus.cc:2468] T 792f32cbd66449ca98813c669b2a261d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:49.964371 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 204) took 8058 ms. Trace:
I20251025 07:52:49.964459 2166 rpcz_store.cc:276] 1025 07:52:41.905606 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.906890 (+ 1284us) service_pool.cc:225] Handling call
1025 07:52:49.871268 (+7964378us) ts_tablet_manager.cc:1151] Deleting tablet e47c4f741f354a19b49777bd06e54764
1025 07:52:49.871277 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.917569 (+ 46292us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.963016 (+ 45447us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:49.964352 (+ 1336us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7964235}
I20251025 07:52:49.964960 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet e47c4f741f354a19b49777bd06e54764 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]) successfully deleted
I20251025 07:52:49.965971 2166 tablet_replica.cc:333] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:49.966212 2166 raft_consensus.cc:2243] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:49.966447 2166 pending_rounds.cc:70] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b: Trying to abort 1 pending ops.
I20251025 07:52:49.966499 2166 pending_rounds.cc:77] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214607440641433600 op_type: NO_OP noop_request { }
I20251025 07:52:49.966585 2166 raft_consensus.cc:2889] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 07:52:49.966661 2166 raft_consensus.cc:2272] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:49.971364 1167 tablet_bootstrap.cc:492] T b71bdfd09a27449d98509bb220680147 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:49.971491 1167 ts_tablet_manager.cc:1403] T b71bdfd09a27449d98509bb220680147 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20251025 07:52:49.972090 1167 raft_consensus.cc:359] T b71bdfd09a27449d98509bb220680147 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.972231 1167 raft_consensus.cc:385] T b71bdfd09a27449d98509bb220680147 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.972275 1167 raft_consensus.cc:740] T b71bdfd09a27449d98509bb220680147 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.972486 1167 consensus_queue.cc:260] T b71bdfd09a27449d98509bb220680147 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.973069 2988 raft_consensus.cc:493] T 48b73e0c0dce4be686d05f991a21bd19 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.973765 1167 ts_tablet_manager.cc:1434] T b71bdfd09a27449d98509bb220680147 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:49.973677 2988 raft_consensus.cc:515] T 48b73e0c0dce4be686d05f991a21bd19 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:49.975021 1167 tablet_bootstrap.cc:492] T 22f7db660616457b83b24e947a0b0bae P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:49.975368 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "48b73e0c0dce4be686d05f991a21bd19" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:49.975816 850 leader_election.cc:343] T 48b73e0c0dce4be686d05f991a21bd19 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.976826 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 284) took 5371 ms. Trace:
I20251025 07:52:49.976950 2146 rpcz_store.cc:276] 1025 07:52:44.605045 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.605422 (+ 377us) service_pool.cc:225] Handling call
1025 07:52:49.904606 (+5299184us) ts_tablet_manager.cc:1151] Deleting tablet f3001314b56345f5a70314777cae0125
1025 07:52:49.904614 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.976807 (+ 72193us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5299051}
I20251025 07:52:49.977120 1167 tablet_bootstrap.cc:654] T 22f7db660616457b83b24e947a0b0bae P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.977690 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 283) took 5372 ms. Trace:
I20251025 07:52:49.978350 2166 ts_tablet_manager.cc:1916] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:49.979930 2888 raft_consensus.cc:493] T 294d7d20c2f545b8813e632aa058a60c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:49.980055 2888 raft_consensus.cc:515] T 294d7d20c2f545b8813e632aa058a60c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.980604 2888 leader_election.cc:290] T 294d7d20c2f545b8813e632aa058a60c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:49.981266 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "294d7d20c2f545b8813e632aa058a60c" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:49.981770 712 leader_election.cc:343] T 294d7d20c2f545b8813e632aa058a60c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:49.982513 446 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:42387 (request call id 216) took 5278 ms (client timeout 1000 ms). Trace:
W20251025 07:52:49.982682 446 rpcz_store.cc:269] 1025 07:52:44.703689 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.745352 (+ 41663us) service_pool.cc:225] Handling call
1025 07:52:45.618721 (+873369us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.618730 (+ 9us) write_op.cc:454] Released schema lock
1025 07:52:49.982448 (+4363718us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:45.601193 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:45.601333 (+ 140us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:45.601340 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:45.601343 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:45.601464 (+ 121us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:45.601480 (+ 16us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:45.601485 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:45.601603 (+ 118us) tablet.cc:699] Row locks acquired
1025 07:52:45.601617 (+ 14us) write_op.cc:260] PREPARE: finished
1025 07:52:45.601652 (+ 35us) write_op.cc:270] Start()
1025 07:52:45.601688 (+ 36us) write_op.cc:276] Timestamp: P: 1761378765599128 usec, L: 0
1025 07:52:45.601692 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:45.601881 (+ 189us) log.cc:844] Serialized 901 byte log entry
1025 07:52:45.607580 (+ 5699us) op_driver.cc:464] REPLICATION: finished
1025 07:52:45.607710 (+ 130us) write_op.cc:301] APPLY: starting
1025 07:52:45.607728 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:45.607788 (+ 60us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:45.607819 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:45.607889 (+ 70us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:45.607923 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1025 07:52:45.607930 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:45.608003 (+ 73us) log.cc:844] Serialized 29 byte log entry
1025 07:52:45.608065 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.608086 (+ 21us) write_op.cc:454] Released schema lock
1025 07:52:45.608092 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":13095,"reported_tablets":242,"child_traces":[["op",{"apply.queue_time_us":76,"apply.run_cpu_time_us":435,"apply.run_wall_time_us":434,"num_ops":2,"prepare.queue_time_us":65,"prepare.run_cpu_time_us":799,"prepare.run_wall_time_us":800,"raft.queue_time_us":1218,"raft.run_cpu_time_us":486,"raft.run_wall_time_us":487,"replication_time_us":5858}]]}
I20251025 07:52:49.986100 1169 tablet_bootstrap.cc:654] T 645263a61b654f04985c40d04506c087 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:49.995152 2146 rpcz_store.cc:276] 1025 07:52:44.604927 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.605650 (+ 723us) service_pool.cc:225] Handling call
1025 07:52:49.977599 (+5371949us) ts_tablet_manager.cc:1151] Deleting tablet 71432a6848944a0da06ef9b81f160dc4
1025 07:52:49.977611 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.977677 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5371819}
I20251025 07:52:49.995543 2146 tablet_replica.cc:333] T 570201e7bf05406a97c81dde071d894b P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:49.995785 2146 raft_consensus.cc:2243] T 570201e7bf05406a97c81dde071d894b P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:49.996156 2146 raft_consensus.cc:2272] T 570201e7bf05406a97c81dde071d894b P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:49.996269 1167 tablet_bootstrap.cc:492] T 22f7db660616457b83b24e947a0b0bae P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:49.996426 1167 ts_tablet_manager.cc:1403] T 22f7db660616457b83b24e947a0b0bae P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20251025 07:52:49.996795 2146 ts_tablet_manager.cc:1916] T 570201e7bf05406a97c81dde071d894b P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:49.997088 1167 raft_consensus.cc:359] T 22f7db660616457b83b24e947a0b0bae P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.997259 1167 raft_consensus.cc:385] T 22f7db660616457b83b24e947a0b0bae P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:49.997324 1167 raft_consensus.cc:740] T 22f7db660616457b83b24e947a0b0bae P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:49.997552 1167 consensus_queue.cc:260] T 22f7db660616457b83b24e947a0b0bae P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:49.998239 1167 ts_tablet_manager.cc:1434] T 22f7db660616457b83b24e947a0b0bae P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:49.998664 1167 tablet_bootstrap.cc:492] T 3aeaa0e9dc594e21a9ddd923e314017d P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:50.000681 1167 tablet_bootstrap.cc:654] T 3aeaa0e9dc594e21a9ddd923e314017d P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:50.003752 2169 ts_tablet_manager.cc:1929] T 716fb61312e94a4d9eb8ab8336ca140c P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:50.003878 2169 log.cc:1199] T 716fb61312e94a4d9eb8ab8336ca140c P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/716fb61312e94a4d9eb8ab8336ca140c
I20251025 07:52:50.004146 2169 ts_tablet_manager.cc:1950] T 716fb61312e94a4d9eb8ab8336ca140c P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:50.004813 2990 raft_consensus.cc:493] T 2e098c9c4613459295b6043048cba587 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:50.005002 2990 raft_consensus.cc:515] T 2e098c9c4613459295b6043048cba587 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:50.005973 2990 leader_election.cc:290] T 2e098c9c4613459295b6043048cba587 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:49.979379 2988 leader_election.cc:290] T 48b73e0c0dce4be686d05f991a21bd19 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:49.986496 875 tablet_service.cc:1552] Processing DeleteTablet for tablet 3abfd52327a04ba88625f391a8658fbb with delete_type TABLET_DATA_DELETED (replaced by tablet 04d27d54fdb7483b869408af65ff85c9) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:50.006258 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e098c9c4613459295b6043048cba587" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:49.986579 894 tablet_service.cc:1552] Processing DeleteTablet for tablet af743ee344e543a0aa6da3cd7a30e467 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.986687 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 34315f64b89148efbf1f20a43d5c5efb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.986828 883 tablet_service.cc:1552] Processing DeleteTablet for tablet a4e4b99be0434baaa75d988281797846 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.987291 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 92aa605fabbc4257a6a204c087e7cc7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.987727 893 tablet_service.cc:1552] Processing DeleteTablet for tablet ff3daed6c9af47e3976b82b4b9a41086 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:49.988504 905 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aea8fcf0154a4bd4a05b19e27ab78b79" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:50.006901 905 raft_consensus.cc:2468] T aea8fcf0154a4bd4a05b19e27ab78b79 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:49.988757 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "294d7d20c2f545b8813e632aa058a60c" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:50.007547 850 leader_election.cc:343] T 2e098c9c4613459295b6043048cba587 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:49.989216 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66134661be834f7b811d7927ab629812" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:49.985265 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet f3001314b56345f5a70314777cae0125 because the tablet was not found. No further retry: Not found: Tablet not found: f3001314b56345f5a70314777cae0125
W20251025 07:52:50.012200 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 71432a6848944a0da06ef9b81f160dc4 because the tablet was not found. No further retry: Not found: Tablet not found: 71432a6848944a0da06ef9b81f160dc4
I20251025 07:52:50.005183 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 716fb61312e94a4d9eb8ab8336ca140c (table default.Table_ba46ca0cf5d941e0879dead1c3544190 [id=3cf1d7f589ff457b9d47c88a1ccc3b2b]) successfully deleted
I20251025 07:52:50.003480 2166 ts_tablet_manager.cc:1929] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:50.012537 2166 log.cc:1199] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/72dc084f8efc45478771c247e07a16f4
I20251025 07:52:50.013227 2166 ts_tablet_manager.cc:1950] T 72dc084f8efc45478771c247e07a16f4 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:50.013841 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet b947ecc352e54f4caa73c6ca97372256 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.014185 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet f656945454fe4921ab819c772a089eb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.014981 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet d77704b4fe6c4456a2ba40ffec0c60fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.014955 3179 tablet.cc:1722] T 8e6d76b1a97945cfb6c3055e1b622f8a P 23eeffd3de254ce18db5febfde971c6b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251025 07:52:50.015278 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet e4da89f863cc43f3bf6df908bb9dc6bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.015659 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet e47c4f741f354a19b49777bd06e54764 with delete_type TABLET_DATA_DELETED (replaced by tablet 755f0d6c87044e20a8020f8410940123) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.015610 3175 tablet.cc:1722] T 8e6d76b1a97945cfb6c3055e1b622f8a P db42be7ad3c741a0aa427f27e245993b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251025 07:52:50.016010 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f7c15529be54a11961f385a953b0b03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.016196 3180 tablet.cc:1722] T 8e6d76b1a97945cfb6c3055e1b622f8a P 3aea2869ff814cc7b142012faa274382: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251025 07:52:50.016412 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 95efd553493e4966bc48a01cac6e1e10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.016896 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b278ce7768349e49dcd1131e404ab34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:44 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.017306 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet c11f674385184ae6b235f1db8bf1fb91 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.017689 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e098c9c4613459295b6043048cba587 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.020808 755 tablet_service.cc:1552] Processing DeleteTablet for tablet e47c4f741f354a19b49777bd06e54764 with delete_type TABLET_DATA_DELETED (replaced by tablet 755f0d6c87044e20a8020f8410940123) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:50.023217 755 tablet_service.cc:1552] Processing DeleteTablet for tablet f656945454fe4921ab819c772a089eb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:50.024981 771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "48b73e0c0dce4be686d05f991a21bd19" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:50.025161 771 raft_consensus.cc:2468] T 48b73e0c0dce4be686d05f991a21bd19 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
W20251025 07:52:50.027076 847 leader_election.cc:343] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.027258 847 leader_election.cc:304] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.027804 847 leader_election.cc:304] T a6ae1315b28940d49b667e3c80347021 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.028123 2686 raft_consensus.cc:2804] T a6ae1315b28940d49b667e3c80347021 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:50.028192 2686 raft_consensus.cc:493] T a6ae1315b28940d49b667e3c80347021 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:50.028250 2686 raft_consensus.cc:3060] T a6ae1315b28940d49b667e3c80347021 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.027892 2990 raft_consensus.cc:2749] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:50.030944 2686 raft_consensus.cc:515] T a6ae1315b28940d49b667e3c80347021 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.031453 2686 leader_election.cc:290] T a6ae1315b28940d49b667e3c80347021 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:50.031826 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6ae1315b28940d49b667e3c80347021" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:50.032231 850 leader_election.cc:343] T a6ae1315b28940d49b667e3c80347021 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.032224 847 leader_election.cc:304] T 6ce62847e80f48e8807fdbbbdc50803c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.032579 2686 raft_consensus.cc:2804] T 6ce62847e80f48e8807fdbbbdc50803c P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:50.032718 2686 raft_consensus.cc:697] T 6ce62847e80f48e8807fdbbbdc50803c P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:50.032996 2686 consensus_queue.cc:237] T 6ce62847e80f48e8807fdbbbdc50803c P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.034288 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 205) took 8128 ms. Trace:
I20251025 07:52:50.034386 2166 rpcz_store.cc:276] 1025 07:52:41.905709 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.907058 (+ 1349us) service_pool.cc:225] Handling call
1025 07:52:49.965925 (+8058867us) ts_tablet_manager.cc:1151] Deleting tablet 72dc084f8efc45478771c247e07a16f4
1025 07:52:49.965935 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:49.988032 (+ 22097us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.003386 (+ 15354us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.034266 (+ 30880us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8058749}
I20251025 07:52:50.034704 2166 tablet_replica.cc:333] T 716fb61312e94a4d9eb8ab8336ca140c P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:50.034902 2166 raft_consensus.cc:2243] T 716fb61312e94a4d9eb8ab8336ca140c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:50.034898 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 72dc084f8efc45478771c247e07a16f4 (table default.Table_5a8143ecfd74418f928ceb9ae9cd50ba [id=23d0aa66d6f545d4b579cfd63b7e7143]) successfully deleted
I20251025 07:52:50.035137 2166 raft_consensus.cc:2272] T 716fb61312e94a4d9eb8ab8336ca140c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:50.035705 2166 ts_tablet_manager.cc:1916] T 716fb61312e94a4d9eb8ab8336ca140c P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:50.036417 2888 raft_consensus.cc:493] T d40519ed9ee746348b17be2786d60e95 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:50.036535 2888 raft_consensus.cc:515] T d40519ed9ee746348b17be2786d60e95 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.037091 2888 leader_election.cc:290] T d40519ed9ee746348b17be2786d60e95 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:50.037798 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d40519ed9ee746348b17be2786d60e95" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:50.038318 712 leader_election.cc:343] T d40519ed9ee746348b17be2786d60e95 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.039266 1045 raft_consensus.cc:3060] T 1068f566477e4c6db7dde566fff4c2fb P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.043661 1167 tablet_bootstrap.cc:492] T 3aeaa0e9dc594e21a9ddd923e314017d P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:50.043844 1167 ts_tablet_manager.cc:1403] T 3aeaa0e9dc594e21a9ddd923e314017d P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.000s
I20251025 07:52:50.044598 1167 raft_consensus.cc:359] T 3aeaa0e9dc594e21a9ddd923e314017d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.044823 1167 raft_consensus.cc:385] T 3aeaa0e9dc594e21a9ddd923e314017d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.044907 1167 raft_consensus.cc:740] T 3aeaa0e9dc594e21a9ddd923e314017d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.045167 1167 consensus_queue.cc:260] T 3aeaa0e9dc594e21a9ddd923e314017d P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.045881 1167 ts_tablet_manager.cc:1434] T 3aeaa0e9dc594e21a9ddd923e314017d P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:50.047600 755 tablet_service.cc:1552] Processing DeleteTablet for tablet e54fe7cbcffa40c4b04144b47bee2b48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:50.050160 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 71432a6848944a0da06ef9b81f160dc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:50.051379 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 8956ad28b35d45f28c683ac055bba441 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:50.053957 2166 ts_tablet_manager.cc:1929] T 716fb61312e94a4d9eb8ab8336ca140c P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:50.054057 2166 log.cc:1199] T 716fb61312e94a4d9eb8ab8336ca140c P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/716fb61312e94a4d9eb8ab8336ca140c
I20251025 07:52:50.054394 2166 ts_tablet_manager.cc:1950] T 716fb61312e94a4d9eb8ab8336ca140c P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:50.054706 755 tablet_service.cc:1552] Processing DeleteTablet for tablet ffe7c8e1d6c44872b7b65e512764a56a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:50.055851 755 tablet_service.cc:1552] Processing DeleteTablet for tablet db99a1e68daf4e948a3d66ad276cc5f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:50.056115 2169 tablet_replica.cc:333] T dda0b26e4f1a44928bc4b84f333901c0 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:50.056320 2169 raft_consensus.cc:2243] T dda0b26e4f1a44928bc4b84f333901c0 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:50.056465 2169 raft_consensus.cc:2272] T dda0b26e4f1a44928bc4b84f333901c0 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:50.057029 2169 ts_tablet_manager.cc:1916] T dda0b26e4f1a44928bc4b84f333901c0 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:50.057057 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e6544202d4f489d82f22d6073ef240d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:50.058629 755 tablet_service.cc:1552] Processing DeleteTablet for tablet d5e5e5bbcf9f43118eea502b69880af5 with delete_type TABLET_DATA_DELETED (replaced by tablet f4578ce69a0d4322944e5ce7c8a59831) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:50.059510 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet cc407c7eb12e4eadb41fb21385611c94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.059942 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 53198283718d4b45b30df65b5bde96a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:50.059955 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 6897405bbc1c47a4ab0aa3af80e60e3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.060864 1169 tablet_bootstrap.cc:492] T 645263a61b654f04985c40d04506c087 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:50.061015 1169 ts_tablet_manager.cc:1403] T 645263a61b654f04985c40d04506c087 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.292s user 0.007s sys 0.007s
I20251025 07:52:50.061609 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c4970c8a491460e8b4b1c29fde83e4c with delete_type TABLET_DATA_DELETED (replaced by tablet 783487b60c144f719ed2eb3c67b5ccab) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.061992 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 32a143a715cb485cb4501f4e28c68252 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c389d798cb94d8a972683c57b1f9f1f) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.063349 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 206) took 8157 ms. Trace:
I20251025 07:52:50.063444 2166 rpcz_store.cc:276] 1025 07:52:41.905837 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.907198 (+ 1361us) service_pool.cc:225] Handling call
1025 07:52:50.034661 (+8127463us) ts_tablet_manager.cc:1151] Deleting tablet 716fb61312e94a4d9eb8ab8336ca140c
1025 07:52:50.034672 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.050481 (+ 15809us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.053870 (+ 3389us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.063329 (+ 9459us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8182,"tablet-delete.queue_time_us":8127342}
I20251025 07:52:50.063882 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 716fb61312e94a4d9eb8ab8336ca140c (table default.Table_ba46ca0cf5d941e0879dead1c3544190 [id=3cf1d7f589ff457b9d47c88a1ccc3b2b]) successfully deleted
I20251025 07:52:50.064311 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 207) took 8158 ms. Trace:
I20251025 07:52:50.064232 1169 raft_consensus.cc:359] T 645263a61b654f04985c40d04506c087 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:50.064429 1169 raft_consensus.cc:385] T 645263a61b654f04985c40d04506c087 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.064509 1169 raft_consensus.cc:740] T 645263a61b654f04985c40d04506c087 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.064467 2166 rpcz_store.cc:276] 1025 07:52:41.905953 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.907343 (+ 1390us) service_pool.cc:225] Handling call
1025 07:52:50.064197 (+8156854us) ts_tablet_manager.cc:1151] Deleting tablet bef0ef23d90d480d9c99aef21d7271fd
1025 07:52:50.064208 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.064295 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8156732}
W20251025 07:52:50.065037 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet bef0ef23d90d480d9c99aef21d7271fd because the tablet was not found. No further retry: Not found: Tablet not found: bef0ef23d90d480d9c99aef21d7271fd
I20251025 07:52:50.065384 2166 tablet_replica.cc:333] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:50.065624 2166 raft_consensus.cc:2243] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:50.065845 2166 raft_consensus.cc:2272] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:50.066494 2166 ts_tablet_manager.cc:1916] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:50.067130 847 leader_election.cc:304] T 243b8828fc344bacab46d7bab6bd2dec P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.067365 2686 consensus_queue.cc:1048] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251025 07:52:50.067682 450 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:35777 (request call id 198) took 3078 ms (client timeout 1000 ms). Trace:
I20251025 07:52:50.067873 2686 raft_consensus.cc:2804] T 243b8828fc344bacab46d7bab6bd2dec P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
W20251025 07:52:50.067801 450 rpcz_store.cc:269] 1025 07:52:46.989101 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:47.872419 (+883318us) service_pool.cc:225] Handling call
1025 07:52:50.067613 (+2195194us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32348,"reported_tablets":246}
I20251025 07:52:50.067987 2686 raft_consensus.cc:697] T 243b8828fc344bacab46d7bab6bd2dec P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:50.069650 2686 consensus_queue.cc:237] T 243b8828fc344bacab46d7bab6bd2dec P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:50.073503 447 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:35777 (request call id 197) took 4215 ms (client timeout 1000 ms). Trace:
W20251025 07:52:50.073670 447 rpcz_store.cc:269] 1025 07:52:45.857896 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.918546 (+ 60650us) service_pool.cc:225] Handling call
1025 07:52:50.073433 (+4154887us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:47.126403 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:47.126574 (+ 171us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:47.126582 (+ 8us) write_op.cc:435] Acquired schema lock
1025 07:52:47.126584 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:47.126696 (+ 112us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:47.126710 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:47.126715 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:47.126793 (+ 78us) tablet.cc:699] Row locks acquired
1025 07:52:47.126795 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:47.126838 (+ 43us) write_op.cc:270] Start()
1025 07:52:47.126874 (+ 36us) write_op.cc:276] Timestamp: P: 1761378767126815 usec, L: 0
1025 07:52:47.126877 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:47.127116 (+ 239us) log.cc:844] Serialized 527 byte log entry
1025 07:52:47.131229 (+ 4113us) op_driver.cc:464] REPLICATION: finished
1025 07:52:47.131332 (+ 103us) write_op.cc:301] APPLY: starting
1025 07:52:47.131348 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:47.131390 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:47.131427 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:47.131510 (+ 83us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:47.131560 (+ 50us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:47.131572 (+ 12us) write_op.cc:312] APPLY: finished
1025 07:52:47.131666 (+ 94us) log.cc:844] Serialized 23 byte log entry
1025 07:52:47.131742 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:47.131771 (+ 29us) write_op.cc:454] Released schema lock
1025 07:52:47.131780 (+ 9us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:47.131870 (+ 90us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:47.131876 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":123289,"reported_tablets":247,"child_traces":[["op",{"apply.queue_time_us":56,"apply.run_cpu_time_us":516,"apply.run_wall_time_us":516,"num_ops":1,"prepare.queue_time_us":49576,"prepare.run_cpu_time_us":857,"prepare.run_wall_time_us":858,"raft.queue_time_us":1133,"raft.run_cpu_time_us":545,"raft.run_wall_time_us":545,"replication_time_us":4317}]]}
I20251025 07:52:50.080299 3126 raft_consensus.cc:493] T 22f7db660616457b83b24e947a0b0bae P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:50.080502 3126 raft_consensus.cc:515] T 22f7db660616457b83b24e947a0b0bae P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.080787 2166 ts_tablet_manager.cc:1929] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:50.080940 2166 log.cc:1199] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/4f0c4f0e8f014e59be9e46d2afe0be9d
I20251025 07:52:50.081461 3126 leader_election.cc:290] T 22f7db660616457b83b24e947a0b0bae P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:50.081904 2166 ts_tablet_manager.cc:1950] T 4f0c4f0e8f014e59be9e46d2afe0be9d P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:50.082818 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 208) took 8176 ms. Trace:
I20251025 07:52:50.082895 2166 rpcz_store.cc:276] 1025 07:52:41.906052 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.907485 (+ 1433us) service_pool.cc:225] Handling call
1025 07:52:50.065337 (+8157852us) ts_tablet_manager.cc:1151] Deleting tablet 4f0c4f0e8f014e59be9e46d2afe0be9d
1025 07:52:50.065346 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.071094 (+ 5748us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.080645 (+ 9551us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.082801 (+ 2156us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8157732}
I20251025 07:52:50.083328 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 4f0c4f0e8f014e59be9e46d2afe0be9d (table default.Table_c902adc24467448f9b5d4f585cf614f5 [id=56e169f79a37465d82dd1d1c3c285740]) successfully deleted
I20251025 07:52:50.083662 2166 tablet_replica.cc:333] T f661d5909657455581dcb5e6eadd3c0b P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:50.083885 2166 raft_consensus.cc:2243] T f661d5909657455581dcb5e6eadd3c0b P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:50.084069 2166 raft_consensus.cc:2272] T f661d5909657455581dcb5e6eadd3c0b P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:50.084609 2166 ts_tablet_manager.cc:1916] T f661d5909657455581dcb5e6eadd3c0b P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:50.089941 847 leader_election.cc:304] T 792f32cbd66449ca98813c669b2a261d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:50.090469 712 consensus_peers.cc:597] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0551e5d694894e8fbeb66b5e0c1cd75b. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:50.090567 2990 raft_consensus.cc:2804] T 792f32cbd66449ca98813c669b2a261d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:50.090683 2990 raft_consensus.cc:493] T 792f32cbd66449ca98813c669b2a261d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:50.090814 2990 raft_consensus.cc:3060] T 792f32cbd66449ca98813c669b2a261d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.091419 1166 tablet_bootstrap.cc:492] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:50.091554 1166 ts_tablet_manager.cc:1403] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.368s user 0.002s sys 0.014s
I20251025 07:52:50.092199 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 783487b60c144f719ed2eb3c67b5ccab (table default.Table_09095c0829ed4d1cb8b763fc695e7fa8 [id=e372ca4384644779a6b37f389f549dec]). Expected version 1 got 0
I20251025 07:52:50.093657 2990 raft_consensus.cc:515] T 792f32cbd66449ca98813c669b2a261d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.094220 3126 raft_consensus.cc:493] T 8707b93d491e4e4386d148087a2f0870 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:50.094386 3126 raft_consensus.cc:515] T 8707b93d491e4e4386d148087a2f0870 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.094466 2990 leader_election.cc:290] T 792f32cbd66449ca98813c669b2a261d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:50.095428 3126 leader_election.cc:290] T 8707b93d491e4e4386d148087a2f0870 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:50.095836 455 catalog_manager.cc:5649] T c187a40f8ff14c6a8c184fbeb030ad13 P 23eeffd3de254ce18db5febfde971c6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } }
I20251025 07:52:50.096351 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f]). Expected version 1 got 0
I20251025 07:52:50.096771 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7]). Expected version 1 got 0
I20251025 07:52:50.097280 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 872cb7512d104827b1f1f186d33a4a44 (table default.Table_d5c65dce8b1b4214ba519d964012bfc0 [id=d4be274b36a845a99a4b96b8846248fd]). Expected version 1 got 0
I20251025 07:52:50.100013 1166 raft_consensus.cc:359] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.100205 1166 raft_consensus.cc:385] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.100252 1166 raft_consensus.cc:740] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.100461 1166 consensus_queue.cc:260] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.101071 1166 ts_tablet_manager.cc:1434] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251025 07:52:50.098150 455 catalog_manager.cc:5649] T 1068f566477e4c6db7dde566fff4c2fb P 23eeffd3de254ce18db5febfde971c6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } }
I20251025 07:52:50.101379 1166 tablet_bootstrap.cc:492] T 22f7db660616457b83b24e947a0b0bae P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:50.101959 455 catalog_manager.cc:5649] T c7691c7bdd89437c8db060254c0db25d P 23eeffd3de254ce18db5febfde971c6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } }
I20251025 07:52:50.103083 1166 tablet_bootstrap.cc:654] T 22f7db660616457b83b24e947a0b0bae P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:50.103250 2166 ts_tablet_manager.cc:1929] T f661d5909657455581dcb5e6eadd3c0b P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:50.103417 2166 log.cc:1199] T f661d5909657455581dcb5e6eadd3c0b P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/f661d5909657455581dcb5e6eadd3c0b
I20251025 07:52:50.103575 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 6fe43ccecf7047319090ebd035fe24a5 (table default.Table_68b39cfaff194c3d8dbe9eaee7d8b44c [id=c95fc6d40893462eacee1276fb9cacfe]). Expected version 1 got 0
I20251025 07:52:50.103852 2166 ts_tablet_manager.cc:1950] T f661d5909657455581dcb5e6eadd3c0b P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:50.104260 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]). Expected version 1 got 0
I20251025 07:52:50.065227 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 1912e59548874e0e8e989050ec0064c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.081596 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22f7db660616457b83b24e947a0b0bae" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:50.088546 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet 0540d6a10cf04fa89756623e9aaaeb68 with delete_type TABLET_DATA_DELETED (replaced by tablet 55a4a524676b4b9aa9464e934b662f30) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.088617 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet c59f6e6eb9594a8e83f109a09b034045 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.088869 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet b02f5c0ab0b24288bb50a3ba0e6bd78a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.088950 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 29dbaf9149134f548a974a5779f7da67 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:44 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.089208 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 6dcab53a020847a1aa27930265ba0205 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.095273 1043 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8707b93d491e4e4386d148087a2f0870" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:50.095475 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "792f32cbd66449ca98813c669b2a261d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:50.106846 712 leader_election.cc:343] T 22f7db660616457b83b24e947a0b0bae P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.107237 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]). Expected version 1 got 0
W20251025 07:52:50.107482 712 leader_election.cc:343] T 8707b93d491e4e4386d148087a2f0870 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.107498 455 catalog_manager.cc:5649] T 682c1e22d7764dce862d970b33b88cb3 P 23eeffd3de254ce18db5febfde971c6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } }
I20251025 07:52:50.107932 1169 consensus_queue.cc:260] T 645263a61b654f04985c40d04506c087 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:50.109133 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_fa83d0528b4b481989bfc0ca35c6de44 [id=bf77b5ff6f1048dd996d948437f9ff5e]). Expected version 3 got 0
I20251025 07:52:50.109570 2988 raft_consensus.cc:493] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:50.109745 2988 raft_consensus.cc:515] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.110505 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 209) took 8202 ms. Trace:
I20251025 07:52:50.110594 2166 rpcz_store.cc:276] 1025 07:52:41.908226 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:41.908936 (+ 710us) service_pool.cc:225] Handling call
1025 07:52:50.083617 (+8174681us) ts_tablet_manager.cc:1151] Deleting tablet f661d5909657455581dcb5e6eadd3c0b
1025 07:52:50.083626 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.093440 (+ 9814us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.103121 (+ 9681us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.110485 (+ 7364us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8174524}
I20251025 07:52:50.110467 2988 leader_election.cc:290] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:50.111352 2166 tablet_replica.cc:333] T 4c155aa7fa5b4d68bbab1918b8432372 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:50.111542 2166 raft_consensus.cc:2243] T 4c155aa7fa5b4d68bbab1918b8432372 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:50.111723 2166 raft_consensus.cc:2272] T 4c155aa7fa5b4d68bbab1918b8432372 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:50.111042 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet f661d5909657455581dcb5e6eadd3c0b (table default.Table_b8a12e9a7a974006904fcef12bb8225a [id=9bfd1a38f56e4b70850139909aff091f]) successfully deleted
I20251025 07:52:50.112498 2166 ts_tablet_manager.cc:1916] T 4c155aa7fa5b4d68bbab1918b8432372 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:50.112664 455 catalog_manager.cc:5649] T 6a33cfb2f2de48d4a5e9e39730d4900e P 23eeffd3de254ce18db5febfde971c6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } }
I20251025 07:52:50.114748 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e]). Expected version 1 got 0
I20251025 07:52:50.115177 847 leader_election.cc:304] T 48b73e0c0dce4be686d05f991a21bd19 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.116165 2991 raft_consensus.cc:2804] T 48b73e0c0dce4be686d05f991a21bd19 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:50.116379 2991 raft_consensus.cc:493] T 48b73e0c0dce4be686d05f991a21bd19 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:50.116533 2991 raft_consensus.cc:3060] T 48b73e0c0dce4be686d05f991a21bd19 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.115211 455 catalog_manager.cc:5649] T 22fc90c704f64d2f9f4b8c54dee45576 P 23eeffd3de254ce18db5febfde971c6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 23eeffd3de254ce18db5febfde971c6b (127.0.104.1). New cstate: current_term: 1 leader_uuid: "23eeffd3de254ce18db5febfde971c6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } }
I20251025 07:52:50.116976 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]). Expected version 3 got 0
I20251025 07:52:50.126447 455 catalog_manager.cc:5649] T c501ef5d70e742598c1a902efe47feb1 P 23eeffd3de254ce18db5febfde971c6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } }
I20251025 07:52:50.127126 2166 ts_tablet_manager.cc:1929] T 4c155aa7fa5b4d68bbab1918b8432372 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:50.127247 2166 log.cc:1199] T 4c155aa7fa5b4d68bbab1918b8432372 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/4c155aa7fa5b4d68bbab1918b8432372
I20251025 07:52:50.127270 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet ee1c8ea694284d80a30f9c05be4cbf49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.127457 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_d11bba62f65740319701b4337a5c255a [id=ad812862698d421184ebf501ce79b7be]). Expected version 3 got 0
I20251025 07:52:50.127574 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 1b5ac7e845a24b2eb5b2429d47361232 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.127606 2166 ts_tablet_manager.cc:1950] T 4c155aa7fa5b4d68bbab1918b8432372 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:50.127746 1169 ts_tablet_manager.cc:1434] T 645263a61b654f04985c40d04506c087 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.067s user 0.001s sys 0.001s
I20251025 07:52:50.127766 455 catalog_manager.cc:5649] T e8402fd33ba842e981c6c68787c7c548 P 23eeffd3de254ce18db5febfde971c6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } }
I20251025 07:52:50.128137 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]). Expected version 1 got 0
I20251025 07:52:50.128368 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 22f7db660616457b83b24e947a0b0bae (table default.Table_4a86191163f2408f9bcc795cd8d1bbf6 [id=bb316a7e000748ca902a9bfb494802d7]). Expected version 1 got 0
I20251025 07:52:50.128400 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 213) took 6534 ms. Trace:
I20251025 07:52:50.128469 2166 rpcz_store.cc:276] 1025 07:52:43.593457 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:43.593523 (+ 66us) service_pool.cc:225] Handling call
1025 07:52:50.111305 (+6517782us) ts_tablet_manager.cc:1151] Deleting tablet 4c155aa7fa5b4d68bbab1918b8432372
1025 07:52:50.111316 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.115803 (+ 4487us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.127017 (+ 11214us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.128378 (+ 1361us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6517579}
I20251025 07:52:50.128611 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 5fc092c2bb2d41ac8c5b4fc07460c8ec (table default.Table_cd20e202397645f8821dcde13b5c0ce3 [id=630c274339664b9db18ab4ef7c29c16f]). Expected version 2 got 0
I20251025 07:52:50.128804 1169 tablet_bootstrap.cc:492] T 29dbaf9149134f548a974a5779f7da67 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:50.128803 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 3760348dbe284c41a01a4fe479a2103f (table default.Table_088aad2a46b041bfa29ed8e6929760b1 [id=834669c54dae449790818684fd8659bc]). Expected version 2 got 0
I20251025 07:52:50.129416 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 830fdbc6812e416690d0db42a73af503 (table default.Table_92824b3f2a774564b3aea04aaa25d204 [id=fb21a45b86ab446e8c3dae4512a9598f]). Expected version 1 got 0
I20251025 07:52:50.129729 455 catalog_manager.cc:5649] T ed243540f2d746b980c25687fbca7414 P 23eeffd3de254ce18db5febfde971c6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 23eeffd3de254ce18db5febfde971c6b (127.0.104.1). New cstate: current_term: 1 leader_uuid: "23eeffd3de254ce18db5febfde971c6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: UNKNOWN } } }
I20251025 07:52:50.130488 1169 tablet_bootstrap.cc:654] T 29dbaf9149134f548a974a5779f7da67 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:50.131547 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 9f039d9107ae493486d799ab6a1d731d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:44 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.131901 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 50ae9bca6bed4f749d54191ac1543410 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.133320 2166 tablet_replica.cc:333] T fce535a0c0bd45c6bf17f81add297f40 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:50.133503 2166 raft_consensus.cc:2243] T fce535a0c0bd45c6bf17f81add297f40 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:50.133643 2166 raft_consensus.cc:2272] T fce535a0c0bd45c6bf17f81add297f40 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:50.134104 2166 ts_tablet_manager.cc:1916] T fce535a0c0bd45c6bf17f81add297f40 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:50.136087 1167 tablet_bootstrap.cc:492] T f7011f1dcad74fe4ba0356c618890e06 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:50.130100 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c]). Expected version 1 got 0
I20251025 07:52:50.137315 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_67b99749f3bf43228b40d8b9807e69b4 [id=fc8180c09d7b4e74b2e161f7acb8fe0e]). Expected version 2 got 0
I20251025 07:52:50.137981 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525]). Expected version 1 got 0
I20251025 07:52:50.138250 455 catalog_manager.cc:5649] T 9a8e2658a81e436298fe2190726080d4 P 23eeffd3de254ce18db5febfde971c6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } }
I20251025 07:52:50.138561 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3]). Expected version 1 got 0
I20251025 07:52:50.138715 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 72dc084f8efc45478771c247e07a16f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.138764 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 755f0d6c87044e20a8020f8410940123 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 4 got 0
I20251025 07:52:50.138995 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet e330e1b53bed4e7cace1c4fb7fa6b565 (table default.Table_a9026f5e095c4063b86d4c544c686e22 [id=7944225d3ce7439b881bfb9289d9f040]). Expected version 2 got 0
I20251025 07:52:50.139048 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet fd265cdbf6a8440393a2b5d8240d1872 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.139348 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b594d4718344aafbdaec34040695589 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.139650 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 570201e7bf05406a97c81dde071d894b with delete_type TABLET_DATA_DELETED (replaced by tablet 3a0f50c8000144a4acb5219484b01800) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.139742 455 catalog_manager.cc:5649] T e9e8fffb8e984463b5fa2c7dd9e93227 P 23eeffd3de254ce18db5febfde971c6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } }
I20251025 07:52:50.140064 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5]). Expected version 1 got 0
I20251025 07:52:50.140333 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet f70091024c034c118fb86578db19a31c (table default.Table_b7da40b96d6c4359bed6f898e8cf6275 [id=e041946c093b4c789ce175c736cc230b]). Expected version 2 got 0
I20251025 07:52:50.140887 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 3a0f50c8000144a4acb5219484b01800 (table default.Table_c62e0dc1ae504f9bb912013506ff3031 [id=6f395892b09240fea02482b3199cfb5e]). Expected version 1 got 0
I20251025 07:52:50.141069 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236]). Expected version 1 got 0
I20251025 07:52:50.141245 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 8707b93d491e4e4386d148087a2f0870 (table default.Table_799dcb1cef2f4fe6b9b8f17f9f1026c5 [id=59840a567e5244e8a9024cbad42471e4]). Expected version 1 got 0
I20251025 07:52:50.141763 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 5dab439bc50d4e869548154e1d222ad6 (table default.Table_fb9def2302784c999bf0f5707c60eb4c [id=45325c188eac4d34bb016d01de091f87]). Expected version 3 got 0
I20251025 07:52:50.142144 455 catalog_manager.cc:5649] T 645263a61b654f04985c40d04506c087 P 23eeffd3de254ce18db5febfde971c6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } }
I20251025 07:52:50.142829 450 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_208cd3f3c07744ed92b1260ebb401310" } modify_external_catalogs: true
I20251025 07:52:50.143034 450 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_208cd3f3c07744ed92b1260ebb401310" } modify_external_catalogs: true
I20251025 07:52:50.144690 1167 tablet_bootstrap.cc:654] T f7011f1dcad74fe4ba0356c618890e06 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:50.147069 2888 raft_consensus.cc:493] T 19f1b56cdbd24ebe8688cc05fb795d07 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:50.147208 2888 raft_consensus.cc:515] T 19f1b56cdbd24ebe8688cc05fb795d07 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.147768 2888 leader_election.cc:290] T 19f1b56cdbd24ebe8688cc05fb795d07 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:50.150596 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19f1b56cdbd24ebe8688cc05fb795d07" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:50.151289 712 leader_election.cc:343] T 19f1b56cdbd24ebe8688cc05fb795d07 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.152976 710 leader_election.cc:304] T aea8fcf0154a4bd4a05b19e27ab78b79 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.153012 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e098c9c4613459295b6043048cba587" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:50.153144 768 raft_consensus.cc:2468] T 2e098c9c4613459295b6043048cba587 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:50.153671 2888 raft_consensus.cc:2804] T aea8fcf0154a4bd4a05b19e27ab78b79 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:50.153745 2888 raft_consensus.cc:493] T aea8fcf0154a4bd4a05b19e27ab78b79 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:50.153805 2888 raft_consensus.cc:3060] T aea8fcf0154a4bd4a05b19e27ab78b79 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.154280 1169 tablet_bootstrap.cc:492] T 29dbaf9149134f548a974a5779f7da67 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:50.154417 1169 ts_tablet_manager.cc:1403] T 29dbaf9149134f548a974a5779f7da67 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20251025 07:52:50.154443 2166 ts_tablet_manager.cc:1929] T fce535a0c0bd45c6bf17f81add297f40 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:50.154558 2166 log.cc:1199] T fce535a0c0bd45c6bf17f81add297f40 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/fce535a0c0bd45c6bf17f81add297f40
I20251025 07:52:50.154848 2166 ts_tablet_manager.cc:1950] T fce535a0c0bd45c6bf17f81add297f40 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:50.155102 1169 raft_consensus.cc:359] T 29dbaf9149134f548a974a5779f7da67 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:50.155275 1169 raft_consensus.cc:385] T 29dbaf9149134f548a974a5779f7da67 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.155354 1169 raft_consensus.cc:740] T 29dbaf9149134f548a974a5779f7da67 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.155676 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 218) took 6381 ms. Trace:
I20251025 07:52:50.155772 2166 rpcz_store.cc:276] 1025 07:52:43.774177 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:43.774865 (+ 688us) service_pool.cc:225] Handling call
1025 07:52:50.133266 (+6358401us) ts_tablet_manager.cc:1151] Deleting tablet fce535a0c0bd45c6bf17f81add297f40
1025 07:52:50.133284 (+ 18us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.150931 (+ 17647us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.154341 (+ 3410us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.155652 (+ 1311us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6300583}
I20251025 07:52:50.156349 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet fce535a0c0bd45c6bf17f81add297f40 (table default.Table_1de0b151cf4c4c10aca35e45623caf2c [id=a947f21398254a53bc6668c4f8d39e53]) successfully deleted
I20251025 07:52:50.156612 2888 raft_consensus.cc:515] T aea8fcf0154a4bd4a05b19e27ab78b79 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.157548 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aea8fcf0154a4bd4a05b19e27ab78b79" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:50.157822 2888 leader_election.cc:290] T aea8fcf0154a4bd4a05b19e27ab78b79 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:50.158010 712 leader_election.cc:343] T aea8fcf0154a4bd4a05b19e27ab78b79 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.158350 2686 consensus_queue.cc:1048] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:50.158197 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet b56209e4934e4407a8b4518ca4af0d46 (table default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de]). Expected version 1 got 0
W20251025 07:52:50.159422 850 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:45417, user_credentials={real_user=slave}} blocked reactor thread for 56039.8us
W20251025 07:52:50.159976 850 leader_election.cc:343] T 792f32cbd66449ca98813c669b2a261d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.160583 1048 raft_consensus.cc:3060] T 645263a61b654f04985c40d04506c087 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:50.161790 710 leader_election.cc:343] T 294d7d20c2f545b8813e632aa058a60c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.161901 710 leader_election.cc:304] T 294d7d20c2f545b8813e632aa058a60c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.162287 2888 raft_consensus.cc:2749] T 294d7d20c2f545b8813e632aa058a60c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:50.162688 710 leader_election.cc:343] T 66134661be834f7b811d7927ab629812 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.162777 710 leader_election.cc:304] T 66134661be834f7b811d7927ab629812 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.163087 2888 raft_consensus.cc:2749] T 66134661be834f7b811d7927ab629812 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:50.162117 2169 ts_tablet_manager.cc:1929] T dda0b26e4f1a44928bc4b84f333901c0 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:50.164091 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 219) took 6252 ms. Trace:
I20251025 07:52:50.164170 2166 rpcz_store.cc:276] 1025 07:52:43.911908 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:43.914373 (+ 2465us) service_pool.cc:225] Handling call
1025 07:52:50.163972 (+6249599us) ts_tablet_manager.cc:1151] Deleting tablet 32a143a715cb485cb4501f4e28c68252
1025 07:52:50.163982 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.164074 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6249418}
I20251025 07:52:50.165066 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 220) took 6196 ms. Trace:
I20251025 07:52:50.165174 2991 raft_consensus.cc:515] T 48b73e0c0dce4be686d05f991a21bd19 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:50.163995 1043 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee1c8ea694284d80a30f9c05be4cbf49" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:50.165156 2166 rpcz_store.cc:276] 1025 07:52:43.968563 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:43.968928 (+ 365us) service_pool.cc:225] Handling call
1025 07:52:50.164935 (+6196007us) ts_tablet_manager.cc:1151] Deleting tablet 0540d6a10cf04fa89756623e9aaaeb68
1025 07:52:50.164948 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.165046 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6195747}
I20251025 07:52:50.165400 2169 log.cc:1199] T dda0b26e4f1a44928bc4b84f333901c0 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/dda0b26e4f1a44928bc4b84f333901c0
I20251025 07:52:50.165602 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 225) took 5655 ms. Trace:
I20251025 07:52:50.165719 2169 ts_tablet_manager.cc:1950] T dda0b26e4f1a44928bc4b84f333901c0 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:50.165694 2166 rpcz_store.cc:276] 1025 07:52:44.509812 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.509880 (+ 68us) service_pool.cc:225] Handling call
1025 07:52:50.165509 (+5655629us) ts_tablet_manager.cc:1151] Deleting tablet 60c60a9ee37f43978910bd067a098bb7
1025 07:52:50.165520 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.165585 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5655395}
I20251025 07:52:50.165966 2991 leader_election.cc:290] T 48b73e0c0dce4be686d05f991a21bd19 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:50.163223 1169 consensus_queue.cc:260] T 29dbaf9149134f548a974a5779f7da67 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:50.166108 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 226) took 5596 ms. Trace:
I20251025 07:52:50.166244 3143 raft_consensus.cc:493] T 29dbaf9149134f548a974a5779f7da67 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:50.166203 2166 rpcz_store.cc:276] 1025 07:52:44.569636 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.569692 (+ 56us) service_pool.cc:225] Handling call
1025 07:52:50.166032 (+5596340us) ts_tablet_manager.cc:1151] Deleting tablet f3001314b56345f5a70314777cae0125
1025 07:52:50.166040 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.166093 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5596163}
W20251025 07:52:50.166319 850 leader_election.cc:343] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.166409 3143 raft_consensus.cc:515] T 29dbaf9149134f548a974a5779f7da67 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:50.166628 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 227) took 5596 ms. Trace:
I20251025 07:52:50.166689 2166 rpcz_store.cc:276] 1025 07:52:44.570165 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.570219 (+ 54us) service_pool.cc:225] Handling call
1025 07:52:50.166552 (+5596333us) ts_tablet_manager.cc:1151] Deleting tablet f170424be2f643e7af499834955472b0
1025 07:52:50.166563 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.166616 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5596173}
I20251025 07:52:50.166756 1043 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "48b73e0c0dce4be686d05f991a21bd19" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:50.167080 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 229) took 5596 ms. Trace:
I20251025 07:52:50.167158 2166 rpcz_store.cc:276] 1025 07:52:44.570891 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.570942 (+ 51us) service_pool.cc:225] Handling call
1025 07:52:50.166999 (+5596057us) ts_tablet_manager.cc:1151] Deleting tablet be5ab0db6ac34c2c9a370445e251eb46
1025 07:52:50.167010 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.167064 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5595908}
I20251025 07:52:50.167742 765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6ae1315b28940d49b667e3c80347021" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:50.167901 765 raft_consensus.cc:3060] T a6ae1315b28940d49b667e3c80347021 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.168638 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 228) took 5597 ms. Trace:
I20251025 07:52:50.168721 2166 rpcz_store.cc:276] 1025 07:52:44.570745 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.571013 (+ 268us) service_pool.cc:225] Handling call
1025 07:52:50.168560 (+5597547us) ts_tablet_manager.cc:1151] Deleting tablet a13bf4d7eb1440f7a28b85cb1bf99667
1025 07:52:50.168570 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.168623 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5597364}
I20251025 07:52:50.168917 3143 leader_election.cc:290] T 29dbaf9149134f548a974a5779f7da67 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:50.170324 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29dbaf9149134f548a974a5779f7da67" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:50.170571 768 raft_consensus.cc:2393] T 29dbaf9149134f548a974a5779f7da67 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:50.170859 765 raft_consensus.cc:2468] T a6ae1315b28940d49b667e3c80347021 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:50.172401 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29dbaf9149134f548a974a5779f7da67" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:50.164738 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 32a143a715cb485cb4501f4e28c68252 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32a143a715cb485cb4501f4e28c68252 already in progress: creating tablet
I20251025 07:52:50.172631 2166 tablet_replica.cc:333] T 95efd553493e4966bc48a01cac6e1e10 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:50.172809 2166 raft_consensus.cc:2243] T 95efd553493e4966bc48a01cac6e1e10 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:50.172946 2166 raft_consensus.cc:2272] T 95efd553493e4966bc48a01cac6e1e10 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:50.172919 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 0540d6a10cf04fa89756623e9aaaeb68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0540d6a10cf04fa89756623e9aaaeb68 already in progress: creating tablet
W20251025 07:52:50.172955 850 leader_election.cc:343] T 48b73e0c0dce4be686d05f991a21bd19 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:50.173171 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 60c60a9ee37f43978910bd067a098bb7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60c60a9ee37f43978910bd067a098bb7 already in progress: creating tablet
I20251025 07:52:50.173216 983 leader_election.cc:304] T 29dbaf9149134f548a974a5779f7da67 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
W20251025 07:52:50.173373 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet f3001314b56345f5a70314777cae0125 because the tablet was not found. No further retry: Not found: Tablet not found: f3001314b56345f5a70314777cae0125
I20251025 07:52:50.173467 2166 ts_tablet_manager.cc:1916] T 95efd553493e4966bc48a01cac6e1e10 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:50.173583 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet f170424be2f643e7af499834955472b0 because the tablet was not found. No further retry: Not found: Tablet not found: f170424be2f643e7af499834955472b0
I20251025 07:52:50.173635 3011 raft_consensus.cc:3060] T 29dbaf9149134f548a974a5779f7da67 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:50.173746 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet be5ab0db6ac34c2c9a370445e251eb46 because the tablet was not found. No further retry: Not found: Tablet not found: be5ab0db6ac34c2c9a370445e251eb46
W20251025 07:52:50.173982 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet a13bf4d7eb1440f7a28b85cb1bf99667 because the tablet was not found. No further retry: Not found: Tablet not found: a13bf4d7eb1440f7a28b85cb1bf99667
I20251025 07:52:50.174166 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet fa998b9ca6724c4cb1eb68a3eecb0b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.174806 455 catalog_manager.cc:5649] T fa826b85f0c94150a17d4e417dd3d62f P 23eeffd3de254ce18db5febfde971c6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } }
I20251025 07:52:50.175081 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet ff3daed6c9af47e3976b82b4b9a41086 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.176263 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet a7ab3150a35b4caca9119a1374d3f2f7 with delete_type TABLET_DATA_DELETED (replaced by tablet c19c97f881cb45218d4a6f6f231aa96e) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.178061 3011 raft_consensus.cc:2749] T 29dbaf9149134f548a974a5779f7da67 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:50.180816 1169 ts_tablet_manager.cc:1434] T 29dbaf9149134f548a974a5779f7da67 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20251025 07:52:50.181965 2888 raft_consensus.cc:493] T 3aeaa0e9dc594e21a9ddd923e314017d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:50.182088 2888 raft_consensus.cc:515] T 3aeaa0e9dc594e21a9ddd923e314017d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.182626 2888 leader_election.cc:290] T 3aeaa0e9dc594e21a9ddd923e314017d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:50.183566 1166 tablet_bootstrap.cc:492] T 22f7db660616457b83b24e947a0b0bae P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:50.183754 1166 ts_tablet_manager.cc:1403] T 22f7db660616457b83b24e947a0b0bae P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.082s user 0.004s sys 0.000s
I20251025 07:52:50.184527 1166 raft_consensus.cc:359] T 22f7db660616457b83b24e947a0b0bae P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.184753 1166 raft_consensus.cc:385] T 22f7db660616457b83b24e947a0b0bae P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.184881 1166 raft_consensus.cc:740] T 22f7db660616457b83b24e947a0b0bae P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.185166 1166 consensus_queue.cc:260] T 22f7db660616457b83b24e947a0b0bae P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.185743 1166 ts_tablet_manager.cc:1434] T 22f7db660616457b83b24e947a0b0bae P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:50.186024 1043 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3aeaa0e9dc594e21a9ddd923e314017d" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:50.186502 712 leader_election.cc:343] T 3aeaa0e9dc594e21a9ddd923e314017d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.186728 1166 tablet_bootstrap.cc:492] T d40519ed9ee746348b17be2786d60e95 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:50.191932 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84]). Expected version 1 got 0
I20251025 07:52:50.192437 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet dda0b26e4f1a44928bc4b84f333901c0 (table default.Table_c1bf6bb7c62c4648853b9a5659c73da0 [id=d0b9f20f28d340f4944e85ee0f563fab]) successfully deleted
I20251025 07:52:50.192373 455 catalog_manager.cc:5649] T dff7f86ae8924685b26011e504e2247b P 23eeffd3de254ce18db5febfde971c6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } }
I20251025 07:52:50.192699 2166 ts_tablet_manager.cc:1929] T 95efd553493e4966bc48a01cac6e1e10 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:50.192807 2166 log.cc:1199] T 95efd553493e4966bc48a01cac6e1e10 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/95efd553493e4966bc48a01cac6e1e10
I20251025 07:52:50.193101 2166 ts_tablet_manager.cc:1950] T 95efd553493e4966bc48a01cac6e1e10 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:50.193531 1169 tablet_bootstrap.cc:492] T a7ab3150a35b4caca9119a1374d3f2f7 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:50.193825 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 231) took 5622 ms. Trace:
I20251025 07:52:50.193914 2166 rpcz_store.cc:276] 1025 07:52:44.571522 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.571568 (+ 46us) service_pool.cc:225] Handling call
1025 07:52:50.170186 (+5598618us) ts_tablet_manager.cc:1151] Deleting tablet 95efd553493e4966bc48a01cac6e1e10
1025 07:52:50.172251 (+ 2065us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.176857 (+ 4606us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.192583 (+ 15726us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.193811 (+ 1228us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5598469}
I20251025 07:52:50.194308 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 230) took 5622 ms. Trace:
I20251025 07:52:50.194389 2166 rpcz_store.cc:276] 1025 07:52:44.571409 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.571810 (+ 401us) service_pool.cc:225] Handling call
1025 07:52:50.194204 (+5622394us) ts_tablet_manager.cc:1151] Deleting tablet 7f29b64704e147aa9b0b5f6cda860f09
1025 07:52:50.194212 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.194293 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5622243}
W20251025 07:52:50.192937 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 80ee75935d7e461d9d4066fac9819bef because the tablet was not found. No further retry: Not found: Tablet not found: 80ee75935d7e461d9d4066fac9819bef
I20251025 07:52:50.196107 1166 tablet_bootstrap.cc:654] T d40519ed9ee746348b17be2786d60e95 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:50.196189 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet c88c46e022ff41a58b02538f2151e260 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:44572
W20251025 07:52:50.196343 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 3b940fa5ae26427dafbcfd26372754b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b940fa5ae26427dafbcfd26372754b3 already in progress: creating tablet
W20251025 07:52:50.196535 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet b3a0af893a8947848e12093d0a7f0229 because the tablet was not found. No further retry: Not found: Tablet not found: b3a0af893a8947848e12093d0a7f0229
W20251025 07:52:50.196669 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet f661d5909657455581dcb5e6eadd3c0b because the tablet was not found. No further retry: Not found: Tablet not found: f661d5909657455581dcb5e6eadd3c0b
W20251025 07:52:50.196805 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 88eb51447e0f4080baa49dd902deb24f because the tablet was not found. No further retry: Not found: Tablet not found: 88eb51447e0f4080baa49dd902deb24f
W20251025 07:52:50.196928 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 60c60a9ee37f43978910bd067a098bb7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60c60a9ee37f43978910bd067a098bb7 already in progress: creating tablet
I20251025 07:52:50.196944 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet 53198283718d4b45b30df65b5bde96a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:44572
W20251025 07:52:50.197062 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet f97acc9a638b4d5a968dcc3412a384bc because the tablet was not found. No further retry: Not found: Tablet not found: f97acc9a638b4d5a968dcc3412a384bc
W20251025 07:52:50.197180 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 26c849d376d143c6bef0c8e0e1d7d03a because the tablet was not found. No further retry: Not found: Tablet not found: 26c849d376d143c6bef0c8e0e1d7d03a
I20251025 07:52:50.194406 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 95efd553493e4966bc48a01cac6e1e10 (table default.Table_c4716673eec04f3baa6fc852c668fed8 [id=1d09f9e02b4a4a4d8d38ee522fea841b]) successfully deleted
I20251025 07:52:50.197350 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet 08da9d2687704725aacd11cd92c6a9f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:44572
W20251025 07:52:50.197543 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 7f29b64704e147aa9b0b5f6cda860f09 because the tablet was not found. No further retry: Not found: Tablet not found: 7f29b64704e147aa9b0b5f6cda860f09
I20251025 07:52:50.197839 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet 7fbae8be73bf4e18b5a5c4ec4cdebc12 with delete_type TABLET_DATA_DELETED (replaced by tablet 4c155aa7fa5b4d68bbab1918b8432372) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.198019 1015 tablet_service.cc:1552] Processing DeleteTablet for tablet 3abfd52327a04ba88625f391a8658fbb with delete_type TABLET_DATA_DELETED (replaced by tablet 04d27d54fdb7483b869408af65ff85c9) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.198019 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ec421eb820b4f40afcff5a5bf80c508 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.198251 455 catalog_manager.cc:5649] T a0ea1edeca7347fc844faa5e4be1ae94 P 23eeffd3de254ce18db5febfde971c6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } }
I20251025 07:52:50.198508 2169 tablet_replica.cc:333] T 48ff4ce7846b4e56a37eee9bcb17bd5e P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:50.198710 2169 raft_consensus.cc:2243] T 48ff4ce7846b4e56a37eee9bcb17bd5e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:50.198915 2169 raft_consensus.cc:2272] T 48ff4ce7846b4e56a37eee9bcb17bd5e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:50.199142 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "792f32cbd66449ca98813c669b2a261d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:50.199360 772 raft_consensus.cc:3060] T 792f32cbd66449ca98813c669b2a261d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.199199 455 catalog_manager.cc:5649] T 34c63b69904d499bb0d286941a8303f2 P 23eeffd3de254ce18db5febfde971c6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } }
I20251025 07:52:50.199517 2169 ts_tablet_manager.cc:1916] T 48ff4ce7846b4e56a37eee9bcb17bd5e P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:50.202433 772 raft_consensus.cc:2468] T 792f32cbd66449ca98813c669b2a261d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:50.202579 455 catalog_manager.cc:5649] T 37c7938e8c5e41f08506aff6747801c9 P 23eeffd3de254ce18db5febfde971c6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } }
I20251025 07:52:50.203791 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee1c8ea694284d80a30f9c05be4cbf49" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:50.203953 772 raft_consensus.cc:2468] T ee1c8ea694284d80a30f9c05be4cbf49 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:50.204792 1167 tablet_bootstrap.cc:492] T f7011f1dcad74fe4ba0356c618890e06 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:50.204958 1167 ts_tablet_manager.cc:1403] T f7011f1dcad74fe4ba0356c618890e06 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.069s user 0.002s sys 0.010s
I20251025 07:52:50.205677 1167 raft_consensus.cc:359] T f7011f1dcad74fe4ba0356c618890e06 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.205875 1167 raft_consensus.cc:385] T f7011f1dcad74fe4ba0356c618890e06 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.205964 1167 raft_consensus.cc:740] T f7011f1dcad74fe4ba0356c618890e06 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.206234 1167 consensus_queue.cc:260] T f7011f1dcad74fe4ba0356c618890e06 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.206207 2991 raft_consensus.cc:493] T e429b94326cc4827a03968fd92de38e0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:50.206636 2991 raft_consensus.cc:515] T e429b94326cc4827a03968fd92de38e0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:50.207363 1167 ts_tablet_manager.cc:1434] T f7011f1dcad74fe4ba0356c618890e06 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251025 07:52:50.207466 2991 leader_election.cc:290] T e429b94326cc4827a03968fd92de38e0 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:50.207532 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3]). Expected version 2 got 0
I20251025 07:52:50.208052 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d6a9937d10a4cc2813671dd1bfa1664 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.208122 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e429b94326cc4827a03968fd92de38e0" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:50.208261 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet d5e5e5bbcf9f43118eea502b69880af5 with delete_type TABLET_DATA_DELETED (replaced by tablet f4578ce69a0d4322944e5ce7c8a59831) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.208460 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet f3c87a83082b421cb4be546ac4ce7ad8 with delete_type TABLET_DATA_DELETED (replaced by tablet b17d2a9ad6be49d29f2413666fba7e69) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.208493 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet 551d07a9242740dcb9366bb2d707a4a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.208635 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b28215f61574cf48c447dd01955ac77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.209098 1167 tablet_bootstrap.cc:492] T 32a143a715cb485cb4501f4e28c68252 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
W20251025 07:52:50.209092 850 leader_election.cc:343] T e429b94326cc4827a03968fd92de38e0 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.209957 1169 tablet_bootstrap.cc:654] T a7ab3150a35b4caca9119a1374d3f2f7 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:50.210470 850 consensus_peers.cc:597] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cb829649f73d4d3f8862d268c953bd86. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:50.207876 455 catalog_manager.cc:5649] T 1aeba18be0cb4eb9949aeaf6ad73a812 P 23eeffd3de254ce18db5febfde971c6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } }
I20251025 07:52:50.211498 1167 tablet_bootstrap.cc:654] T 32a143a715cb485cb4501f4e28c68252 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:50.212131 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 288e543bb279410c91c01dd1623e110f (table default.Table_6e242ba55b194c23a6701f4b064fdbe5 [id=4c34b64b299f4162a53866cf01c5298f]). Expected version 1 got 0
I20251025 07:52:50.211989 2166 tablet_replica.cc:333] T c11f674385184ae6b235f1db8bf1fb91 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:50.215742 1166 tablet_bootstrap.cc:492] T d40519ed9ee746348b17be2786d60e95 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:50.215993 1166 ts_tablet_manager.cc:1403] T d40519ed9ee746348b17be2786d60e95 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20251025 07:52:50.217365 2146 ts_tablet_manager.cc:1929] T 570201e7bf05406a97c81dde071d894b P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:50.217568 2146 log.cc:1199] T 570201e7bf05406a97c81dde071d894b P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/570201e7bf05406a97c81dde071d894b
I20251025 07:52:50.217754 1166 raft_consensus.cc:359] T d40519ed9ee746348b17be2786d60e95 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.217967 1166 raft_consensus.cc:385] T d40519ed9ee746348b17be2786d60e95 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.218055 1166 raft_consensus.cc:740] T d40519ed9ee746348b17be2786d60e95 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.218209 2146 ts_tablet_manager.cc:1950] T 570201e7bf05406a97c81dde071d894b P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:50.218365 1166 consensus_queue.cc:260] T d40519ed9ee746348b17be2786d60e95 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.219142 455 catalog_manager.cc:5741] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231) does not have the latest schema for tablet 8e6d76b1a97945cfb6c3055e1b622f8a (table default.Table_a692a40ea2ef44ffa5ab6ca9e29ef70a [id=f5c1be203e2b4831a025871cf8a2ef17]). Expected version 3 got 2
I20251025 07:52:50.219503 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 276) took 5632 ms. Trace:
I20251025 07:52:50.219696 1166 ts_tablet_manager.cc:1434] T d40519ed9ee746348b17be2786d60e95 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251025 07:52:50.219993 1166 tablet_bootstrap.cc:492] T 8707b93d491e4e4386d148087a2f0870 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:50.223336 3203 raft_consensus.cc:493] T 3aeaa0e9dc594e21a9ddd923e314017d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:50.223529 3203 raft_consensus.cc:515] T 3aeaa0e9dc594e21a9ddd923e314017d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.224118 3203 leader_election.cc:290] T 3aeaa0e9dc594e21a9ddd923e314017d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:50.224830 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3aeaa0e9dc594e21a9ddd923e314017d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:50.226846 2166 raft_consensus.cc:2243] T c11f674385184ae6b235f1db8bf1fb91 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:50.227100 2166 raft_consensus.cc:2272] T c11f674385184ae6b235f1db8bf1fb91 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:50.227346 850 leader_election.cc:343] T 3aeaa0e9dc594e21a9ddd923e314017d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.227661 2166 ts_tablet_manager.cc:1916] T c11f674385184ae6b235f1db8bf1fb91 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:50.229568 2169 ts_tablet_manager.cc:1929] T 48ff4ce7846b4e56a37eee9bcb17bd5e P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:50.230741 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d40519ed9ee746348b17be2786d60e95" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:50.230988 910 raft_consensus.cc:2468] T d40519ed9ee746348b17be2786d60e95 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:50.231197 2169 log.cc:1199] T 48ff4ce7846b4e56a37eee9bcb17bd5e P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/48ff4ce7846b4e56a37eee9bcb17bd5e
I20251025 07:52:50.231467 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22f7db660616457b83b24e947a0b0bae" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:50.231647 910 raft_consensus.cc:2468] T 22f7db660616457b83b24e947a0b0bae P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:50.232008 2169 ts_tablet_manager.cc:1950] T 48ff4ce7846b4e56a37eee9bcb17bd5e P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:50.232067 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8707b93d491e4e4386d148087a2f0870" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:50.232560 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19f1b56cdbd24ebe8688cc05fb795d07" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:50.233035 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aea8fcf0154a4bd4a05b19e27ab78b79" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:50.233191 910 raft_consensus.cc:3060] T aea8fcf0154a4bd4a05b19e27ab78b79 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.234023 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 48ff4ce7846b4e56a37eee9bcb17bd5e (table default.Table_5eaf42285e474860a44c19683f1fa70f [id=0da8afa0245842cfa80b852852e5235a]) successfully deleted
I20251025 07:52:50.235231 2169 tablet_replica.cc:333] T 7e6544202d4f489d82f22d6073ef240d P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:50.236210 2166 ts_tablet_manager.cc:1929] T c11f674385184ae6b235f1db8bf1fb91 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:50.236429 2169 raft_consensus.cc:2243] T 7e6544202d4f489d82f22d6073ef240d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:50.236517 910 raft_consensus.cc:2468] T aea8fcf0154a4bd4a05b19e27ab78b79 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:50.237252 2169 raft_consensus.cc:2272] T 7e6544202d4f489d82f22d6073ef240d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:50.237838 2169 ts_tablet_manager.cc:1916] T 7e6544202d4f489d82f22d6073ef240d P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:50.238214 2166 log.cc:1199] T c11f674385184ae6b235f1db8bf1fb91 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/c11f674385184ae6b235f1db8bf1fb91
I20251025 07:52:50.238600 2166 ts_tablet_manager.cc:1950] T c11f674385184ae6b235f1db8bf1fb91 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
W20251025 07:52:50.238559 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 062c0f3e51a3487880cea373f8cf1f54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 062c0f3e51a3487880cea373f8cf1f54 already in progress: creating tablet
I20251025 07:52:50.239427 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 234) took 5667 ms. Trace:
I20251025 07:52:50.239564 2166 rpcz_store.cc:276] 1025 07:52:44.572415 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.572857 (+ 442us) service_pool.cc:225] Handling call
1025 07:52:50.211945 (+5639088us) ts_tablet_manager.cc:1151] Deleting tablet c11f674385184ae6b235f1db8bf1fb91
1025 07:52:50.211954 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.230761 (+ 18807us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.236116 (+ 5355us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.239406 (+ 3290us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5638913}
I20251025 07:52:50.240731 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet c11f674385184ae6b235f1db8bf1fb91 (table default.Table_ce27d8ed3ede46579f2ed3ae81536572 [id=c4c41c3b236141eb8e03dbb171f6b56d]) successfully deleted
I20251025 07:52:50.243038 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet d66ad049d5484228a268b0facb00ecf1 with delete_type TABLET_DATA_DELETED (replaced by tablet 2e098c9c4613459295b6043048cba587) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.243824 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet af743ee344e543a0aa6da3cd7a30e467 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.244035 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet 34315f64b89148efbf1f20a43d5c5efb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.244915 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet a4e4b99be0434baaa75d988281797846 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.245597 1048 raft_consensus.cc:3060] T e8402fd33ba842e981c6c68787c7c548 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.245606 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet 92aa605fabbc4257a6a204c087e7cc7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.246302 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet b94634ac5077439fadfbbce569f60e8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.246755 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet e54fe7cbcffa40c4b04144b47bee2b48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.247478 1166 tablet_bootstrap.cc:654] T 8707b93d491e4e4386d148087a2f0870 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:50.248520 452 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:38361 (request call id 214) took 5563 ms (client timeout 1000 ms). Trace:
I20251025 07:52:50.258771 710 leader_election.cc:304] T d40519ed9ee746348b17be2786d60e95 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.259220 2888 raft_consensus.cc:2804] T d40519ed9ee746348b17be2786d60e95 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:50.259338 2888 raft_consensus.cc:493] T d40519ed9ee746348b17be2786d60e95 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:50.259408 2888 raft_consensus.cc:3060] T d40519ed9ee746348b17be2786d60e95 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.260887 710 leader_election.cc:304] T 22f7db660616457b83b24e947a0b0bae P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.261269 3126 raft_consensus.cc:2804] T 22f7db660616457b83b24e947a0b0bae P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:50.261354 3126 raft_consensus.cc:493] T 22f7db660616457b83b24e947a0b0bae P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:50.261425 3126 raft_consensus.cc:3060] T 22f7db660616457b83b24e947a0b0bae P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.264151 3126 raft_consensus.cc:515] T 22f7db660616457b83b24e947a0b0bae P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.264706 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3aeaa0e9dc594e21a9ddd923e314017d" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:50.264942 910 raft_consensus.cc:2468] T 3aeaa0e9dc594e21a9ddd923e314017d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
W20251025 07:52:50.265655 710 leader_election.cc:343] T 8707b93d491e4e4386d148087a2f0870 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.265805 710 leader_election.cc:304] T 8707b93d491e4e4386d148087a2f0870 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.266284 3142 raft_consensus.cc:2749] T 8707b93d491e4e4386d148087a2f0870 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:50.266872 710 leader_election.cc:343] T 19f1b56cdbd24ebe8688cc05fb795d07 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.266978 710 leader_election.cc:304] T 19f1b56cdbd24ebe8688cc05fb795d07 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.267302 3142 raft_consensus.cc:2749] T 19f1b56cdbd24ebe8688cc05fb795d07 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:50.267625 710 leader_election.cc:304] T aea8fcf0154a4bd4a05b19e27ab78b79 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.267940 3142 raft_consensus.cc:2804] T aea8fcf0154a4bd4a05b19e27ab78b79 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:50.268047 3142 raft_consensus.cc:697] T aea8fcf0154a4bd4a05b19e27ab78b79 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:50.270546 2991 raft_consensus.cc:493] T df9db9dd31d840e0b1394db04d662a62 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:50.270632 1043 raft_consensus.cc:3060] T c501ef5d70e742598c1a902efe47feb1 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.270763 2991 raft_consensus.cc:515] T df9db9dd31d840e0b1394db04d662a62 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.271656 2991 leader_election.cc:290] T df9db9dd31d840e0b1394db04d662a62 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:50.272665 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df9db9dd31d840e0b1394db04d662a62" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:50.273402 850 leader_election.cc:343] T df9db9dd31d840e0b1394db04d662a62 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.219663 2146 rpcz_store.cc:276] 1025 07:52:44.586949 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.587009 (+ 60us) service_pool.cc:225] Handling call
1025 07:52:49.995496 (+5408487us) ts_tablet_manager.cc:1151] Deleting tablet 570201e7bf05406a97c81dde071d894b
1025 07:52:49.995505 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.002896 (+ 7391us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.217260 (+214364us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.219039 (+ 1779us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5388238}
I20251025 07:52:50.276414 1169 tablet_bootstrap.cc:492] T a7ab3150a35b4caca9119a1374d3f2f7 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:50.276698 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 286) took 5670 ms. Trace:
I20251025 07:52:50.276809 2146 rpcz_store.cc:276] 1025 07:52:44.605975 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.607336 (+ 1361us) service_pool.cc:225] Handling call
1025 07:52:50.276538 (+5669202us) ts_tablet_manager.cc:1151] Deleting tablet a13bf4d7eb1440f7a28b85cb1bf99667
1025 07:52:50.276550 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.276622 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5669073}
I20251025 07:52:50.276947 1169 ts_tablet_manager.cc:1403] T a7ab3150a35b4caca9119a1374d3f2f7 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.084s user 0.011s sys 0.005s
I20251025 07:52:50.277158 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 287) took 5666 ms. Trace:
I20251025 07:52:50.277248 2146 rpcz_store.cc:276] 1025 07:52:44.610243 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.610454 (+ 211us) service_pool.cc:225] Handling call
1025 07:52:50.277039 (+5666585us) ts_tablet_manager.cc:1151] Deleting tablet c88c46e022ff41a58b02538f2151e260
1025 07:52:50.277048 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.277102 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5666432}
I20251025 07:52:50.277623 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 288) took 5657 ms. Trace:
I20251025 07:52:50.277720 2146 rpcz_store.cc:276] 1025 07:52:44.620357 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.620409 (+ 52us) service_pool.cc:225] Handling call
1025 07:52:50.277491 (+5657082us) ts_tablet_manager.cc:1151] Deleting tablet 9d6a9937d10a4cc2813671dd1bfa1664
1025 07:52:50.277499 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.277548 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5656917}
I20251025 07:52:50.278064 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 289) took 5653 ms. Trace:
I20251025 07:52:50.278810 847 leader_election.cc:304] T 2e098c9c4613459295b6043048cba587 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.278693 1169 raft_consensus.cc:359] T a7ab3150a35b4caca9119a1374d3f2f7 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:50.278967 1169 raft_consensus.cc:385] T a7ab3150a35b4caca9119a1374d3f2f7 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.276762 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 570201e7bf05406a97c81dde071d894b (table default.Table_c62e0dc1ae504f9bb912013506ff3031 [id=6f395892b09240fea02482b3199cfb5e]) successfully deleted
I20251025 07:52:50.279210 2990 raft_consensus.cc:2804] T 2e098c9c4613459295b6043048cba587 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:50.279372 2990 raft_consensus.cc:493] T 2e098c9c4613459295b6043048cba587 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:50.279490 2990 raft_consensus.cc:3060] T 2e098c9c4613459295b6043048cba587 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.279575 1169 raft_consensus.cc:740] T a7ab3150a35b4caca9119a1374d3f2f7 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.279937 1169 consensus_queue.cc:260] T a7ab3150a35b4caca9119a1374d3f2f7 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:50.282811 2990 raft_consensus.cc:515] T 2e098c9c4613459295b6043048cba587 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:50.283315 1169 ts_tablet_manager.cc:1434] T a7ab3150a35b4caca9119a1374d3f2f7 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20251025 07:52:50.283970 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e098c9c4613459295b6043048cba587" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:50.284756 850 leader_election.cc:343] T 2e098c9c4613459295b6043048cba587 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.284982 2990 leader_election.cc:290] T 2e098c9c4613459295b6043048cba587 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
W20251025 07:52:50.279372 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet a13bf4d7eb1440f7a28b85cb1bf99667 because the tablet was not found. No further retry: Not found: Tablet not found: a13bf4d7eb1440f7a28b85cb1bf99667
W20251025 07:52:50.285593 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet c88c46e022ff41a58b02538f2151e260 because the tablet was not found. No further retry: Not found: Tablet not found: c88c46e022ff41a58b02538f2151e260
W20251025 07:52:50.286506 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 9d6a9937d10a4cc2813671dd1bfa1664 because the tablet was not found. No further retry: Not found: Tablet not found: 9d6a9937d10a4cc2813671dd1bfa1664
W20251025 07:52:50.290849 850 consensus_peers.cc:597] T 2d24e3e7a9e4431db609331ad1fd57c9 P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:50.291327 847 leader_election.cc:304] T a6ae1315b28940d49b667e3c80347021 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.284879 1169 tablet_bootstrap.cc:492] T e47c4f741f354a19b49777bd06e54764 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
W20251025 07:52:50.248638 452 rpcz_store.cc:269] 1025 07:52:44.684945 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.719694 (+ 34749us) service_pool.cc:225] Handling call
1025 07:52:45.565065 (+845371us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.565070 (+ 5us) write_op.cc:454] Released schema lock
1025 07:52:50.248483 (+4683413us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:45.209962 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:45.210088 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:45.210094 (+ 6us) write_op.cc:435] Acquired schema lock
1025 07:52:45.210096 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:45.210264 (+ 168us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:45.210278 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:45.210283 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:45.210349 (+ 66us) tablet.cc:699] Row locks acquired
1025 07:52:45.210351 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:45.210385 (+ 34us) write_op.cc:270] Start()
1025 07:52:45.210423 (+ 38us) write_op.cc:276] Timestamp: P: 1761378765210370 usec, L: 0
1025 07:52:45.210429 (+ 6us) op_driver.cc:348] REPLICATION: starting
1025 07:52:45.210627 (+ 198us) log.cc:844] Serialized 527 byte log entry
1025 07:52:45.214613 (+ 3986us) op_driver.cc:464] REPLICATION: finished
1025 07:52:45.214750 (+ 137us) write_op.cc:301] APPLY: starting
1025 07:52:45.214766 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:45.214810 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:45.214840 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:45.214896 (+ 56us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:45.214933 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:45.214940 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:45.215016 (+ 76us) log.cc:844] Serialized 23 byte log entry
1025 07:52:45.215078 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.215098 (+ 20us) write_op.cc:454] Released schema lock
1025 07:52:45.215105 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:45.215175 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.215179 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
1025 07:52:45.554345 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:45.554485 (+ 140us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:45.554493 (+ 8us) write_op.cc:435] Acquired schema lock
1025 07:52:45.554496 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:45.554624 (+ 128us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:45.554640 (+ 16us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:45.554645 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1025 07:52:45.554716 (+ 71us) tablet.cc:699] Row locks acquired
1025 07:52:45.554719 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:45.554759 (+ 40us) write_op.cc:270] Start()
1025 07:52:45.554796 (+ 37us) write_op.cc:276] Timestamp: P: 1761378765552234 usec, L: 0
1025 07:52:45.554800 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:45.555041 (+ 241us) log.cc:844] Serialized 356 byte log entry
1025 07:52:45.560492 (+ 5451us) op_driver.cc:464] REPLICATION: finished
1025 07:52:45.560588 (+ 96us) write_op.cc:301] APPLY: starting
1025 07:52:45.560603 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:45.560654 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:45.560686 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:45.560736 (+ 50us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:45.560766 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:45.560773 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:45.560844 (+ 71us) log.cc:844] Serialized 23 byte log entry
1025 07:52:45.560899 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:45.560917 (+ 18us) write_op.cc:454] Released schema lock
1025 07:52:45.560924 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":248,"child_traces":[["op",{"apply.queue_time_us":74,"apply.run_cpu_time_us":420,"apply.run_wall_time_us":418,"num_ops":1,"prepare.queue_time_us":51,"prepare.run_cpu_time_us":774,"prepare.run_wall_time_us":789,"raft.queue_time_us":570,"raft.run_cpu_time_us":515,"raft.run_wall_time_us":515,"replication_time_us":4154,"spinlock_wait_cycles":96000}],["op",{"apply.queue_time_us":51,"apply.run_cpu_time_us":383,"apply.run_wall_time_us":383,"num_ops":1,"prepare.queue_time_us":9731,"prepare.run_cpu_time_us":828,"prepare.run_wall_time_us":828,"raft.queue_time_us":499,"raft.run_cpu_time_us":546,"raft.run_wall_time_us":544,"replication_time_us":5667}]]}
I20251025 07:52:50.278183 2146 rpcz_store.cc:276] 1025 07:52:44.624770 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.625348 (+ 578us) service_pool.cc:225] Handling call
1025 07:52:50.277937 (+5652589us) ts_tablet_manager.cc:1151] Deleting tablet d5e5e5bbcf9f43118eea502b69880af5
1025 07:52:50.277946 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.278002 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5649041}
I20251025 07:52:50.296700 452 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_ac3211ef7b0941d2ad4db2fab62231b2" } modify_external_catalogs: true
I20251025 07:52:50.295976 2991 raft_consensus.cc:2804] T a6ae1315b28940d49b667e3c80347021 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
W20251025 07:52:50.297406 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet d5e5e5bbcf9f43118eea502b69880af5 because the tablet was not found. No further retry: Not found: Tablet not found: d5e5e5bbcf9f43118eea502b69880af5
I20251025 07:52:50.297781 2146 tablet_replica.cc:333] T 4b278ce7768349e49dcd1131e404ab34 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:50.297994 2146 raft_consensus.cc:2243] T 4b278ce7768349e49dcd1131e404ab34 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:50.298256 1169 tablet_bootstrap.cc:654] T e47c4f741f354a19b49777bd06e54764 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:50.300891 2991 raft_consensus.cc:697] T a6ae1315b28940d49b667e3c80347021 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:50.301203 2991 consensus_queue.cc:237] T a6ae1315b28940d49b667e3c80347021 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.311374 1166 tablet_bootstrap.cc:492] T 8707b93d491e4e4386d148087a2f0870 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:50.311578 1166 ts_tablet_manager.cc:1403] T 8707b93d491e4e4386d148087a2f0870 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.092s user 0.015s sys 0.006s
I20251025 07:52:50.312148 847 leader_election.cc:304] T 792f32cbd66449ca98813c669b2a261d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.312373 1166 raft_consensus.cc:359] T 8707b93d491e4e4386d148087a2f0870 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.312577 1166 raft_consensus.cc:385] T 8707b93d491e4e4386d148087a2f0870 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.312680 1166 raft_consensus.cc:740] T 8707b93d491e4e4386d148087a2f0870 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.312903 1166 consensus_queue.cc:260] T 8707b93d491e4e4386d148087a2f0870 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.315430 1169 tablet_bootstrap.cc:492] T e47c4f741f354a19b49777bd06e54764 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:50.315639 1169 ts_tablet_manager.cc:1403] T e47c4f741f354a19b49777bd06e54764 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.003s
I20251025 07:52:50.316447 1169 raft_consensus.cc:359] T e47c4f741f354a19b49777bd06e54764 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:50.316682 1169 raft_consensus.cc:385] T e47c4f741f354a19b49777bd06e54764 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.316788 1169 raft_consensus.cc:740] T e47c4f741f354a19b49777bd06e54764 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.317040 1169 consensus_queue.cc:260] T e47c4f741f354a19b49777bd06e54764 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:50.317271 2990 raft_consensus.cc:2804] T 792f32cbd66449ca98813c669b2a261d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:50.317546 2990 raft_consensus.cc:697] T 792f32cbd66449ca98813c669b2a261d P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:50.317704 1166 ts_tablet_manager.cc:1434] T 8707b93d491e4e4386d148087a2f0870 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251025 07:52:50.318001 1166 tablet_bootstrap.cc:492] T 0582ee42ab564c07ae3facfc5935617c P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:50.318284 2990 consensus_queue.cc:237] T 792f32cbd66449ca98813c669b2a261d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.318481 1169 ts_tablet_manager.cc:1434] T e47c4f741f354a19b49777bd06e54764 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251025 07:52:50.319324 1169 tablet_bootstrap.cc:492] T dc4a788dca9d4a8ba40f8fdf1a4b915e P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:50.320063 1166 tablet_bootstrap.cc:654] T 0582ee42ab564c07ae3facfc5935617c P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:50.320997 1169 tablet_bootstrap.cc:654] T dc4a788dca9d4a8ba40f8fdf1a4b915e P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:50.322311 2146 raft_consensus.cc:2272] T 4b278ce7768349e49dcd1131e404ab34 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:50.322909 2146 ts_tablet_manager.cc:1916] T 4b278ce7768349e49dcd1131e404ab34 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:50.323697 850 consensus_peers.cc:597] T 1c40b2e06d894a06af9e58285892ea8d P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:50.324651 847 leader_election.cc:304] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.324942 2686 raft_consensus.cc:2804] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:50.325049 2686 raft_consensus.cc:493] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:50.325129 2686 raft_consensus.cc:3060] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.328004 2686 raft_consensus.cc:515] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.329095 2686 leader_election.cc:290] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:50.329100 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee1c8ea694284d80a30f9c05be4cbf49" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:50.296926 452 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_ac3211ef7b0941d2ad4db2fab62231b2" } modify_external_catalogs: true
W20251025 07:52:50.329787 850 leader_election.cc:343] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.331140 2166 tablet_replica.cc:333] T 2e098c9c4613459295b6043048cba587 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:50.331360 2166 raft_consensus.cc:2243] T 2e098c9c4613459295b6043048cba587 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:50.331524 2166 raft_consensus.cc:2272] T 2e098c9c4613459295b6043048cba587 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:50.332048 2166 ts_tablet_manager.cc:1916] T 2e098c9c4613459295b6043048cba587 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:50.332208 764 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3aeaa0e9dc594e21a9ddd923e314017d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:50.332360 764 raft_consensus.cc:2468] T 3aeaa0e9dc594e21a9ddd923e314017d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:50.332638 2888 raft_consensus.cc:515] T d40519ed9ee746348b17be2786d60e95 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.332794 1166 tablet_bootstrap.cc:492] T 0582ee42ab564c07ae3facfc5935617c P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:50.332947 1166 ts_tablet_manager.cc:1403] T 0582ee42ab564c07ae3facfc5935617c P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20251025 07:52:50.333220 2888 leader_election.cc:290] T d40519ed9ee746348b17be2786d60e95 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:50.333515 3126 leader_election.cc:290] T 22f7db660616457b83b24e947a0b0bae P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:50.333664 1166 raft_consensus.cc:359] T 0582ee42ab564c07ae3facfc5935617c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.333838 1166 raft_consensus.cc:385] T 0582ee42ab564c07ae3facfc5935617c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.333859 771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e429b94326cc4827a03968fd92de38e0" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:50.333901 1166 raft_consensus.cc:740] T 0582ee42ab564c07ae3facfc5935617c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.333992 771 raft_consensus.cc:2468] T e429b94326cc4827a03968fd92de38e0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:50.334153 1166 consensus_queue.cc:260] T 0582ee42ab564c07ae3facfc5935617c P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.334784 3142 consensus_queue.cc:237] T aea8fcf0154a4bd4a05b19e27ab78b79 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.334964 1166 ts_tablet_manager.cc:1434] T 0582ee42ab564c07ae3facfc5935617c P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251025 07:52:50.335245 1166 tablet_bootstrap.cc:492] T 66134661be834f7b811d7927ab629812 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:50.336853 1166 tablet_bootstrap.cc:654] T 66134661be834f7b811d7927ab629812 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:50.336818 765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "48b73e0c0dce4be686d05f991a21bd19" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:50.336970 765 raft_consensus.cc:3060] T 48b73e0c0dce4be686d05f991a21bd19 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.338287 1043 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22f7db660616457b83b24e947a0b0bae" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:50.338682 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d40519ed9ee746348b17be2786d60e95" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:50.339856 2166 ts_tablet_manager.cc:1929] T 2e098c9c4613459295b6043048cba587 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:50.339946 2166 log.cc:1199] T 2e098c9c4613459295b6043048cba587 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/2e098c9c4613459295b6043048cba587
I20251025 07:52:50.340194 2166 ts_tablet_manager.cc:1950] T 2e098c9c4613459295b6043048cba587 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:50.340591 2169 ts_tablet_manager.cc:1929] T 7e6544202d4f489d82f22d6073ef240d P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:50.340683 2169 log.cc:1199] T 7e6544202d4f489d82f22d6073ef240d P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/7e6544202d4f489d82f22d6073ef240d
I20251025 07:52:50.340955 2169 ts_tablet_manager.cc:1950] T 7e6544202d4f489d82f22d6073ef240d P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
W20251025 07:52:50.343657 712 leader_election.cc:343] T 22f7db660616457b83b24e947a0b0bae P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:50.343941 712 leader_election.cc:343] T d40519ed9ee746348b17be2786d60e95 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.348402 765 raft_consensus.cc:2468] T 48b73e0c0dce4be686d05f991a21bd19 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:50.348676 3203 raft_consensus.cc:493] T 0582ee42ab564c07ae3facfc5935617c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:50.348860 3203 raft_consensus.cc:515] T 0582ee42ab564c07ae3facfc5935617c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.349467 3203 leader_election.cc:290] T 0582ee42ab564c07ae3facfc5935617c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:50.351483 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0582ee42ab564c07ae3facfc5935617c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:50.352394 850 leader_election.cc:343] T 0582ee42ab564c07ae3facfc5935617c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.352955 2169 tablet_replica.cc:333] T 4f0c4f0e8f014e59be9e46d2afe0be9d P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:50.354121 2169 raft_consensus.cc:2243] T 4f0c4f0e8f014e59be9e46d2afe0be9d P db42be7ad3c741a0aa427f27e245993b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:50.354489 2169 raft_consensus.cc:2272] T 4f0c4f0e8f014e59be9e46d2afe0be9d P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:50.355042 2169 ts_tablet_manager.cc:1916] T 4f0c4f0e8f014e59be9e46d2afe0be9d P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:50.354820 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 7e6544202d4f489d82f22d6073ef240d (table default.Table_8da73d66704143de8a4b4480c28fd1d9 [id=93fcfeca6dbf48baa3691b7939d0e4ec]) successfully deleted
W20251025 07:52:50.355252 850 consensus_peers.cc:597] T 9f039d9107ae493486d799ab6a1d731d P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9f039d9107ae493486d799ab6a1d731d. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:50.355073 1166 tablet_bootstrap.cc:492] T 66134661be834f7b811d7927ab629812 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
W20251025 07:52:50.355389 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet ffe7c8e1d6c44872b7b65e512764a56a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ffe7c8e1d6c44872b7b65e512764a56a already in progress: creating tablet
I20251025 07:52:50.355558 1166 ts_tablet_manager.cc:1403] T 66134661be834f7b811d7927ab629812 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.002s
I20251025 07:52:50.356395 1166 raft_consensus.cc:359] T 66134661be834f7b811d7927ab629812 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.356603 1166 raft_consensus.cc:385] T 66134661be834f7b811d7927ab629812 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.356705 1166 raft_consensus.cc:740] T 66134661be834f7b811d7927ab629812 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.356958 1166 consensus_queue.cc:260] T 66134661be834f7b811d7927ab629812 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.357527 1166 ts_tablet_manager.cc:1434] T 66134661be834f7b811d7927ab629812 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:50.357829 1166 tablet_bootstrap.cc:492] T 9903e80a26fd469a8744dc855534ba75 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:50.359725 1166 tablet_bootstrap.cc:654] T 9903e80a26fd469a8744dc855534ba75 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:50.362609 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 235) took 5790 ms. Trace:
I20251025 07:52:50.362782 2166 rpcz_store.cc:276] 1025 07:52:44.572538 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.573060 (+ 522us) service_pool.cc:225] Handling call
1025 07:52:50.240670 (+5667610us) ts_tablet_manager.cc:1151] Deleting tablet 2e098c9c4613459295b6043048cba587
1025 07:52:50.240682 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.335793 (+ 95111us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.339770 (+ 3977us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.362568 (+ 22798us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5667478}
I20251025 07:52:50.363379 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 2e098c9c4613459295b6043048cba587 (table default.Table_9f52d94037424befbdea5c711068396a [id=55440328803042a28df1b6869d279f59]) successfully deleted
I20251025 07:52:50.364235 2166 tablet_replica.cc:333] T 4f7c15529be54a11961f385a953b0b03 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:50.364493 2166 raft_consensus.cc:2243] T 4f7c15529be54a11961f385a953b0b03 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:50.364637 2166 raft_consensus.cc:2272] T 4f7c15529be54a11961f385a953b0b03 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:50.365137 2166 ts_tablet_manager.cc:1916] T 4f7c15529be54a11961f385a953b0b03 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:50.366685 2146 ts_tablet_manager.cc:1929] T 4b278ce7768349e49dcd1131e404ab34 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:50.366819 2146 log.cc:1199] T 4b278ce7768349e49dcd1131e404ab34 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/4b278ce7768349e49dcd1131e404ab34
I20251025 07:52:50.367157 2146 ts_tablet_manager.cc:1950] T 4b278ce7768349e49dcd1131e404ab34 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:50.367910 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 290) took 5681 ms. Trace:
I20251025 07:52:50.368021 2146 rpcz_store.cc:276] 1025 07:52:44.686320 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.686360 (+ 40us) service_pool.cc:225] Handling call
1025 07:52:50.297695 (+5611335us) ts_tablet_manager.cc:1151] Deleting tablet 4b278ce7768349e49dcd1131e404ab34
1025 07:52:50.297707 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.342743 (+ 45036us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.366175 (+ 23432us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.367871 (+ 1696us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24091,"tablet-delete.queue_time_us":5611041}
I20251025 07:52:50.368589 2146 tablet_replica.cc:333] T 9f039d9107ae493486d799ab6a1d731d P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:50.368811 2146 raft_consensus.cc:2243] T 9f039d9107ae493486d799ab6a1d731d P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:50.369148 2146 raft_consensus.cc:2272] T 9f039d9107ae493486d799ab6a1d731d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:50.369225 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 4b278ce7768349e49dcd1131e404ab34 (table default.Table_309913c03a4d46019d35478363f149e8 [id=bad99c3045fc4b938e20d2dcfdd0b1bc]) successfully deleted
I20251025 07:52:50.369721 2146 ts_tablet_manager.cc:1916] T 9f039d9107ae493486d799ab6a1d731d P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:50.380712 2169 ts_tablet_manager.cc:1929] T 4f0c4f0e8f014e59be9e46d2afe0be9d P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:50.380883 2169 log.cc:1199] T 4f0c4f0e8f014e59be9e46d2afe0be9d P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/4f0c4f0e8f014e59be9e46d2afe0be9d
I20251025 07:52:50.381546 1166 tablet_bootstrap.cc:492] T 9903e80a26fd469a8744dc855534ba75 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:50.381753 1166 ts_tablet_manager.cc:1403] T 9903e80a26fd469a8744dc855534ba75 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20251025 07:52:50.382730 1166 raft_consensus.cc:359] T 9903e80a26fd469a8744dc855534ba75 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.382974 1166 raft_consensus.cc:385] T 9903e80a26fd469a8744dc855534ba75 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.383049 1166 raft_consensus.cc:740] T 9903e80a26fd469a8744dc855534ba75 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.383399 1166 consensus_queue.cc:260] T 9903e80a26fd469a8744dc855534ba75 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.384145 1166 ts_tablet_manager.cc:1434] T 9903e80a26fd469a8744dc855534ba75 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:50.384518 1166 tablet_bootstrap.cc:492] T 1b22bd172e0e4f86896e9ecdef666b42 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:50.385042 2169 ts_tablet_manager.cc:1950] T 4f0c4f0e8f014e59be9e46d2afe0be9d P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:50.386112 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 4f0c4f0e8f014e59be9e46d2afe0be9d (table default.Table_c902adc24467448f9b5d4f585cf614f5 [id=56e169f79a37465d82dd1d1c3c285740]) successfully deleted
I20251025 07:52:50.386529 1166 tablet_bootstrap.cc:654] T 1b22bd172e0e4f86896e9ecdef666b42 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:50.387357 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 80fe3255ed614270bd8352a2853fbc43 because the tablet was not found. No further retry: Not found: Tablet not found: 80fe3255ed614270bd8352a2853fbc43
W20251025 07:52:50.387768 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 5b6383c99e9e4c90a374da9ccd830a6f because the tablet was not found. No further retry: Not found: Tablet not found: 5b6383c99e9e4c90a374da9ccd830a6f
I20251025 07:52:50.387908 2169 tablet_replica.cc:333] T fce535a0c0bd45c6bf17f81add297f40 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:50.388126 2169 raft_consensus.cc:2243] T fce535a0c0bd45c6bf17f81add297f40 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:50.388281 2169 raft_consensus.cc:2272] T fce535a0c0bd45c6bf17f81add297f40 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:50.388796 2169 ts_tablet_manager.cc:1916] T fce535a0c0bd45c6bf17f81add297f40 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:50.394563 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:46231, user_credentials={real_user=slave}} blocked reactor thread for 69926.2us
I20251025 07:52:50.395594 3203 consensus_queue.cc:1048] T 1c40b2e06d894a06af9e58285892ea8d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:50.396519 847 leader_election.cc:304] T 3aeaa0e9dc594e21a9ddd923e314017d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.396785 3203 raft_consensus.cc:2804] T 3aeaa0e9dc594e21a9ddd923e314017d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:50.396893 3203 raft_consensus.cc:493] T 3aeaa0e9dc594e21a9ddd923e314017d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:50.396976 3203 raft_consensus.cc:3060] T 3aeaa0e9dc594e21a9ddd923e314017d P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.398507 2146 ts_tablet_manager.cc:1929] T 9f039d9107ae493486d799ab6a1d731d P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:50.400177 3203 raft_consensus.cc:515] T 3aeaa0e9dc594e21a9ddd923e314017d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.406966 3203 leader_election.cc:290] T 3aeaa0e9dc594e21a9ddd923e314017d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:50.407217 2146 log.cc:1199] T 9f039d9107ae493486d799ab6a1d731d P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/9f039d9107ae493486d799ab6a1d731d
I20251025 07:52:50.407254 1043 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3aeaa0e9dc594e21a9ddd923e314017d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:50.407660 2146 ts_tablet_manager.cc:1950] T 9f039d9107ae493486d799ab6a1d731d P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
W20251025 07:52:50.407826 850 leader_election.cc:343] T 3aeaa0e9dc594e21a9ddd923e314017d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.408515 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 291) took 5709 ms. Trace:
I20251025 07:52:50.408628 2146 rpcz_store.cc:276] 1025 07:52:44.699238 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.699286 (+ 48us) service_pool.cc:225] Handling call
1025 07:52:50.368516 (+5669230us) ts_tablet_manager.cc:1151] Deleting tablet 9f039d9107ae493486d799ab6a1d731d
1025 07:52:50.368527 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.394721 (+ 26194us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.398375 (+ 3654us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.408450 (+ 10075us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5668562}
I20251025 07:52:50.409102 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 9f039d9107ae493486d799ab6a1d731d (table default.Table_3b10c71d2c1b4bd2849e0cad9ec767d7 [id=ee09c3f16a194bbf97dd2768b5231c06]) successfully deleted
I20251025 07:52:50.411753 2146 tablet_replica.cc:333] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:50.412220 2146 raft_consensus.cc:2243] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:50.412804 2146 pending_rounds.cc:70] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382: Trying to abort 1 pending ops.
I20251025 07:52:50.412887 2169 ts_tablet_manager.cc:1929] T fce535a0c0bd45c6bf17f81add297f40 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:50.413005 2169 log.cc:1199] T fce535a0c0bd45c6bf17f81add297f40 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/fce535a0c0bd45c6bf17f81add297f40
I20251025 07:52:50.413231 2146 pending_rounds.cc:77] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214607436690837504 op_type: NO_OP noop_request { }
I20251025 07:52:50.413561 2146 raft_consensus.cc:2889] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 07:52:50.413803 2146 raft_consensus.cc:2272] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:50.414851 2169 ts_tablet_manager.cc:1950] T fce535a0c0bd45c6bf17f81add297f40 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:50.415859 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet fce535a0c0bd45c6bf17f81add297f40 (table default.Table_1de0b151cf4c4c10aca35e45623caf2c [id=a947f21398254a53bc6668c4f8d39e53]) successfully deleted
I20251025 07:52:50.416411 2146 ts_tablet_manager.cc:1916] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:50.418807 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet ebc824ec4c60411dab1a2ea4a385148e because the tablet was not found. No further retry: Not found: Tablet not found: ebc824ec4c60411dab1a2ea4a385148e
W20251025 07:52:50.419217 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet a77eb82f26ae43b48de1e85693372d34 because the tablet was not found. No further retry: Not found: Tablet not found: a77eb82f26ae43b48de1e85693372d34
W20251025 07:52:50.419595 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet ce1ebb1ec14b4801930a72765bca1a7d because the tablet was not found. No further retry: Not found: Tablet not found: ce1ebb1ec14b4801930a72765bca1a7d
W20251025 07:52:50.419948 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 11bfe0c87ca941a3846462c8a4512d83 because the tablet was not found. No further retry: Not found: Tablet not found: 11bfe0c87ca941a3846462c8a4512d83
W20251025 07:52:50.420326 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 91728987303b41fd840eb56f3a311c5f because the tablet was not found. No further retry: Not found: Tablet not found: 91728987303b41fd840eb56f3a311c5f
W20251025 07:52:50.420701 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 050a5d3b5f934dd19bf2e221f2fe0780 because the tablet was not found. No further retry: Not found: Tablet not found: 050a5d3b5f934dd19bf2e221f2fe0780
I20251025 07:52:50.424397 2166 ts_tablet_manager.cc:1929] T 4f7c15529be54a11961f385a953b0b03 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:50.424561 2166 log.cc:1199] T 4f7c15529be54a11961f385a953b0b03 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/4f7c15529be54a11961f385a953b0b03
I20251025 07:52:50.424916 2166 ts_tablet_manager.cc:1950] T 4f7c15529be54a11961f385a953b0b03 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
W20251025 07:52:50.425625 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 36c8fcafd5464b4e999276a8062a812a because the tablet was not found. No further retry: Not found: Tablet not found: 36c8fcafd5464b4e999276a8062a812a
I20251025 07:52:50.425766 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 236) took 5853 ms. Trace:
I20251025 07:52:50.400866 847 leader_election.cc:304] T e429b94326cc4827a03968fd92de38e0 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:50.426012 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet aaef24ccdfba45fe8204327723df834e because the tablet was not found. No further retry: Not found: Tablet not found: aaef24ccdfba45fe8204327723df834e
I20251025 07:52:50.426306 3122 raft_consensus.cc:2804] T e429b94326cc4827a03968fd92de38e0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:50.426484 3122 raft_consensus.cc:493] T e429b94326cc4827a03968fd92de38e0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:50.426622 3122 raft_consensus.cc:3060] T e429b94326cc4827a03968fd92de38e0 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.427407 847 leader_election.cc:304] T 48b73e0c0dce4be686d05f991a21bd19 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.427812 3203 raft_consensus.cc:2804] T 48b73e0c0dce4be686d05f991a21bd19 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:50.427968 3203 raft_consensus.cc:697] T 48b73e0c0dce4be686d05f991a21bd19 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:50.428239 3203 consensus_queue.cc:237] T 48b73e0c0dce4be686d05f991a21bd19 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
W20251025 07:52:50.432718 850 consensus_peers.cc:597] T 5dab439bc50d4e869548154e1d222ad6 P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:50.433634 3011 raft_consensus.cc:493] T d77704b4fe6c4456a2ba40ffec0c60fc P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:50.433749 3011 raft_consensus.cc:515] T d77704b4fe6c4456a2ba40ffec0c60fc P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:50.434343 3011 leader_election.cc:290] T d77704b4fe6c4456a2ba40ffec0c60fc P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
W20251025 07:52:50.435967 983 leader_election.cc:343] T d77704b4fe6c4456a2ba40ffec0c60fc P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: d77704b4fe6c4456a2ba40ffec0c60fc
I20251025 07:52:50.436168 2146 ts_tablet_manager.cc:1929] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251025 07:52:50.437180 982 leader_election.cc:343] T d77704b4fe6c4456a2ba40ffec0c60fc P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: d77704b4fe6c4456a2ba40ffec0c60fc
I20251025 07:52:50.437278 982 leader_election.cc:304] T d77704b4fe6c4456a2ba40ffec0c60fc P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:50.437685 3011 raft_consensus.cc:2749] T d77704b4fe6c4456a2ba40ffec0c60fc P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:50.440596 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 0f5f6ca6c3cd4d369690e425387a4344 because the tablet was not found. No further retry: Not found: Tablet not found: 0f5f6ca6c3cd4d369690e425387a4344
W20251025 07:52:50.440910 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 4c155aa7fa5b4d68bbab1918b8432372 because the tablet was not found. No further retry: Not found: Tablet not found: 4c155aa7fa5b4d68bbab1918b8432372
W20251025 07:52:50.441148 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet d9bfd7c0c3ad4e5b98b6708793d61d41 because the tablet was not found. No further retry: Not found: Tablet not found: d9bfd7c0c3ad4e5b98b6708793d61d41
I20251025 07:52:50.441184 2169 tablet_replica.cc:333] T d77704b4fe6c4456a2ba40ffec0c60fc P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:50.441318 1169 tablet_bootstrap.cc:492] T dc4a788dca9d4a8ba40f8fdf1a4b915e P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
W20251025 07:52:50.441332 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 04449cc97f404a19b6a1b20dea074f95 because the tablet was not found. No further retry: Not found: Tablet not found: 04449cc97f404a19b6a1b20dea074f95
I20251025 07:52:50.441390 2169 raft_consensus.cc:2243] T d77704b4fe6c4456a2ba40ffec0c60fc P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:50.441437 1169 ts_tablet_manager.cc:1403] T dc4a788dca9d4a8ba40f8fdf1a4b915e P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.122s user 0.013s sys 0.017s
I20251025 07:52:50.441533 2169 raft_consensus.cc:2272] T d77704b4fe6c4456a2ba40ffec0c60fc P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:50.441565 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet f43ef51ee45d47a99ec732b49d50d3b4 because the tablet was not found. No further retry: Not found: Tablet not found: f43ef51ee45d47a99ec732b49d50d3b4
I20251025 07:52:50.441975 2169 ts_tablet_manager.cc:1916] T d77704b4fe6c4456a2ba40ffec0c60fc P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:50.442107 1169 raft_consensus.cc:359] T dc4a788dca9d4a8ba40f8fdf1a4b915e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.442277 1169 raft_consensus.cc:385] T dc4a788dca9d4a8ba40f8fdf1a4b915e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.442345 1169 raft_consensus.cc:740] T dc4a788dca9d4a8ba40f8fdf1a4b915e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.442534 1169 consensus_queue.cc:260] T dc4a788dca9d4a8ba40f8fdf1a4b915e P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.442981 1169 ts_tablet_manager.cc:1434] T dc4a788dca9d4a8ba40f8fdf1a4b915e P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251025 07:52:50.443269 1169 tablet_bootstrap.cc:492] T 67cf96a7dd3f42e19b86746c87e689f6 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:50.444895 1169 tablet_bootstrap.cc:654] T 67cf96a7dd3f42e19b86746c87e689f6 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:50.445611 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5]). Expected version 1 got 0
W20251025 07:52:50.439988 850 consensus_peers.cc:597] T ab42909afc524f2fa3c1041495d13a59 P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab42909afc524f2fa3c1041495d13a59. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:50.440543 1166 tablet_bootstrap.cc:492] T 1b22bd172e0e4f86896e9ecdef666b42 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:50.446343 1166 ts_tablet_manager.cc:1403] T 1b22bd172e0e4f86896e9ecdef666b42 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.062s user 0.007s sys 0.005s
I20251025 07:52:50.446795 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3]). Expected version 1 got 0
I20251025 07:52:50.447120 1166 raft_consensus.cc:359] T 1b22bd172e0e4f86896e9ecdef666b42 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.447294 1166 raft_consensus.cc:385] T 1b22bd172e0e4f86896e9ecdef666b42 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.447348 1166 raft_consensus.cc:740] T 1b22bd172e0e4f86896e9ecdef666b42 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.447572 1166 consensus_queue.cc:260] T 1b22bd172e0e4f86896e9ecdef666b42 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:50.441721 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet b947ecc352e54f4caa73c6ca97372256 because the tablet was not found. No further retry: Not found: Tablet not found: b947ecc352e54f4caa73c6ca97372256
W20251025 07:52:50.447980 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet f656945454fe4921ab819c772a089eb3 because the tablet was not found. No further retry: Not found: Tablet not found: f656945454fe4921ab819c772a089eb3
I20251025 07:52:50.448100 1166 ts_tablet_manager.cc:1434] T 1b22bd172e0e4f86896e9ecdef666b42 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:50.448372 1166 tablet_bootstrap.cc:492] T b71bdfd09a27449d98509bb220680147 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:50.449258 1013 tablet_service.cc:1505] Processing CreateTablet for tablet b53103dcdabb4d53aba641d1133155fc (DEFAULT_TABLE table=default.Table_438c3dc0bb6a45fa8e469ee9ab9138ae [id=86d6a0e55e1549769c6db7e58adf46cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:50.449604 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b53103dcdabb4d53aba641d1133155fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:50.449659 893 tablet_service.cc:1505] Processing CreateTablet for tablet b53103dcdabb4d53aba641d1133155fc (DEFAULT_TABLE table=default.Table_438c3dc0bb6a45fa8e469ee9ab9138ae [id=86d6a0e55e1549769c6db7e58adf46cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:50.449975 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b53103dcdabb4d53aba641d1133155fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:50.449621 710 leader_election.cc:304] T 3aeaa0e9dc594e21a9ddd923e314017d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.450435 755 tablet_service.cc:1505] Processing CreateTablet for tablet b53103dcdabb4d53aba641d1133155fc (DEFAULT_TABLE table=default.Table_438c3dc0bb6a45fa8e469ee9ab9138ae [id=86d6a0e55e1549769c6db7e58adf46cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:50.454643 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b53103dcdabb4d53aba641d1133155fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:50.455237 3142 raft_consensus.cc:2804] T 3aeaa0e9dc594e21a9ddd923e314017d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:50.455317 3142 raft_consensus.cc:493] T 3aeaa0e9dc594e21a9ddd923e314017d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:50.455389 3142 raft_consensus.cc:3060] T 3aeaa0e9dc594e21a9ddd923e314017d P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.458087 1166 tablet_bootstrap.cc:654] T b71bdfd09a27449d98509bb220680147 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:50.458271 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df9db9dd31d840e0b1394db04d662a62" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:50.458451 773 raft_consensus.cc:2468] T df9db9dd31d840e0b1394db04d662a62 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:50.458992 2146 log.cc:1199] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/29dbaf9149134f548a974a5779f7da67
I20251025 07:52:50.459475 3142 raft_consensus.cc:515] T 3aeaa0e9dc594e21a9ddd923e314017d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.460332 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3aeaa0e9dc594e21a9ddd923e314017d" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:50.460414 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee1c8ea694284d80a30f9c05be4cbf49" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:50.460590 773 raft_consensus.cc:3060] T ee1c8ea694284d80a30f9c05be4cbf49 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.460860 767 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3aeaa0e9dc594e21a9ddd923e314017d" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:50.461021 767 raft_consensus.cc:2393] T 3aeaa0e9dc594e21a9ddd923e314017d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3aea2869ff814cc7b142012faa274382 in current term 1: Already voted for candidate 23eeffd3de254ce18db5febfde971c6b in this term.
W20251025 07:52:50.461115 712 leader_election.cc:343] T 3aeaa0e9dc594e21a9ddd923e314017d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.461409 771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0582ee42ab564c07ae3facfc5935617c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:50.461556 771 raft_consensus.cc:2468] T 0582ee42ab564c07ae3facfc5935617c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:50.461572 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d40519ed9ee746348b17be2786d60e95" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:50.461726 910 raft_consensus.cc:3060] T d40519ed9ee746348b17be2786d60e95 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.461738 2146 ts_tablet_manager.cc:1950] T 29dbaf9149134f548a974a5779f7da67 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:50.463001 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 294) took 5648 ms. Trace:
I20251025 07:52:50.463207 773 raft_consensus.cc:2468] T ee1c8ea694284d80a30f9c05be4cbf49 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:50.463085 2146 rpcz_store.cc:276] 1025 07:52:44.814680 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.814919 (+ 239us) service_pool.cc:225] Handling call
1025 07:52:50.411667 (+5596748us) ts_tablet_manager.cc:1151] Deleting tablet 29dbaf9149134f548a974a5779f7da67
1025 07:52:50.411678 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.419653 (+ 7975us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.436037 (+ 16384us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.462981 (+ 26944us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5596560}
I20251025 07:52:50.463595 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 29dbaf9149134f548a974a5779f7da67 (table default.Table_c43ac1adc76d4ccf819411ec76775932 [id=106e3728b0b846178eb48ccbf076e24e]) successfully deleted
I20251025 07:52:50.464713 2146 tablet_replica.cc:333] T 11bfe0c87ca941a3846462c8a4512d83 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:50.464952 2146 raft_consensus.cc:2243] T 11bfe0c87ca941a3846462c8a4512d83 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:50.465277 2146 raft_consensus.cc:2272] T 11bfe0c87ca941a3846462c8a4512d83 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:50.465641 2166 rpcz_store.cc:276] 1025 07:52:44.572684 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.573215 (+ 531us) service_pool.cc:225] Handling call
1025 07:52:50.363471 (+5790256us) ts_tablet_manager.cc:1151] Deleting tablet 4f7c15529be54a11961f385a953b0b03
1025 07:52:50.364176 (+ 705us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.376523 (+ 12347us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.424187 (+ 47664us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.425747 (+ 1560us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5790117}
I20251025 07:52:50.466706 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 4f7c15529be54a11961f385a953b0b03 (table default.Table_06053a14dcc84e79917273c10b06b751 [id=c42ad6bf27aa4744b66b4462ac97e57d]) successfully deleted
I20251025 07:52:50.464685 910 raft_consensus.cc:2468] T d40519ed9ee746348b17be2786d60e95 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:50.467275 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22f7db660616457b83b24e947a0b0bae" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:50.467466 909 raft_consensus.cc:3060] T 22f7db660616457b83b24e947a0b0bae P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.469309 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_fa83d0528b4b481989bfc0ca35c6de44 [id=bf77b5ff6f1048dd996d948437f9ff5e]). Expected version 3 got 0
I20251025 07:52:50.469657 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 783487b60c144f719ed2eb3c67b5ccab (table default.Table_09095c0829ed4d1cb8b763fc695e7fa8 [id=e372ca4384644779a6b37f389f549dec]). Expected version 1 got 0
I20251025 07:52:50.470170 2146 ts_tablet_manager.cc:1916] T 11bfe0c87ca941a3846462c8a4512d83 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:50.470082 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f]). Expected version 1 got 0
I20251025 07:52:50.471110 909 raft_consensus.cc:2468] T 22f7db660616457b83b24e947a0b0bae P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:50.471292 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7]). Expected version 1 got 0
I20251025 07:52:50.471508 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 872cb7512d104827b1f1f186d33a4a44 (table default.Table_d5c65dce8b1b4214ba519d964012bfc0 [id=d4be274b36a845a99a4b96b8846248fd]). Expected version 1 got 0
I20251025 07:52:50.472378 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 6fe43ccecf7047319090ebd035fe24a5 (table default.Table_68b39cfaff194c3d8dbe9eaee7d8b44c [id=c95fc6d40893462eacee1276fb9cacfe]). Expected version 1 got 0
I20251025 07:52:50.466408 3142 leader_election.cc:290] T 3aeaa0e9dc594e21a9ddd923e314017d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:50.473023 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 5dab439bc50d4e869548154e1d222ad6 (table default.Table_fb9def2302784c999bf0f5707c60eb4c [id=45325c188eac4d34bb016d01de091f87]). Expected version 3 got 0
I20251025 07:52:50.473184 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3aeaa0e9dc594e21a9ddd923e314017d" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:50.473373 909 raft_consensus.cc:2393] T 3aeaa0e9dc594e21a9ddd923e314017d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 23eeffd3de254ce18db5febfde971c6b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:50.472856 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 237) took 5900 ms. Trace:
I20251025 07:52:50.474874 2166 rpcz_store.cc:276] 1025 07:52:44.572814 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.573374 (+ 560us) service_pool.cc:225] Handling call
1025 07:52:50.472721 (+5899347us) ts_tablet_manager.cc:1151] Deleting tablet 27651312cc2e42a8bfcd245f5d301679
1025 07:52:50.472736 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.472832 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5899220}
I20251025 07:52:50.479167 1166 tablet_bootstrap.cc:492] T b71bdfd09a27449d98509bb220680147 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:50.479370 1166 ts_tablet_manager.cc:1403] T b71bdfd09a27449d98509bb220680147 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.002s
I20251025 07:52:50.480320 1166 raft_consensus.cc:359] T b71bdfd09a27449d98509bb220680147 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.480528 1166 raft_consensus.cc:385] T b71bdfd09a27449d98509bb220680147 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.480597 1166 raft_consensus.cc:740] T b71bdfd09a27449d98509bb220680147 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.480581 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 233) took 5906 ms. Trace:
I20251025 07:52:50.480870 1166 consensus_queue.cc:260] T b71bdfd09a27449d98509bb220680147 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.481451 1166 ts_tablet_manager.cc:1434] T b71bdfd09a27449d98509bb220680147 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251025 07:52:50.480818 710 leader_election.cc:304] T d40519ed9ee746348b17be2786d60e95 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.481604 2166 rpcz_store.cc:276] 1025 07:52:44.572323 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.573648 (+ 1325us) service_pool.cc:225] Handling call
1025 07:52:50.479140 (+5905492us) ts_tablet_manager.cc:1151] Deleting tablet fab67b56eaa44b54886c17181d8c52b2
1025 07:52:50.479149 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.479235 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5905357}
I20251025 07:52:50.481961 710 leader_election.cc:304] T 22f7db660616457b83b24e947a0b0bae P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.482550 710 leader_election.cc:304] T 3aeaa0e9dc594e21a9ddd923e314017d P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.482972 3137 raft_consensus.cc:2804] T d40519ed9ee746348b17be2786d60e95 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:50.483203 3126 raft_consensus.cc:2804] T 22f7db660616457b83b24e947a0b0bae P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:50.483428 2888 raft_consensus.cc:2749] T 3aeaa0e9dc594e21a9ddd923e314017d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251025 07:52:50.483558 2888 consensus_queue.cc:1048] T 22fc90c704f64d2f9f4b8c54dee45576 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251025 07:52:50.486497 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 27651312cc2e42a8bfcd245f5d301679 because the tablet was not found. No further retry: Not found: Tablet not found: 27651312cc2e42a8bfcd245f5d301679
I20251025 07:52:50.490257 1167 tablet_bootstrap.cc:492] T 32a143a715cb485cb4501f4e28c68252 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:50.493325 2169 ts_tablet_manager.cc:1929] T d77704b4fe6c4456a2ba40ffec0c60fc P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:50.494247 1167 ts_tablet_manager.cc:1403] T 32a143a715cb485cb4501f4e28c68252 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.285s user 0.021s sys 0.007s
I20251025 07:52:50.495048 1167 raft_consensus.cc:359] T 32a143a715cb485cb4501f4e28c68252 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.495731 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 5fc092c2bb2d41ac8c5b4fc07460c8ec (table default.Table_cd20e202397645f8821dcde13b5c0ce3 [id=630c274339664b9db18ab4ef7c29c16f]). Expected version 2 got 0
I20251025 07:52:50.496372 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 3760348dbe284c41a01a4fe479a2103f (table default.Table_088aad2a46b041bfa29ed8e6929760b1 [id=834669c54dae449790818684fd8659bc]). Expected version 2 got 0
I20251025 07:52:50.496631 1167 raft_consensus.cc:385] T 32a143a715cb485cb4501f4e28c68252 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.496815 1167 raft_consensus.cc:740] T 32a143a715cb485cb4501f4e28c68252 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.497195 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]). Expected version 3 got 0
I20251025 07:52:50.497463 1167 consensus_queue.cc:260] T 32a143a715cb485cb4501f4e28c68252 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.498096 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 22f7db660616457b83b24e947a0b0bae (table default.Table_4a86191163f2408f9bcc795cd8d1bbf6 [id=bb316a7e000748ca902a9bfb494802d7]). Expected version 1 got 0
I20251025 07:52:50.498965 2146 ts_tablet_manager.cc:1929] T 11bfe0c87ca941a3846462c8a4512d83 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:50.499130 2146 log.cc:1199] T 11bfe0c87ca941a3846462c8a4512d83 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/11bfe0c87ca941a3846462c8a4512d83
I20251025 07:52:50.499545 2146 ts_tablet_manager.cc:1950] T 11bfe0c87ca941a3846462c8a4512d83 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:50.499977 1167 ts_tablet_manager.cc:1434] T 32a143a715cb485cb4501f4e28c68252 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20251025 07:52:50.501055 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]). Expected version 1 got 0
I20251025 07:52:50.502298 1167 tablet_bootstrap.cc:492] T ab68e5a89b234064b27aa7cae9467e14 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:50.503034 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 830fdbc6812e416690d0db42a73af503 (table default.Table_92824b3f2a774564b3aea04aaa25d204 [id=fb21a45b86ab446e8c3dae4512a9598f]). Expected version 1 got 0
I20251025 07:52:50.504496 1167 tablet_bootstrap.cc:654] T ab68e5a89b234064b27aa7cae9467e14 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:50.505126 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525]). Expected version 1 got 0
I20251025 07:52:50.505429 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 755f0d6c87044e20a8020f8410940123 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 4 got 0
I20251025 07:52:50.509013 3137 raft_consensus.cc:697] T d40519ed9ee746348b17be2786d60e95 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:50.509361 3137 consensus_queue.cc:237] T d40519ed9ee746348b17be2786d60e95 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.510373 3126 raft_consensus.cc:697] T 22f7db660616457b83b24e947a0b0bae P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:50.510699 3126 consensus_queue.cc:237] T 22f7db660616457b83b24e947a0b0bae P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.511273 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet e330e1b53bed4e7cace1c4fb7fa6b565 (table default.Table_a9026f5e095c4063b86d4c544c686e22 [id=7944225d3ce7439b881bfb9289d9f040]). Expected version 2 got 0
I20251025 07:52:50.511451 2169 log.cc:1199] T d77704b4fe6c4456a2ba40ffec0c60fc P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/d77704b4fe6c4456a2ba40ffec0c60fc
I20251025 07:52:50.511569 3011 raft_consensus.cc:493] T a7ab3150a35b4caca9119a1374d3f2f7 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:50.511600 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet f70091024c034c118fb86578db19a31c (table default.Table_b7da40b96d6c4359bed6f898e8cf6275 [id=e041946c093b4c789ce175c736cc230b]). Expected version 2 got 0
I20251025 07:52:50.511703 3011 raft_consensus.cc:515] T a7ab3150a35b4caca9119a1374d3f2f7 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:50.511883 2169 ts_tablet_manager.cc:1950] T d77704b4fe6c4456a2ba40ffec0c60fc P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:50.512147 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 3a0f50c8000144a4acb5219484b01800 (table default.Table_c62e0dc1ae504f9bb912013506ff3031 [id=6f395892b09240fea02482b3199cfb5e]). Expected version 1 got 0
I20251025 07:52:50.512311 3011 leader_election.cc:290] T a7ab3150a35b4caca9119a1374d3f2f7 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:50.512327 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236]). Expected version 1 got 0
I20251025 07:52:50.512444 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 8707b93d491e4e4386d148087a2f0870 (table default.Table_799dcb1cef2f4fe6b9b8f17f9f1026c5 [id=59840a567e5244e8a9024cbad42471e4]). Expected version 1 got 0
I20251025 07:52:50.512878 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet d77704b4fe6c4456a2ba40ffec0c60fc (table default.Table_e42a1f7287a34753b7e4daf36893d9c2 [id=ff115adf26ec4e499d44caa9b0366a49]) successfully deleted
I20251025 07:52:50.503979 1169 tablet_bootstrap.cc:492] T 67cf96a7dd3f42e19b86746c87e689f6 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:50.513099 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_67b99749f3bf43228b40d8b9807e69b4 [id=fc8180c09d7b4e74b2e161f7acb8fe0e]). Expected version 2 got 0
I20251025 07:52:50.513202 1169 ts_tablet_manager.cc:1403] T 67cf96a7dd3f42e19b86746c87e689f6 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.070s user 0.001s sys 0.003s
I20251025 07:52:50.513603 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c]). Expected version 1 got 0
I20251025 07:52:50.513912 1169 raft_consensus.cc:359] T 67cf96a7dd3f42e19b86746c87e689f6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.513993 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet b56209e4934e4407a8b4518ca4af0d46 (table default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de]). Expected version 1 got 0
I20251025 07:52:50.514086 1169 raft_consensus.cc:385] T 67cf96a7dd3f42e19b86746c87e689f6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251025 07:52:50.514113 983 leader_election.cc:343] T a7ab3150a35b4caca9119a1374d3f2f7 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Not found: Tablet not found: a7ab3150a35b4caca9119a1374d3f2f7
I20251025 07:52:50.514196 1169 raft_consensus.cc:740] T 67cf96a7dd3f42e19b86746c87e689f6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.514460 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7ab3150a35b4caca9119a1374d3f2f7" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:50.514518 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84]). Expected version 1 got 0
I20251025 07:52:50.514509 1169 consensus_queue.cc:260] T 67cf96a7dd3f42e19b86746c87e689f6 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.514762 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_d11bba62f65740319701b4337a5c255a [id=ad812862698d421184ebf501ce79b7be]). Expected version 3 got 0
I20251025 07:52:50.514979 1169 ts_tablet_manager.cc:1434] T 67cf96a7dd3f42e19b86746c87e689f6 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:50.515961 772 raft_consensus.cc:2393] T a7ab3150a35b4caca9119a1374d3f2f7 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:50.516439 982 leader_election.cc:304] T a7ab3150a35b4caca9119a1374d3f2f7 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:50.516474 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 288e543bb279410c91c01dd1623e110f (table default.Table_6e242ba55b194c23a6701f4b064fdbe5 [id=4c34b64b299f4162a53866cf01c5298f]). Expected version 1 got 0
I20251025 07:52:50.516789 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 8e6d76b1a97945cfb6c3055e1b622f8a (table default.Table_a692a40ea2ef44ffa5ab6ca9e29ef70a [id=f5c1be203e2b4831a025871cf8a2ef17]). Expected version 3 got 2
I20251025 07:52:50.516956 3011 raft_consensus.cc:3060] T a7ab3150a35b4caca9119a1374d3f2f7 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.519624 3011 raft_consensus.cc:2749] T a7ab3150a35b4caca9119a1374d3f2f7 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:50.520890 1167 tablet_bootstrap.cc:492] T ab68e5a89b234064b27aa7cae9467e14 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:50.521080 1167 ts_tablet_manager.cc:1403] T ab68e5a89b234064b27aa7cae9467e14 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.002s
I20251025 07:52:50.521713 3122 raft_consensus.cc:515] T e429b94326cc4827a03968fd92de38e0 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:50.521759 1167 raft_consensus.cc:359] T ab68e5a89b234064b27aa7cae9467e14 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.521942 1167 raft_consensus.cc:385] T ab68e5a89b234064b27aa7cae9467e14 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.522046 1167 raft_consensus.cc:740] T ab68e5a89b234064b27aa7cae9467e14 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.522356 1167 consensus_queue.cc:260] T ab68e5a89b234064b27aa7cae9467e14 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.522709 1169 tablet_bootstrap.cc:492] T 95efd553493e4966bc48a01cac6e1e10 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:50.522867 1167 ts_tablet_manager.cc:1434] T ab68e5a89b234064b27aa7cae9467e14 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251025 07:52:50.524076 1167 tablet_bootstrap.cc:492] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:50.524163 1169 tablet_bootstrap.cc:654] T 95efd553493e4966bc48a01cac6e1e10 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:50.525384 1167 tablet_bootstrap.cc:654] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:50.526752 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3]). Expected version 2 got 0
I20251025 07:52:50.528028 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e]). Expected version 1 got 0
I20251025 07:52:50.528283 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]). Expected version 1 got 0
I20251025 07:52:50.528631 446 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]). Expected version 1 got 0
W20251025 07:52:50.528817 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet fab67b56eaa44b54886c17181d8c52b2 because the tablet was not found. No further retry: Not found: Tablet not found: fab67b56eaa44b54886c17181d8c52b2
I20251025 07:52:50.529254 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 232) took 5957 ms. Trace:
I20251025 07:52:50.529376 2166 rpcz_store.cc:276] 1025 07:52:44.572213 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.573863 (+ 1650us) service_pool.cc:225] Handling call
1025 07:52:50.529159 (+5955296us) ts_tablet_manager.cc:1151] Deleting tablet 896bd7870d87408e8adfdb7a78998111
1025 07:52:50.529168 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.529237 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5955151}
W20251025 07:52:50.529899 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 896bd7870d87408e8adfdb7a78998111 because the tablet was not found. No further retry: Not found: Tablet not found: 896bd7870d87408e8adfdb7a78998111
I20251025 07:52:50.530247 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 240) took 5955 ms. Trace:
I20251025 07:52:50.530359 2166 rpcz_store.cc:276] 1025 07:52:44.574433 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.574600 (+ 167us) service_pool.cc:225] Handling call
1025 07:52:50.530171 (+5955571us) ts_tablet_manager.cc:1151] Deleting tablet 1912e59548874e0e8e989050ec0064c5
1025 07:52:50.530180 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.530232 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5955399}
W20251025 07:52:50.530570 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet e4da89f863cc43f3bf6df908bb9dc6bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4da89f863cc43f3bf6df908bb9dc6bf already in progress: creating tablet
W20251025 07:52:50.530802 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 1912e59548874e0e8e989050ec0064c5 because the tablet was not found. No further retry: Not found: Tablet not found: 1912e59548874e0e8e989050ec0064c5
I20251025 07:52:50.530963 2169 tablet_replica.cc:333] T e47c4f741f354a19b49777bd06e54764 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:50.531101 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 241) took 5956 ms. Trace:
I20251025 07:52:50.531172 2166 rpcz_store.cc:276] 1025 07:52:44.574558 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.574800 (+ 242us) service_pool.cc:225] Handling call
1025 07:52:50.531005 (+5956205us) ts_tablet_manager.cc:1151] Deleting tablet cc407c7eb12e4eadb41fb21385611c94
1025 07:52:50.531014 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.531087 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5956034}
I20251025 07:52:50.531247 2169 raft_consensus.cc:2243] T e47c4f741f354a19b49777bd06e54764 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:50.531535 2169 raft_consensus.cc:2272] T e47c4f741f354a19b49777bd06e54764 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:50.531553 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet cc407c7eb12e4eadb41fb21385611c94 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc407c7eb12e4eadb41fb21385611c94 already in progress: creating tablet
I20251025 07:52:50.531839 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 239) took 5957 ms. Trace:
I20251025 07:52:50.531945 2166 rpcz_store.cc:276] 1025 07:52:44.574321 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.575150 (+ 829us) service_pool.cc:225] Handling call
1025 07:52:50.531765 (+5956615us) ts_tablet_manager.cc:1151] Deleting tablet abf94c32ff2945119b710905c075d63c
1025 07:52:50.531773 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.531823 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5956478}
W20251025 07:52:50.532294 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet abf94c32ff2945119b710905c075d63c because the tablet was not found. No further retry: Not found: Tablet not found: abf94c32ff2945119b710905c075d63c
I20251025 07:52:50.532526 2169 ts_tablet_manager.cc:1916] T e47c4f741f354a19b49777bd06e54764 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:50.532578 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 238) took 5958 ms. Trace:
I20251025 07:52:50.532663 2166 rpcz_store.cc:276] 1025 07:52:44.574210 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.575372 (+ 1162us) service_pool.cc:225] Handling call
1025 07:52:50.532506 (+5957134us) ts_tablet_manager.cc:1151] Deleting tablet 887e7c7a5d0e4aaf88f26246306df19c
1025 07:52:50.532513 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.532564 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5956979}
W20251025 07:52:50.533025 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 887e7c7a5d0e4aaf88f26246306df19c because the tablet was not found. No further retry: Not found: Tablet not found: 887e7c7a5d0e4aaf88f26246306df19c
I20251025 07:52:50.533315 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 243) took 5957 ms. Trace:
I20251025 07:52:50.533391 2166 rpcz_store.cc:276] 1025 07:52:44.575902 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.575970 (+ 68us) service_pool.cc:225] Handling call
1025 07:52:50.533238 (+5957268us) ts_tablet_manager.cc:1151] Deleting tablet 6dcab53a020847a1aa27930265ba0205
1025 07:52:50.533248 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.533300 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5957107}
W20251025 07:52:50.533767 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 6dcab53a020847a1aa27930265ba0205 because the tablet was not found. No further retry: Not found: Tablet not found: 6dcab53a020847a1aa27930265ba0205
I20251025 07:52:50.534020 2166 tablet_replica.cc:333] T ee1c8ea694284d80a30f9c05be4cbf49 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:50.534237 2166 raft_consensus.cc:2243] T ee1c8ea694284d80a30f9c05be4cbf49 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:50.535382 2166 raft_consensus.cc:2272] T ee1c8ea694284d80a30f9c05be4cbf49 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:50.535933 2166 ts_tablet_manager.cc:1916] T ee1c8ea694284d80a30f9c05be4cbf49 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:50.542711 1169 tablet_bootstrap.cc:492] T 95efd553493e4966bc48a01cac6e1e10 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:50.545188 1167 tablet_bootstrap.cc:492] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:50.545331 1167 ts_tablet_manager.cc:1403] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.001s
I20251025 07:52:50.543018 1169 ts_tablet_manager.cc:1403] T 95efd553493e4966bc48a01cac6e1e10 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20251025 07:52:50.546432 1167 raft_consensus.cc:359] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.546705 1167 raft_consensus.cc:385] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.546805 1167 raft_consensus.cc:740] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.547070 1167 consensus_queue.cc:260] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.547408 2991 raft_consensus.cc:493] T 3e79d69d5d004cc395b8e35569e25440 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:50.547775 2991 raft_consensus.cc:515] T 3e79d69d5d004cc395b8e35569e25440 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.548147 1167 ts_tablet_manager.cc:1434] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251025 07:52:50.548300 2991 leader_election.cc:290] T 3e79d69d5d004cc395b8e35569e25440 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:50.548394 1167 tablet_bootstrap.cc:492] T 1b22bd172e0e4f86896e9ecdef666b42 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:50.548398 3122 leader_election.cc:290] T e429b94326cc4827a03968fd92de38e0 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:50.547079 1169 raft_consensus.cc:359] T 95efd553493e4966bc48a01cac6e1e10 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.548694 1169 raft_consensus.cc:385] T 95efd553493e4966bc48a01cac6e1e10 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.548808 1169 raft_consensus.cc:740] T 95efd553493e4966bc48a01cac6e1e10 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.549572 1169 consensus_queue.cc:260] T 95efd553493e4966bc48a01cac6e1e10 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.549675 1046 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e79d69d5d004cc395b8e35569e25440" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:50.549799 1167 tablet_bootstrap.cc:654] T 1b22bd172e0e4f86896e9ecdef666b42 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:50.548574 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e429b94326cc4827a03968fd92de38e0" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:50.550789 850 leader_election.cc:343] T 3e79d69d5d004cc395b8e35569e25440 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:50.550969 850 leader_election.cc:343] T e429b94326cc4827a03968fd92de38e0 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.551373 1166 tablet_bootstrap.cc:492] T 32a143a715cb485cb4501f4e28c68252 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
W20251025 07:52:50.547502 847 leader_election.cc:343] T 2e098c9c4613459295b6043048cba587 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Not found: Tablet not found: 2e098c9c4613459295b6043048cba587
I20251025 07:52:50.551867 847 leader_election.cc:304] T 2e098c9c4613459295b6043048cba587 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.556885 2169 ts_tablet_manager.cc:1929] T e47c4f741f354a19b49777bd06e54764 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:50.556999 2169 log.cc:1199] T e47c4f741f354a19b49777bd06e54764 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/e47c4f741f354a19b49777bd06e54764
I20251025 07:52:50.557300 2169 ts_tablet_manager.cc:1950] T e47c4f741f354a19b49777bd06e54764 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:50.558249 1169 ts_tablet_manager.cc:1434] T 95efd553493e4966bc48a01cac6e1e10 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
W20251025 07:52:50.558303 712 consensus_peers.cc:597] T 22fc90c704f64d2f9f4b8c54dee45576 P 23eeffd3de254ce18db5febfde971c6b -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:50.558382 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet e47c4f741f354a19b49777bd06e54764 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]) successfully deleted
I20251025 07:52:50.559216 1169 tablet_bootstrap.cc:492] T 288e543bb279410c91c01dd1623e110f P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:50.560715 1169 tablet_bootstrap.cc:654] T 288e543bb279410c91c01dd1623e110f P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:50.561506 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 4f7c15529be54a11961f385a953b0b03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f7c15529be54a11961f385a953b0b03 already in progress: creating tablet
I20251025 07:52:50.561859 2169 tablet_replica.cc:333] T 95efd553493e4966bc48a01cac6e1e10 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:50.553079 1166 tablet_bootstrap.cc:654] T 32a143a715cb485cb4501f4e28c68252 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:50.553663 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 296) took 5734 ms. Trace:
I20251025 07:52:50.562114 2169 raft_consensus.cc:2243] T 95efd553493e4966bc48a01cac6e1e10 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:50.562013 2146 rpcz_store.cc:276] 1025 07:52:44.819448 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.819510 (+ 62us) service_pool.cc:225] Handling call
1025 07:52:50.464637 (+5645127us) ts_tablet_manager.cc:1151] Deleting tablet 11bfe0c87ca941a3846462c8a4512d83
1025 07:52:50.464647 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.483163 (+ 18516us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.498863 (+ 15700us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.553643 (+ 54780us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5644970}
I20251025 07:52:50.562278 2169 raft_consensus.cc:2272] T 95efd553493e4966bc48a01cac6e1e10 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:50.562481 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 300) took 5110 ms. Trace:
I20251025 07:52:50.562565 2146 rpcz_store.cc:276] 1025 07:52:45.451550 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.451622 (+ 72us) service_pool.cc:225] Handling call
1025 07:52:50.562377 (+5110755us) ts_tablet_manager.cc:1151] Deleting tablet 6ec421eb820b4f40afcff5a5bf80c508
1025 07:52:50.562390 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.562463 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5110409}
I20251025 07:52:50.562896 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 298) took 5113 ms. Trace:
I20251025 07:52:50.562969 2169 ts_tablet_manager.cc:1916] T 95efd553493e4966bc48a01cac6e1e10 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:50.562978 2146 rpcz_store.cc:276] 1025 07:52:45.449782 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.451815 (+ 2033us) service_pool.cc:225] Handling call
1025 07:52:50.562802 (+5110987us) ts_tablet_manager.cc:1151] Deleting tablet 5361f08f50b745eb98bf02e979d7c71e
1025 07:52:50.562813 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.562879 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5110813}
I20251025 07:52:50.563304 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 302) took 5111 ms. Trace:
I20251025 07:52:50.563316 3137 raft_consensus.cc:493] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:50.563377 2146 rpcz_store.cc:276] 1025 07:52:45.451850 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.451901 (+ 51us) service_pool.cc:225] Handling call
1025 07:52:50.563203 (+5111302us) ts_tablet_manager.cc:1151] Deleting tablet 7fbae8be73bf4e18b5a5c4ec4cdebc12
1025 07:52:50.563212 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.563288 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5111130}
I20251025 07:52:50.563581 3137 raft_consensus.cc:515] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:50.563684 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 299) took 5113 ms. Trace:
I20251025 07:52:50.563756 2146 rpcz_store.cc:276] 1025 07:52:45.449962 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.452008 (+ 2046us) service_pool.cc:225] Handling call
1025 07:52:50.563612 (+5111604us) ts_tablet_manager.cc:1151] Deleting tablet 53198283718d4b45b30df65b5bde96a6
1025 07:52:50.563622 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.563669 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5111469}
I20251025 07:52:50.564039 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 301) took 5112 ms. Trace:
I20251025 07:52:50.564107 2146 rpcz_store.cc:276] 1025 07:52:45.451699 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.452170 (+ 471us) service_pool.cc:225] Handling call
1025 07:52:50.563970 (+5111800us) ts_tablet_manager.cc:1151] Deleting tablet 551d07a9242740dcb9366bb2d707a4a8
1025 07:52:50.563980 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.564026 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5111678}
I20251025 07:52:50.564373 2146 tablet_replica.cc:333] T 0551e5d694894e8fbeb66b5e0c1cd75b P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:50.564437 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d3c9954693d4e8782ee0128ef066aa9" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:50.564602 1048 raft_consensus.cc:2468] T 6d3c9954693d4e8782ee0128ef066aa9 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:50.564621 2146 raft_consensus.cc:2243] T 0551e5d694894e8fbeb66b5e0c1cd75b P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:50.564808 2146 raft_consensus.cc:2272] T 0551e5d694894e8fbeb66b5e0c1cd75b P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:50.564972 712 leader_election.cc:304] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:50.565197 3236 raft_consensus.cc:2804] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 07:52:50.565260 3236 raft_consensus.cc:493] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:50.565320 3236 raft_consensus.cc:3060] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Advancing to term 2
I20251025 07:52:50.565469 3137 leader_election.cc:290] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:50.565465 2146 ts_tablet_manager.cc:1916] T 0551e5d694894e8fbeb66b5e0c1cd75b P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:50.565862 2686 raft_consensus.cc:2749] T 2e098c9c4613459295b6043048cba587 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251025 07:52:50.566758 1166 tablet_bootstrap.cc:492] T 32a143a715cb485cb4501f4e28c68252 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:50.566763 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 11bfe0c87ca941a3846462c8a4512d83 (table default.Table_00b3b394e5ed46faaac0202afe54b437 [id=304ef5bba5ed4e198622b8c8dc17f43a]) successfully deleted
I20251025 07:52:50.566865 1166 ts_tablet_manager.cc:1403] T 32a143a715cb485cb4501f4e28c68252 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.001s
W20251025 07:52:50.567021 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 6ec421eb820b4f40afcff5a5bf80c508 because the tablet was not found. No further retry: Not found: Tablet not found: 6ec421eb820b4f40afcff5a5bf80c508
W20251025 07:52:50.567231 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 5361f08f50b745eb98bf02e979d7c71e because the tablet was not found. No further retry: Not found: Tablet not found: 5361f08f50b745eb98bf02e979d7c71e
W20251025 07:52:50.567529 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 53198283718d4b45b30df65b5bde96a6 because the tablet was not found. No further retry: Not found: Tablet not found: 53198283718d4b45b30df65b5bde96a6
I20251025 07:52:50.567605 1166 raft_consensus.cc:359] T 32a143a715cb485cb4501f4e28c68252 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.567744 1166 raft_consensus.cc:385] T 32a143a715cb485cb4501f4e28c68252 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.567795 1166 raft_consensus.cc:740] T 32a143a715cb485cb4501f4e28c68252 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
W20251025 07:52:50.567790 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 551d07a9242740dcb9366bb2d707a4a8 because the tablet was not found. No further retry: Not found: Tablet not found: 551d07a9242740dcb9366bb2d707a4a8
I20251025 07:52:50.567981 1166 consensus_queue.cc:260] T 32a143a715cb485cb4501f4e28c68252 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.568123 3236 raft_consensus.cc:515] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:50.568486 1166 ts_tablet_manager.cc:1434] T 32a143a715cb485cb4501f4e28c68252 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 07:52:50.568760 1166 tablet_bootstrap.cc:492] T 3c4970c8a491460e8b4b1c29fde83e4c P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:50.568874 847 leader_election.cc:304] T df9db9dd31d840e0b1394db04d662a62 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.569139 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d3c9954693d4e8782ee0128ef066aa9" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:50.569296 1048 raft_consensus.cc:3060] T 6d3c9954693d4e8782ee0128ef066aa9 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Advancing to term 2
I20251025 07:52:50.570279 1166 tablet_bootstrap.cc:654] T 3c4970c8a491460e8b4b1c29fde83e4c P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:50.571623 2686 raft_consensus.cc:2804] T df9db9dd31d840e0b1394db04d662a62 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:50.571688 2686 raft_consensus.cc:493] T df9db9dd31d840e0b1394db04d662a62 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:50.571741 2686 raft_consensus.cc:3060] T df9db9dd31d840e0b1394db04d662a62 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.572842 1048 raft_consensus.cc:2468] T 6d3c9954693d4e8782ee0128ef066aa9 P db42be7ad3c741a0aa427f27e245993b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 2.
I20251025 07:52:50.573357 712 leader_election.cc:304] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:50.574069 3236 leader_election.cc:290] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:50.574254 2146 ts_tablet_manager.cc:1929] T 0551e5d694894e8fbeb66b5e0c1cd75b P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:50.574321 3236 raft_consensus.cc:2804] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [term 2 FOLLOWER]: Leader election won for term 2
I20251025 07:52:50.574479 3236 raft_consensus.cc:697] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [term 2 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:50.574831 3236 consensus_queue.cc:237] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:50.574332 2686 raft_consensus.cc:515] T df9db9dd31d840e0b1394db04d662a62 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.581897 2166 ts_tablet_manager.cc:1929] T ee1c8ea694284d80a30f9c05be4cbf49 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:50.582016 2166 log.cc:1199] T ee1c8ea694284d80a30f9c05be4cbf49 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/ee1c8ea694284d80a30f9c05be4cbf49
I20251025 07:52:50.582299 2166 ts_tablet_manager.cc:1950] T ee1c8ea694284d80a30f9c05be4cbf49 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:50.582263 2686 leader_election.cc:290] T df9db9dd31d840e0b1394db04d662a62 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:50.576018 2991 consensus_queue.cc:1048] T 075f0a151e024220b51ba7fcaecdcc3d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:50.576224 847 leader_election.cc:304] T 3aeaa0e9dc594e21a9ddd923e314017d P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aea2869ff814cc7b142012faa274382; no voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.583227 2146 log.cc:1199] T 0551e5d694894e8fbeb66b5e0c1cd75b P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/0551e5d694894e8fbeb66b5e0c1cd75b
I20251025 07:52:50.583321 2991 raft_consensus.cc:2749] T 3aeaa0e9dc594e21a9ddd923e314017d P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251025 07:52:50.583324 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df9db9dd31d840e0b1394db04d662a62" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:50.583698 2146 ts_tablet_manager.cc:1950] T 0551e5d694894e8fbeb66b5e0c1cd75b P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:50.583806 2991 consensus_queue.cc:1048] T 2d24e3e7a9e4431db609331ad1fd57c9 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251025 07:52:50.583998 850 leader_election.cc:343] T df9db9dd31d840e0b1394db04d662a62 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.584477 847 leader_election.cc:304] T 0582ee42ab564c07ae3facfc5935617c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.584723 2991 raft_consensus.cc:2804] T 0582ee42ab564c07ae3facfc5935617c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:50.584805 2991 raft_consensus.cc:493] T 0582ee42ab564c07ae3facfc5935617c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:50.584888 2991 raft_consensus.cc:3060] T 0582ee42ab564c07ae3facfc5935617c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.585932 1167 tablet_bootstrap.cc:492] T 1b22bd172e0e4f86896e9ecdef666b42 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:50.586103 1167 ts_tablet_manager.cc:1403] T 1b22bd172e0e4f86896e9ecdef666b42 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.038s user 0.000s sys 0.003s
I20251025 07:52:50.586803 1167 raft_consensus.cc:359] T 1b22bd172e0e4f86896e9ecdef666b42 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.587013 1167 raft_consensus.cc:385] T 1b22bd172e0e4f86896e9ecdef666b42 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.587097 1167 raft_consensus.cc:740] T 1b22bd172e0e4f86896e9ecdef666b42 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.587612 2991 raft_consensus.cc:515] T 0582ee42ab564c07ae3facfc5935617c P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.587350 1167 consensus_queue.cc:260] T 1b22bd172e0e4f86896e9ecdef666b42 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.588157 2991 leader_election.cc:290] T 0582ee42ab564c07ae3facfc5935617c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:50.588310 1167 ts_tablet_manager.cc:1434] T 1b22bd172e0e4f86896e9ecdef666b42 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:50.588948 1169 tablet_bootstrap.cc:492] T 288e543bb279410c91c01dd1623e110f P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:50.589083 1169 ts_tablet_manager.cc:1403] T 288e543bb279410c91c01dd1623e110f P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.002s
I20251025 07:52:50.589195 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0582ee42ab564c07ae3facfc5935617c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:50.589210 847 leader_election.cc:304] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.589422 2991 raft_consensus.cc:2804] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:50.589570 2991 raft_consensus.cc:697] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
W20251025 07:52:50.589654 850 leader_election.cc:343] T 0582ee42ab564c07ae3facfc5935617c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.589803 2991 consensus_queue.cc:237] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.591378 1169 raft_consensus.cc:359] T 288e543bb279410c91c01dd1623e110f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.591539 1169 raft_consensus.cc:385] T 288e543bb279410c91c01dd1623e110f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.591601 1169 raft_consensus.cc:740] T 288e543bb279410c91c01dd1623e110f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.592203 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 242) took 6016 ms. Trace:
I20251025 07:52:50.592286 2166 rpcz_store.cc:276] 1025 07:52:44.575742 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.576218 (+ 476us) service_pool.cc:225] Handling call
1025 07:52:50.533980 (+5957762us) ts_tablet_manager.cc:1151] Deleting tablet ee1c8ea694284d80a30f9c05be4cbf49
1025 07:52:50.533988 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.562944 (+ 28956us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.581793 (+ 18849us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.592184 (+ 10391us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5957630}
I20251025 07:52:50.592628 1167 tablet_bootstrap.cc:492] T 7e18611cbc1348eca927162fec70e36f P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:50.592765 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet ee1c8ea694284d80a30f9c05be4cbf49 (table default.Table_e8d761a95cdf469b9cba89359c58a5a4 [id=b369bec99e4d48c1847a2a327b1d6ac8]) successfully deleted
I20251025 07:52:50.594537 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 244) took 6016 ms. Trace:
I20251025 07:52:50.594767 1169 consensus_queue.cc:260] T 288e543bb279410c91c01dd1623e110f P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.594584 1167 tablet_bootstrap.cc:654] T 7e18611cbc1348eca927162fec70e36f P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:50.594991 2166 rpcz_store.cc:276] 1025 07:52:44.578564 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.578609 (+ 45us) service_pool.cc:225] Handling call
1025 07:52:50.594444 (+6015835us) ts_tablet_manager.cc:1151] Deleting tablet 4f25c9caa05b4da4bd56a8a21fde423c
1025 07:52:50.594455 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.594520 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5998046}
I20251025 07:52:50.595263 3122 consensus_queue.cc:1048] T 5dab439bc50d4e869548154e1d222ad6 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:50.602797 1169 ts_tablet_manager.cc:1434] T 288e543bb279410c91c01dd1623e110f P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
W20251025 07:52:50.603303 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 4f25c9caa05b4da4bd56a8a21fde423c because the tablet was not found. No further retry: Not found: Tablet not found: 4f25c9caa05b4da4bd56a8a21fde423c
I20251025 07:52:50.603714 2166 tablet_replica.cc:333] T dda0b26e4f1a44928bc4b84f333901c0 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:50.603984 2166 raft_consensus.cc:2243] T dda0b26e4f1a44928bc4b84f333901c0 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:50.604233 2169 ts_tablet_manager.cc:1929] T 95efd553493e4966bc48a01cac6e1e10 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:50.604352 2169 log.cc:1199] T 95efd553493e4966bc48a01cac6e1e10 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/95efd553493e4966bc48a01cac6e1e10
I20251025 07:52:50.604652 2169 ts_tablet_manager.cc:1950] T 95efd553493e4966bc48a01cac6e1e10 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:50.611660 1166 tablet_bootstrap.cc:492] T 3c4970c8a491460e8b4b1c29fde83e4c P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:50.611788 1166 ts_tablet_manager.cc:1403] T 3c4970c8a491460e8b4b1c29fde83e4c P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.005s
I20251025 07:52:50.612532 1166 raft_consensus.cc:359] T 3c4970c8a491460e8b4b1c29fde83e4c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.612682 1166 raft_consensus.cc:385] T 3c4970c8a491460e8b4b1c29fde83e4c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.612725 1166 raft_consensus.cc:740] T 3c4970c8a491460e8b4b1c29fde83e4c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.612921 1166 consensus_queue.cc:260] T 3c4970c8a491460e8b4b1c29fde83e4c P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.613391 1166 ts_tablet_manager.cc:1434] T 3c4970c8a491460e8b4b1c29fde83e4c P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:50.614017 1166 tablet_bootstrap.cc:492] T cc407c7eb12e4eadb41fb21385611c94 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:50.611359 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 303) took 4980 ms. Trace:
I20251025 07:52:50.614826 2146 rpcz_store.cc:276] 1025 07:52:45.630634 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.630843 (+ 209us) service_pool.cc:225] Handling call
1025 07:52:50.564322 (+4933479us) ts_tablet_manager.cc:1151] Deleting tablet 0551e5d694894e8fbeb66b5e0c1cd75b
1025 07:52:50.564331 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.571142 (+ 6811us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.574165 (+ 3023us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.611328 (+ 37163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24211,"tablet-delete.queue_time_us":4933303}
I20251025 07:52:50.616808 1169 tablet_bootstrap.cc:492] T ed243540f2d746b980c25687fbca7414 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:50.616062 1166 tablet_bootstrap.cc:654] T cc407c7eb12e4eadb41fb21385611c94 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:50.617512 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 0551e5d694894e8fbeb66b5e0c1cd75b (table default.Table_2cd70245aa3d4809a0647e7fa499f62f [id=a003268ec3da425989fa4041699de4d3]) successfully deleted
I20251025 07:52:50.618356 1169 tablet_bootstrap.cc:654] T ed243540f2d746b980c25687fbca7414 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:50.619352 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 304) took 4969 ms. Trace:
I20251025 07:52:50.619437 2146 rpcz_store.cc:276] 1025 07:52:45.649719 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.649846 (+ 127us) service_pool.cc:225] Handling call
1025 07:52:50.619201 (+4969355us) ts_tablet_manager.cc:1151] Deleting tablet 110a8e9e4c6c434284c0ae14dd6987f7
1025 07:52:50.619214 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.619335 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4969173}
W20251025 07:52:50.619877 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 110a8e9e4c6c434284c0ae14dd6987f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 110a8e9e4c6c434284c0ae14dd6987f7 already in progress: creating tablet
I20251025 07:52:50.621461 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 305) took 4952 ms. Trace:
I20251025 07:52:50.621537 2146 rpcz_store.cc:276] 1025 07:52:45.669467 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.669540 (+ 73us) service_pool.cc:225] Handling call
1025 07:52:50.621354 (+4951814us) ts_tablet_manager.cc:1151] Deleting tablet 623dec4eab1d4666baf2334a01eeb36d
1025 07:52:50.621362 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.621446 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4951628}
W20251025 07:52:50.622044 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 623dec4eab1d4666baf2334a01eeb36d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 623dec4eab1d4666baf2334a01eeb36d already in progress: creating tablet
I20251025 07:52:50.624502 2146 tablet_replica.cc:333] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:50.624702 2146 raft_consensus.cc:2243] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:50.628666 2146 raft_consensus.cc:2272] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:50.629705 2146 ts_tablet_manager.cc:1916] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:50.633556 1166 tablet_bootstrap.cc:492] T cc407c7eb12e4eadb41fb21385611c94 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:50.633677 1166 ts_tablet_manager.cc:1403] T cc407c7eb12e4eadb41fb21385611c94 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20251025 07:52:50.633708 3122 consensus_queue.cc:1048] T df0d86ad301245c1a6bcb44805936d25 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:50.629818 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d3c9954693d4e8782ee0128ef066aa9" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:50.629969 3202 raft_consensus.cc:493] T ab68e5a89b234064b27aa7cae9467e14 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:50.634271 902 raft_consensus.cc:2468] T 6d3c9954693d4e8782ee0128ef066aa9 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:50.634347 3202 raft_consensus.cc:515] T ab68e5a89b234064b27aa7cae9467e14 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.630036 905 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d3c9954693d4e8782ee0128ef066aa9" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:50.634740 905 raft_consensus.cc:3060] T 6d3c9954693d4e8782ee0128ef066aa9 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Advancing to term 2
I20251025 07:52:50.635097 1166 raft_consensus.cc:359] T cc407c7eb12e4eadb41fb21385611c94 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.635239 1166 raft_consensus.cc:385] T cc407c7eb12e4eadb41fb21385611c94 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.630807 2991 consensus_queue.cc:1048] T d46673a283fe42b78ddeb99fbd2678eb P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:50.635289 1166 raft_consensus.cc:740] T cc407c7eb12e4eadb41fb21385611c94 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.635460 1166 consensus_queue.cc:260] T cc407c7eb12e4eadb41fb21385611c94 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.635635 3202 leader_election.cc:290] T ab68e5a89b234064b27aa7cae9467e14 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:50.635948 1166 ts_tablet_manager.cc:1434] T cc407c7eb12e4eadb41fb21385611c94 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:50.636180 1166 tablet_bootstrap.cc:492] T 2c4789b9d89f42cb844e6cb595e687db P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:50.637737 1166 tablet_bootstrap.cc:654] T 2c4789b9d89f42cb844e6cb595e687db P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:50.647111 905 raft_consensus.cc:2468] T 6d3c9954693d4e8782ee0128ef066aa9 P 3aea2869ff814cc7b142012faa274382 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 2.
I20251025 07:52:50.614554 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e429b94326cc4827a03968fd92de38e0" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:50.620051 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df9db9dd31d840e0b1394db04d662a62" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:50.620468 765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e79d69d5d004cc395b8e35569e25440" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:50.668749 772 raft_consensus.cc:3060] T e429b94326cc4827a03968fd92de38e0 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.668859 765 raft_consensus.cc:2468] T 3e79d69d5d004cc395b8e35569e25440 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:50.621111 764 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0582ee42ab564c07ae3facfc5935617c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:50.639256 1167 tablet_bootstrap.cc:492] T 7e18611cbc1348eca927162fec70e36f P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:50.669173 764 raft_consensus.cc:3060] T 0582ee42ab564c07ae3facfc5935617c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.669245 1167 ts_tablet_manager.cc:1403] T 7e18611cbc1348eca927162fec70e36f P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.077s user 0.005s sys 0.003s
I20251025 07:52:50.669932 1167 raft_consensus.cc:359] T 7e18611cbc1348eca927162fec70e36f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.670079 1167 raft_consensus.cc:385] T 7e18611cbc1348eca927162fec70e36f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251025 07:52:50.643586 710 consensus_peers.cc:597] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0551e5d694894e8fbeb66b5e0c1cd75b. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:50.644785 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ab68e5a89b234064b27aa7cae9467e14" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:50.647403 3137 raft_consensus.cc:493] T 32a143a715cb485cb4501f4e28c68252 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:50.670253 774 raft_consensus.cc:2468] T ab68e5a89b234064b27aa7cae9467e14 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:50.670288 3137 raft_consensus.cc:515] T 32a143a715cb485cb4501f4e28c68252 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.670123 1167 raft_consensus.cc:740] T 7e18611cbc1348eca927162fec70e36f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.670616 1167 consensus_queue.cc:260] T 7e18611cbc1348eca927162fec70e36f P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.670835 3137 leader_election.cc:290] T 32a143a715cb485cb4501f4e28c68252 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:50.668778 773 raft_consensus.cc:3060] T df9db9dd31d840e0b1394db04d662a62 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.671559 1167 ts_tablet_manager.cc:1434] T 7e18611cbc1348eca927162fec70e36f P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:50.671927 1167 tablet_bootstrap.cc:492] T 6fb1a1d38fb94f07b79ac4f8334fa719 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:50.672857 772 raft_consensus.cc:2468] T e429b94326cc4827a03968fd92de38e0 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:50.673269 2166 raft_consensus.cc:2272] T dda0b26e4f1a44928bc4b84f333901c0 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:50.673437 773 raft_consensus.cc:2468] T df9db9dd31d840e0b1394db04d662a62 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:50.673358 1167 tablet_bootstrap.cc:654] T 6fb1a1d38fb94f07b79ac4f8334fa719 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:50.674926 2166 ts_tablet_manager.cc:1916] T dda0b26e4f1a44928bc4b84f333901c0 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:50.676066 764 raft_consensus.cc:2468] T 0582ee42ab564c07ae3facfc5935617c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:50.678494 3142 raft_consensus.cc:493] T 1b22bd172e0e4f86896e9ecdef666b42 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:50.678660 3142 raft_consensus.cc:515] T 1b22bd172e0e4f86896e9ecdef666b42 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.679268 3142 leader_election.cc:290] T 1b22bd172e0e4f86896e9ecdef666b42 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:50.679760 447 catalog_manager.cc:5649] T d46673a283fe42b78ddeb99fbd2678eb P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } }
I20251025 07:52:50.680505 1042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ab68e5a89b234064b27aa7cae9467e14" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:50.680886 1040 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32a143a715cb485cb4501f4e28c68252" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:50.681133 1047 raft_consensus.cc:3060] T dc4a788dca9d4a8ba40f8fdf1a4b915e P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:50.681347 712 leader_election.cc:343] T 32a143a715cb485cb4501f4e28c68252 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:50.681475 850 leader_election.cc:343] T ab68e5a89b234064b27aa7cae9467e14 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:50.681667 850 consensus_peers.cc:597] T 075f0a151e024220b51ba7fcaecdcc3d P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251025 07:52:50.685572 850 consensus_peers.cc:597] T df0d86ad301245c1a6bcb44805936d25 P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:50.685624 1039 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b22bd172e0e4f86896e9ecdef666b42" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:50.686170 712 leader_election.cc:343] T 1b22bd172e0e4f86896e9ecdef666b42 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.686882 447 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_fa83d0528b4b481989bfc0ca35c6de44 [id=bf77b5ff6f1048dd996d948437f9ff5e]). Expected version 3 got 0
I20251025 07:52:50.687235 1039 raft_consensus.cc:3060] T 288e543bb279410c91c01dd1623e110f P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.688061 447 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3]). Expected version 1 got 0
I20251025 07:52:50.688803 447 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5]). Expected version 1 got 0
I20251025 07:52:50.689497 447 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 783487b60c144f719ed2eb3c67b5ccab (table default.Table_09095c0829ed4d1cb8b763fc695e7fa8 [id=e372ca4384644779a6b37f389f549dec]). Expected version 1 got 0
I20251025 07:52:50.690474 447 catalog_manager.cc:5649] T 72618756f756484e895e885ef12545c8 P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } }
I20251025 07:52:50.691947 447 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f]). Expected version 1 got 0
I20251025 07:52:50.693357 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32a143a715cb485cb4501f4e28c68252" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:50.693532 909 raft_consensus.cc:2468] T 32a143a715cb485cb4501f4e28c68252 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:50.694042 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b22bd172e0e4f86896e9ecdef666b42" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:50.694259 910 raft_consensus.cc:2468] T 1b22bd172e0e4f86896e9ecdef666b42 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:50.704942 2146 ts_tablet_manager.cc:1929] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 07:52:50.705415 447 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7]). Expected version 1 got 0
I20251025 07:52:50.710273 2146 log.cc:1199] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/cb829649f73d4d3f8862d268c953bd86
I20251025 07:52:50.710996 447 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 872cb7512d104827b1f1f186d33a4a44 (table default.Table_d5c65dce8b1b4214ba519d964012bfc0 [id=d4be274b36a845a99a4b96b8846248fd]). Expected version 1 got 0
I20251025 07:52:50.711956 2146 ts_tablet_manager.cc:1950] T cb829649f73d4d3f8862d268c953bd86 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:50.713081 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 306) took 5002 ms. Trace:
I20251025 07:52:50.714520 447 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 6fe43ccecf7047319090ebd035fe24a5 (table default.Table_68b39cfaff194c3d8dbe9eaee7d8b44c [id=c95fc6d40893462eacee1276fb9cacfe]). Expected version 1 got 0
I20251025 07:52:50.718436 1167 tablet_bootstrap.cc:492] T 6fb1a1d38fb94f07b79ac4f8334fa719 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:50.718619 1167 ts_tablet_manager.cc:1403] T 6fb1a1d38fb94f07b79ac4f8334fa719 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.001s
I20251025 07:52:50.718887 847 leader_election.cc:304] T 3e79d69d5d004cc395b8e35569e25440 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.718793 2146 rpcz_store.cc:276] 1025 07:52:45.710683 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.710960 (+ 277us) service_pool.cc:225] Handling call
1025 07:52:50.624456 (+4913496us) ts_tablet_manager.cc:1151] Deleting tablet cb829649f73d4d3f8862d268c953bd86
1025 07:52:50.624467 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.649780 (+ 25313us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.704827 (+ 55047us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.713064 (+ 8237us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4913355}
I20251025 07:52:50.719648 1167 raft_consensus.cc:359] T 6fb1a1d38fb94f07b79ac4f8334fa719 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.719864 1167 raft_consensus.cc:385] T 6fb1a1d38fb94f07b79ac4f8334fa719 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.719923 1167 raft_consensus.cc:740] T 6fb1a1d38fb94f07b79ac4f8334fa719 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.719961 3204 raft_consensus.cc:2804] T 3e79d69d5d004cc395b8e35569e25440 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:50.720047 3204 raft_consensus.cc:493] T 3e79d69d5d004cc395b8e35569e25440 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:50.720105 3204 raft_consensus.cc:3060] T 3e79d69d5d004cc395b8e35569e25440 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.720191 2146 tablet_replica.cc:333] T ebf3a2a926754305b5bd95fcebb25d29 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:50.720125 1167 consensus_queue.cc:260] T 6fb1a1d38fb94f07b79ac4f8334fa719 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.720383 2146 raft_consensus.cc:2243] T ebf3a2a926754305b5bd95fcebb25d29 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:50.720557 1167 ts_tablet_manager.cc:1434] T 6fb1a1d38fb94f07b79ac4f8334fa719 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251025 07:52:50.720614 2146 raft_consensus.cc:2272] T ebf3a2a926754305b5bd95fcebb25d29 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:50.721119 2146 ts_tablet_manager.cc:1916] T ebf3a2a926754305b5bd95fcebb25d29 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:50.721473 1167 tablet_bootstrap.cc:492] T 421db850a64048218eecb9de65e2bdbf P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:50.723500 3204 raft_consensus.cc:515] T 3e79d69d5d004cc395b8e35569e25440 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.722792 1167 tablet_bootstrap.cc:654] T 421db850a64048218eecb9de65e2bdbf P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:50.724551 3204 leader_election.cc:290] T 3e79d69d5d004cc395b8e35569e25440 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:50.724611 1041 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e79d69d5d004cc395b8e35569e25440" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:50.725406 850 leader_election.cc:343] T 3e79d69d5d004cc395b8e35569e25440 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.719812 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet cb829649f73d4d3f8862d268c953bd86 (table default.Table_a9026f5e095c4063b86d4c544c686e22 [id=7944225d3ce7439b881bfb9289d9f040]) successfully deleted
I20251025 07:52:50.727036 847 leader_election.cc:304] T ab68e5a89b234064b27aa7cae9467e14 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.727293 3204 raft_consensus.cc:2804] T ab68e5a89b234064b27aa7cae9467e14 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:50.727430 3204 raft_consensus.cc:493] T ab68e5a89b234064b27aa7cae9467e14 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:50.727566 3204 raft_consensus.cc:3060] T ab68e5a89b234064b27aa7cae9467e14 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.727859 447 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet f70091024c034c118fb86578db19a31c (table default.Table_b7da40b96d6c4359bed6f898e8cf6275 [id=e041946c093b4c789ce175c736cc230b]). Expected version 2 got 0
I20251025 07:52:50.728307 447 catalog_manager.cc:5649] T 5dab439bc50d4e869548154e1d222ad6 P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } }
I20251025 07:52:50.729221 447 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 5dab439bc50d4e869548154e1d222ad6 (table default.Table_fb9def2302784c999bf0f5707c60eb4c [id=45325c188eac4d34bb016d01de091f87]). Expected version 3 got 0
I20251025 07:52:50.729774 447 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]). Expected version 3 got 0
I20251025 07:52:50.730952 3204 raft_consensus.cc:515] T ab68e5a89b234064b27aa7cae9467e14 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.731865 1167 tablet_bootstrap.cc:492] T 421db850a64048218eecb9de65e2bdbf P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:50.731874 3204 leader_election.cc:290] T ab68e5a89b234064b27aa7cae9467e14 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:50.731997 1167 ts_tablet_manager.cc:1403] T 421db850a64048218eecb9de65e2bdbf P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20251025 07:52:50.732599 1167 raft_consensus.cc:359] T 421db850a64048218eecb9de65e2bdbf P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.732753 1167 raft_consensus.cc:385] T 421db850a64048218eecb9de65e2bdbf P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.732805 1167 raft_consensus.cc:740] T 421db850a64048218eecb9de65e2bdbf P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.734023 847 leader_election.cc:304] T e429b94326cc4827a03968fd92de38e0 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.734552 447 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 5fc092c2bb2d41ac8c5b4fc07460c8ec (table default.Table_cd20e202397645f8821dcde13b5c0ce3 [id=630c274339664b9db18ab4ef7c29c16f]). Expected version 2 got 0
I20251025 07:52:50.734608 847 leader_election.cc:304] T df9db9dd31d840e0b1394db04d662a62 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.734752 1041 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ab68e5a89b234064b27aa7cae9467e14" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:50.734746 1167 consensus_queue.cc:260] T 421db850a64048218eecb9de65e2bdbf P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.734869 2686 raft_consensus.cc:2804] T e429b94326cc4827a03968fd92de38e0 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:50.734992 2686 raft_consensus.cc:697] T e429b94326cc4827a03968fd92de38e0 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:50.735244 1167 ts_tablet_manager.cc:1434] T 421db850a64048218eecb9de65e2bdbf P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251025 07:52:50.735209 2686 consensus_queue.cc:237] T e429b94326cc4827a03968fd92de38e0 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } }
I20251025 07:52:50.735520 1167 tablet_bootstrap.cc:492] T 3c4970c8a491460e8b4b1c29fde83e4c P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:50.736035 2988 raft_consensus.cc:2804] T df9db9dd31d840e0b1394db04d662a62 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:50.736135 2988 raft_consensus.cc:697] T df9db9dd31d840e0b1394db04d662a62 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:50.736342 2988 consensus_queue.cc:237] T df9db9dd31d840e0b1394db04d662a62 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.736845 1167 tablet_bootstrap.cc:654] T 3c4970c8a491460e8b4b1c29fde83e4c P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:50.738387 447 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 3760348dbe284c41a01a4fe479a2103f (table default.Table_088aad2a46b041bfa29ed8e6929760b1 [id=834669c54dae449790818684fd8659bc]). Expected version 2 got 0
W20251025 07:52:50.738698 850 leader_election.cc:343] T ab68e5a89b234064b27aa7cae9467e14 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.739032 447 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 22f7db660616457b83b24e947a0b0bae (table default.Table_4a86191163f2408f9bcc795cd8d1bbf6 [id=bb316a7e000748ca902a9bfb494802d7]). Expected version 1 got 0
W20251025 07:52:50.739063 850 consensus_peers.cc:597] T d0240d20f9904b4bbc8bf7f0f05e8850 P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:50.743273 447 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]). Expected version 1 got 0
I20251025 07:52:50.743769 2990 raft_consensus.cc:493] T b71bdfd09a27449d98509bb220680147 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:50.743927 2990 raft_consensus.cc:515] T b71bdfd09a27449d98509bb220680147 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.744609 2990 leader_election.cc:290] T b71bdfd09a27449d98509bb220680147 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:50.745098 447 catalog_manager.cc:5649] T 2d24e3e7a9e4431db609331ad1fd57c9 P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } }
I20251025 07:52:50.745733 447 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 830fdbc6812e416690d0db42a73af503 (table default.Table_92824b3f2a774564b3aea04aaa25d204 [id=fb21a45b86ab446e8c3dae4512a9598f]). Expected version 1 got 0
I20251025 07:52:50.746526 447 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525]). Expected version 1 got 0
I20251025 07:52:50.750672 1039 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b71bdfd09a27449d98509bb220680147" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:50.751242 850 leader_election.cc:343] T b71bdfd09a27449d98509bb220680147 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.752611 2169 tablet_replica.cc:333] T c11f674385184ae6b235f1db8bf1fb91 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:50.752923 2169 raft_consensus.cc:2243] T c11f674385184ae6b235f1db8bf1fb91 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:50.752044 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 95efd553493e4966bc48a01cac6e1e10 (table default.Table_c4716673eec04f3baa6fc852c668fed8 [id=1d09f9e02b4a4a4d8d38ee522fea841b]) successfully deleted
W20251025 07:52:50.755132 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 4b278ce7768349e49dcd1131e404ab34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b278ce7768349e49dcd1131e404ab34 already in progress: creating tablet
I20251025 07:52:50.755745 447 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 755f0d6c87044e20a8020f8410940123 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 4 got 0
I20251025 07:52:50.756565 447 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet e330e1b53bed4e7cace1c4fb7fa6b565 (table default.Table_a9026f5e095c4063b86d4c544c686e22 [id=7944225d3ce7439b881bfb9289d9f040]). Expected version 2 got 0
I20251025 07:52:50.758172 2169 raft_consensus.cc:2272] T c11f674385184ae6b235f1db8bf1fb91 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:50.758884 2169 ts_tablet_manager.cc:1916] T c11f674385184ae6b235f1db8bf1fb91 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:50.759647 3204 raft_consensus.cc:493] T 66134661be834f7b811d7927ab629812 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:50.759786 3204 raft_consensus.cc:515] T 66134661be834f7b811d7927ab629812 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.760533 3204 leader_election.cc:290] T 66134661be834f7b811d7927ab629812 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:50.760742 1039 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66134661be834f7b811d7927ab629812" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:50.761338 850 leader_election.cc:343] T 66134661be834f7b811d7927ab629812 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.769057 847 leader_election.cc:304] T 0582ee42ab564c07ae3facfc5935617c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.769560 3203 raft_consensus.cc:2804] T 0582ee42ab564c07ae3facfc5935617c P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:50.769711 3203 raft_consensus.cc:697] T 0582ee42ab564c07ae3facfc5935617c P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:50.769945 3203 consensus_queue.cc:237] T 0582ee42ab564c07ae3facfc5935617c P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.778614 447 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 3a0f50c8000144a4acb5219484b01800 (table default.Table_c62e0dc1ae504f9bb912013506ff3031 [id=6f395892b09240fea02482b3199cfb5e]). Expected version 1 got 0
I20251025 07:52:50.779448 447 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236]). Expected version 1 got 0
I20251025 07:52:50.780556 710 leader_election.cc:304] T 32a143a715cb485cb4501f4e28c68252 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.781145 3236 raft_consensus.cc:2804] T 32a143a715cb485cb4501f4e28c68252 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:50.781329 3236 raft_consensus.cc:493] T 32a143a715cb485cb4501f4e28c68252 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:50.781414 3236 raft_consensus.cc:3060] T 32a143a715cb485cb4501f4e28c68252 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.781601 1166 tablet_bootstrap.cc:492] T 2c4789b9d89f42cb844e6cb595e687db P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:50.781780 1166 ts_tablet_manager.cc:1403] T 2c4789b9d89f42cb844e6cb595e687db P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.146s user 0.014s sys 0.003s
I20251025 07:52:50.782550 1166 raft_consensus.cc:359] T 2c4789b9d89f42cb844e6cb595e687db P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.782769 1166 raft_consensus.cc:385] T 2c4789b9d89f42cb844e6cb595e687db P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.782855 1166 raft_consensus.cc:740] T 2c4789b9d89f42cb844e6cb595e687db P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.783116 1166 consensus_queue.cc:260] T 2c4789b9d89f42cb844e6cb595e687db P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.783700 1166 ts_tablet_manager.cc:1434] T 2c4789b9d89f42cb844e6cb595e687db P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:50.784087 1166 tablet_bootstrap.cc:492] T 294d7d20c2f545b8813e632aa058a60c P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
W20251025 07:52:50.785681 850 consensus_peers.cc:597] T d46673a283fe42b78ddeb99fbd2678eb P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:50.783190 3204 raft_consensus.cc:493] T 8707b93d491e4e4386d148087a2f0870 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:50.785802 3236 raft_consensus.cc:515] T 32a143a715cb485cb4501f4e28c68252 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.785931 3204 raft_consensus.cc:515] T 8707b93d491e4e4386d148087a2f0870 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.786495 3204 leader_election.cc:290] T 8707b93d491e4e4386d148087a2f0870 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:50.786796 447 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 8707b93d491e4e4386d148087a2f0870 (table default.Table_799dcb1cef2f4fe6b9b8f17f9f1026c5 [id=59840a567e5244e8a9024cbad42471e4]). Expected version 1 got 0
I20251025 07:52:50.786814 1041 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32a143a715cb485cb4501f4e28c68252" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:50.787145 1039 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8707b93d491e4e4386d148087a2f0870" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:50.787279 712 leader_election.cc:343] T 32a143a715cb485cb4501f4e28c68252 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.787034 3236 leader_election.cc:290] T 32a143a715cb485cb4501f4e28c68252 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:50.787227 447 catalog_manager.cc:5649] T 67cf96a7dd3f42e19b86746c87e689f6 P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } }
W20251025 07:52:50.787539 850 leader_election.cc:343] T 8707b93d491e4e4386d148087a2f0870 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.788100 1169 tablet_bootstrap.cc:492] T ed243540f2d746b980c25687fbca7414 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:50.788205 1166 tablet_bootstrap.cc:654] T 294d7d20c2f545b8813e632aa058a60c P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:50.788221 1169 ts_tablet_manager.cc:1403] T ed243540f2d746b980c25687fbca7414 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.171s user 0.023s sys 0.054s
I20251025 07:52:50.789259 447 catalog_manager.cc:5649] T df0d86ad301245c1a6bcb44805936d25 P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } }
I20251025 07:52:50.790467 447 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_67b99749f3bf43228b40d8b9807e69b4 [id=fc8180c09d7b4e74b2e161f7acb8fe0e]). Expected version 2 got 0
I20251025 07:52:50.790777 447 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c]). Expected version 1 got 0
I20251025 07:52:50.790791 2166 ts_tablet_manager.cc:1929] T dda0b26e4f1a44928bc4b84f333901c0 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:50.790882 2166 log.cc:1199] T dda0b26e4f1a44928bc4b84f333901c0 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/dda0b26e4f1a44928bc4b84f333901c0
I20251025 07:52:50.791163 2166 ts_tablet_manager.cc:1950] T dda0b26e4f1a44928bc4b84f333901c0 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:50.791882 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 245) took 6213 ms. Trace:
I20251025 07:52:50.791970 2166 rpcz_store.cc:276] 1025 07:52:44.578897 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.578980 (+ 83us) service_pool.cc:225] Handling call
1025 07:52:50.603663 (+6024683us) ts_tablet_manager.cc:1151] Deleting tablet dda0b26e4f1a44928bc4b84f333901c0
1025 07:52:50.603673 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.754234 (+150561us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.790699 (+ 36465us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.791864 (+ 1165us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6007186}
I20251025 07:52:50.792297 2169 ts_tablet_manager.cc:1929] T c11f674385184ae6b235f1db8bf1fb91 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:50.792418 2169 log.cc:1199] T c11f674385184ae6b235f1db8bf1fb91 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/c11f674385184ae6b235f1db8bf1fb91
I20251025 07:52:50.792055 447 catalog_manager.cc:5649] T 1c40b2e06d894a06af9e58285892ea8d P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } }
I20251025 07:52:50.792925 447 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet b56209e4934e4407a8b4518ca4af0d46 (table default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de]). Expected version 1 got 0
I20251025 07:52:50.792707 1169 raft_consensus.cc:359] T ed243540f2d746b980c25687fbca7414 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.793212 1169 raft_consensus.cc:385] T ed243540f2d746b980c25687fbca7414 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.793294 1169 raft_consensus.cc:740] T ed243540f2d746b980c25687fbca7414 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.793504 1169 consensus_queue.cc:260] T ed243540f2d746b980c25687fbca7414 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.793200 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet dda0b26e4f1a44928bc4b84f333901c0 (table default.Table_c1bf6bb7c62c4648853b9a5659c73da0 [id=d0b9f20f28d340f4944e85ee0f563fab]) successfully deleted
I20251025 07:52:50.794123 1169 ts_tablet_manager.cc:1434] T ed243540f2d746b980c25687fbca7414 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251025 07:52:50.794399 1169 tablet_bootstrap.cc:492] T 243b8828fc344bacab46d7bab6bd2dec P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:50.793995 447 catalog_manager.cc:5649] T dc4a788dca9d4a8ba40f8fdf1a4b915e P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } }
I20251025 07:52:50.795976 1169 tablet_bootstrap.cc:654] T 243b8828fc344bacab46d7bab6bd2dec P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:50.796272 447 catalog_manager.cc:5649] T 437da5ded9b648abb668a8cb4896452c P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } }
I20251025 07:52:50.796793 2166 tablet_replica.cc:333] T 1b5ac7e845a24b2eb5b2429d47361232 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:50.796917 447 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_d11bba62f65740319701b4337a5c255a [id=ad812862698d421184ebf501ce79b7be]). Expected version 3 got 0
I20251025 07:52:50.797000 2166 raft_consensus.cc:2243] T 1b5ac7e845a24b2eb5b2429d47361232 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:50.797155 2166 raft_consensus.cc:2272] T 1b5ac7e845a24b2eb5b2429d47361232 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:50.797261 447 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84]). Expected version 1 got 0
I20251025 07:52:50.797681 2166 ts_tablet_manager.cc:1916] T 1b5ac7e845a24b2eb5b2429d47361232 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:50.799400 447 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3]). Expected version 2 got 0
I20251025 07:52:50.800016 1095 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
I20251025 07:52:50.800216 1166 tablet_bootstrap.cc:492] T 294d7d20c2f545b8813e632aa058a60c P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:50.800400 1166 ts_tablet_manager.cc:1403] T 294d7d20c2f545b8813e632aa058a60c P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20251025 07:52:50.801189 1166 raft_consensus.cc:359] T 294d7d20c2f545b8813e632aa058a60c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.801404 1166 raft_consensus.cc:385] T 294d7d20c2f545b8813e632aa058a60c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.801510 1166 raft_consensus.cc:740] T 294d7d20c2f545b8813e632aa058a60c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.801766 1166 consensus_queue.cc:260] T 294d7d20c2f545b8813e632aa058a60c P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.792723 2169 ts_tablet_manager.cc:1950] T c11f674385184ae6b235f1db8bf1fb91 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:50.802806 1166 ts_tablet_manager.cc:1434] T 294d7d20c2f545b8813e632aa058a60c P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:50.803356 1166 tablet_bootstrap.cc:492] T f7011f1dcad74fe4ba0356c618890e06 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:50.805028 1167 tablet_bootstrap.cc:492] T 3c4970c8a491460e8b4b1c29fde83e4c P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:50.803023 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet c11f674385184ae6b235f1db8bf1fb91 (table default.Table_ce27d8ed3ede46579f2ed3ae81536572 [id=c4c41c3b236141eb8e03dbb171f6b56d]) successfully deleted
I20251025 07:52:50.805155 822 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
W20251025 07:52:50.805627 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 2e098c9c4613459295b6043048cba587 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e098c9c4613459295b6043048cba587 already in progress: creating tablet
W20251025 07:52:50.805869 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet cc407c7eb12e4eadb41fb21385611c94 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc407c7eb12e4eadb41fb21385611c94 already in progress: creating tablet
I20251025 07:52:50.805155 1167 ts_tablet_manager.cc:1403] T 3c4970c8a491460e8b4b1c29fde83e4c P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.070s user 0.006s sys 0.010s
W20251025 07:52:50.806083 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 6897405bbc1c47a4ab0aa3af80e60e3d because the tablet was not found. No further retry: Not found: Tablet not found: 6897405bbc1c47a4ab0aa3af80e60e3d
I20251025 07:52:50.806823 1167 raft_consensus.cc:359] T 3c4970c8a491460e8b4b1c29fde83e4c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.807041 1167 raft_consensus.cc:385] T 3c4970c8a491460e8b4b1c29fde83e4c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.807116 1167 raft_consensus.cc:740] T 3c4970c8a491460e8b4b1c29fde83e4c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
W20251025 07:52:50.808012 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 3c4970c8a491460e8b4b1c29fde83e4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c4970c8a491460e8b4b1c29fde83e4c already in progress: creating tablet
I20251025 07:52:50.808492 1166 tablet_bootstrap.cc:654] T f7011f1dcad74fe4ba0356c618890e06 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:50.808698 2166 ts_tablet_manager.cc:1929] T 1b5ac7e845a24b2eb5b2429d47361232 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:50.806941 447 catalog_manager.cc:5649] T 288e543bb279410c91c01dd1623e110f P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } }
I20251025 07:52:50.808810 2166 log.cc:1199] T 1b5ac7e845a24b2eb5b2429d47361232 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/1b5ac7e845a24b2eb5b2429d47361232
I20251025 07:52:50.809165 2166 ts_tablet_manager.cc:1950] T 1b5ac7e845a24b2eb5b2429d47361232 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:50.809551 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 246) took 6230 ms. Trace:
I20251025 07:52:50.809664 2166 rpcz_store.cc:276] 1025 07:52:44.579347 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.579392 (+ 45us) service_pool.cc:225] Handling call
1025 07:52:50.792321 (+6212929us) ts_tablet_manager.cc:1151] Deleting tablet 1b5ac7e845a24b2eb5b2429d47361232
1025 07:52:50.792332 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.802806 (+ 10474us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.808589 (+ 5783us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.809533 (+ 944us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6195795}
I20251025 07:52:50.809921 447 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 288e543bb279410c91c01dd1623e110f (table default.Table_6e242ba55b194c23a6701f4b064fdbe5 [id=4c34b64b299f4162a53866cf01c5298f]). Expected version 1 got 0
I20251025 07:52:50.810109 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 1b5ac7e845a24b2eb5b2429d47361232 (table default.Table_f6781cf6e9104f51a278cab3d1781a1a [id=8b4207028ba74a5ebd5d4f6690f46893]) successfully deleted
W20251025 07:52:50.810286 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 32a143a715cb485cb4501f4e28c68252 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32a143a715cb485cb4501f4e28c68252 already in progress: creating tablet
W20251025 07:52:50.810541 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 1912e59548874e0e8e989050ec0064c5 because the tablet was not found. No further retry: Not found: Tablet not found: 1912e59548874e0e8e989050ec0064c5
W20251025 07:52:50.810781 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 0540d6a10cf04fa89756623e9aaaeb68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0540d6a10cf04fa89756623e9aaaeb68 already in progress: creating tablet
W20251025 07:52:50.811033 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet c59f6e6eb9594a8e83f109a09b034045 because the tablet was not found. No further retry: Not found: Tablet not found: c59f6e6eb9594a8e83f109a09b034045
I20251025 07:52:50.811095 1047 raft_consensus.cc:3060] T ed243540f2d746b980c25687fbca7414 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.812929 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 247) took 6233 ms. Trace:
I20251025 07:52:50.813279 447 catalog_manager.cc:5649] T 075f0a151e024220b51ba7fcaecdcc3d P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } }
I20251025 07:52:50.814324 447 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e]). Expected version 1 got 0
I20251025 07:52:50.813292 2166 rpcz_store.cc:276] 1025 07:52:44.579667 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.579714 (+ 47us) service_pool.cc:225] Handling call
1025 07:52:50.812836 (+6233122us) ts_tablet_manager.cc:1151] Deleting tablet 21e52928580c4762b0ae7152490ef955
1025 07:52:50.812847 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.812912 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6216090}
I20251025 07:52:50.814941 710 leader_election.cc:304] T 1b22bd172e0e4f86896e9ecdef666b42 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.815517 3137 raft_consensus.cc:2804] T 1b22bd172e0e4f86896e9ecdef666b42 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:50.815611 3137 raft_consensus.cc:493] T 1b22bd172e0e4f86896e9ecdef666b42 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:50.815693 3137 raft_consensus.cc:3060] T 1b22bd172e0e4f86896e9ecdef666b42 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.818285 2169 tablet_replica.cc:333] T 29dbaf9149134f548a974a5779f7da67 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:50.818495 2169 raft_consensus.cc:2243] T 29dbaf9149134f548a974a5779f7da67 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:50.818990 1167 consensus_queue.cc:260] T 3c4970c8a491460e8b4b1c29fde83e4c P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.819859 3137 raft_consensus.cc:515] T 1b22bd172e0e4f86896e9ecdef666b42 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:50.815282 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 21e52928580c4762b0ae7152490ef955 because the tablet was not found. No further retry: Not found: Tablet not found: 21e52928580c4762b0ae7152490ef955
I20251025 07:52:50.820915 2169 raft_consensus.cc:2272] T 29dbaf9149134f548a974a5779f7da67 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:50.818041 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet b02f5c0ab0b24288bb50a3ba0e6bd78a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b02f5c0ab0b24288bb50a3ba0e6bd78a already in progress: creating tablet
I20251025 07:52:50.821148 1039 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b22bd172e0e4f86896e9ecdef666b42" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:50.821606 712 leader_election.cc:343] T 1b22bd172e0e4f86896e9ecdef666b42 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.821609 447 catalog_manager.cc:5649] T d0240d20f9904b4bbc8bf7f0f05e8850 P 3aea2869ff814cc7b142012faa274382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aea2869ff814cc7b142012faa274382 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "3aea2869ff814cc7b142012faa274382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } health_report { overall_health: HEALTHY } } }
I20251025 07:52:50.822013 447 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]). Expected version 1 got 0
I20251025 07:52:50.822394 447 catalog_manager.cc:5741] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563) does not have the latest schema for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]). Expected version 1 got 0
I20251025 07:52:50.822932 1167 ts_tablet_manager.cc:1434] T 3c4970c8a491460e8b4b1c29fde83e4c P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20251025 07:52:50.823189 3137 leader_election.cc:290] T 1b22bd172e0e4f86896e9ecdef666b42 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:50.823607 2169 ts_tablet_manager.cc:1916] T 29dbaf9149134f548a974a5779f7da67 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:50.825058 1169 tablet_bootstrap.cc:492] T 243b8828fc344bacab46d7bab6bd2dec P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:50.825212 1169 ts_tablet_manager.cc:1403] T 243b8828fc344bacab46d7bab6bd2dec P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.002s
I20251025 07:52:50.825865 1169 raft_consensus.cc:359] T 243b8828fc344bacab46d7bab6bd2dec P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.826064 1169 raft_consensus.cc:385] T 243b8828fc344bacab46d7bab6bd2dec P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.826195 1169 raft_consensus.cc:740] T 243b8828fc344bacab46d7bab6bd2dec P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.826432 1169 consensus_queue.cc:260] T 243b8828fc344bacab46d7bab6bd2dec P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.826647 448 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet 0063a0776b2247afaa8be82f6eff26f1
I20251025 07:52:50.827212 448 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:47610 (request call id 121) took 4975 ms. Trace:
I20251025 07:52:50.827322 448 rpcz_store.cc:276] 1025 07:52:45.851890 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.773776 (+3921886us) service_pool.cc:225] Handling call
1025 07:52:49.774282 (+ 506us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:49.774424 (+ 142us) catalog_manager.cc:2815] Modifying in-memory table state
1025 07:52:49.774485 (+ 61us) catalog_manager.cc:2824] Locking tablets
1025 07:52:49.940590 (+166105us) catalog_manager.cc:2838] Removing table and tablets from system table
1025 07:52:50.015377 (+ 74787us) catalog_manager.cc:2856] Removing table from by-name map
1025 07:52:50.015472 (+ 95us) catalog_manager.cc:2872] Committing in-memory state
1025 07:52:50.826584 (+811112us) catalog_manager.cc:2876] Aborting table tasks
1025 07:52:50.827175 (+ 591us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:49.974736 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:49.974869 (+ 133us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:49.974877 (+ 8us) write_op.cc:435] Acquired schema lock
1025 07:52:49.974882 (+ 5us) tablet.cc:660] Decoding operations
1025 07:52:49.975046 (+ 164us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:49.975060 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:49.975066 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:49.975173 (+ 107us) tablet.cc:699] Row locks acquired
1025 07:52:49.975176 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:49.975213 (+ 37us) write_op.cc:270] Start()
1025 07:52:49.975249 (+ 36us) write_op.cc:276] Timestamp: P: 1761378769975195 usec, L: 0
1025 07:52:49.975252 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:49.975400 (+ 148us) log.cc:844] Serialized 725 byte log entry
1025 07:52:50.014033 (+ 38633us) op_driver.cc:464] REPLICATION: finished
1025 07:52:50.014600 (+ 567us) write_op.cc:301] APPLY: starting
1025 07:52:50.014614 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:50.014650 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:50.014688 (+ 38us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:50.014742 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:50.014774 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1025 07:52:50.014781 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:50.014846 (+ 65us) log.cc:844] Serialized 29 byte log entry
1025 07:52:50.014887 (+ 41us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:50.014902 (+ 15us) write_op.cc:454] Released schema lock
1025 07:52:50.014907 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:50.014968 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:50.014971 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":533,"apply.run_cpu_time_us":361,"apply.run_wall_time_us":361,"num_ops":2,"prepare.queue_time_us":33335,"prepare.run_cpu_time_us":8499,"prepare.run_wall_time_us":25581,"raft.queue_time_us":657,"raft.run_cpu_time_us":40,"raft.run_wall_time_us":41,"replication_time_us":38762,"spinlock_wait_cycles":50560,"thread_start_us":24785,"threads_started":1}]]}
I20251025 07:52:50.828305 448 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_68774a9f47334984a5e9cfae8b304519"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:50.828994 1167 tablet_bootstrap.cc:492] T 60c60a9ee37f43978910bd067a098bb7 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
W20251025 07:52:50.828977 448 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68774a9f47334984a5e9cfae8b304519 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:50.830660 1167 tablet_bootstrap.cc:654] T 60c60a9ee37f43978910bd067a098bb7 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:50.839125 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 0063a0776b2247afaa8be82f6eff26f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:49 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:50.840523 1169 ts_tablet_manager.cc:1434] T 243b8828fc344bacab46d7bab6bd2dec P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20251025 07:52:50.841658 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 0063a0776b2247afaa8be82f6eff26f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:49 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:50.844966 1169 tablet_bootstrap.cc:492] T 4f7c15529be54a11961f385a953b0b03 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:50.845562 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet 0063a0776b2247afaa8be82f6eff26f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:49 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.845719 448 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 124) took 4299 ms. Trace:
I20251025 07:52:50.846312 2166 tablet_replica.cc:333] T 570201e7bf05406a97c81dde071d894b P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:50.845865 448 rpcz_store.cc:276] 1025 07:52:46.546433 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:50.828068 (+4281635us) service_pool.cc:225] Handling call
1025 07:52:50.829083 (+ 1015us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:50.829608 (+ 525us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:50.845431 (+ 15823us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:50.845555 (+ 124us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:50.845673 (+ 118us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:50.830245 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:50.830361 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:50.830367 (+ 6us) write_op.cc:435] Acquired schema lock
1025 07:52:50.830370 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:50.830460 (+ 90us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:50.830475 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:50.830483 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:50.830597 (+ 114us) tablet.cc:699] Row locks acquired
1025 07:52:50.830600 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:50.830642 (+ 42us) write_op.cc:270] Start()
1025 07:52:50.830680 (+ 38us) write_op.cc:276] Timestamp: P: 1761378770830623 usec, L: 0
1025 07:52:50.830684 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:50.830890 (+ 206us) log.cc:844] Serialized 433 byte log entry
1025 07:52:50.842645 (+ 11755us) op_driver.cc:464] REPLICATION: finished
1025 07:52:50.842773 (+ 128us) write_op.cc:301] APPLY: starting
1025 07:52:50.842789 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:50.842836 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:50.842869 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:50.843008 (+ 139us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:50.843041 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:50.843048 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:50.843146 (+ 98us) log.cc:844] Serialized 29 byte log entry
1025 07:52:50.843211 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:50.843254 (+ 43us) write_op.cc:454] Released schema lock
1025 07:52:50.843261 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:50.843334 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:50.843337 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":82,"apply.run_cpu_time_us":549,"apply.run_wall_time_us":548,"num_ops":2,"prepare.queue_time_us":163,"prepare.run_cpu_time_us":773,"prepare.run_wall_time_us":774,"raft.queue_time_us":373,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":213,"replication_time_us":11931}]]}
I20251025 07:52:50.847045 448 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_7a900ed2ccce48ab9116f9e8f381abc6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:50.847559 2166 raft_consensus.cc:2243] T 570201e7bf05406a97c81dde071d894b P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
W20251025 07:52:50.847661 448 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a900ed2ccce48ab9116f9e8f381abc6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:50.848407 2166 raft_consensus.cc:2272] T 570201e7bf05406a97c81dde071d894b P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:50.849006 2166 ts_tablet_manager.cc:1916] T 570201e7bf05406a97c81dde071d894b P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:50.849591 1169 tablet_bootstrap.cc:654] T 4f7c15529be54a11961f385a953b0b03 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:50.850313 2169 ts_tablet_manager.cc:1929] T 29dbaf9149134f548a974a5779f7da67 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:50.858057 2169 log.cc:1199] T 29dbaf9149134f548a974a5779f7da67 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/29dbaf9149134f548a974a5779f7da67
I20251025 07:52:50.858457 2169 ts_tablet_manager.cc:1950] T 29dbaf9149134f548a974a5779f7da67 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:50.860333 2166 ts_tablet_manager.cc:1929] T 570201e7bf05406a97c81dde071d894b P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:50.861313 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 29dbaf9149134f548a974a5779f7da67 (table default.Table_c43ac1adc76d4ccf819411ec76775932 [id=106e3728b0b846178eb48ccbf076e24e]) successfully deleted
W20251025 07:52:50.861701 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 6dcab53a020847a1aa27930265ba0205 because the tablet was not found. No further retry: Not found: Tablet not found: 6dcab53a020847a1aa27930265ba0205
I20251025 07:52:50.862600 2166 log.cc:1199] T 570201e7bf05406a97c81dde071d894b P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/570201e7bf05406a97c81dde071d894b
I20251025 07:52:50.863155 2166 ts_tablet_manager.cc:1950] T 570201e7bf05406a97c81dde071d894b P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
W20251025 07:52:50.863750 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet ee1c8ea694284d80a30f9c05be4cbf49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee1c8ea694284d80a30f9c05be4cbf49 already in progress: creating tablet
I20251025 07:52:50.863873 2169 tablet_replica.cc:333] T 1b5ac7e845a24b2eb5b2429d47361232 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:50.864290 448 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 125) took 4317 ms. Trace:
I20251025 07:52:50.864414 448 rpcz_store.cc:276] 1025 07:52:46.546574 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:50.846848 (+4300274us) service_pool.cc:225] Handling call
1025 07:52:50.854046 (+ 7198us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:50.854937 (+ 891us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:50.863954 (+ 9017us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:50.863960 (+ 6us) write_op.cc:454] Released schema lock
1025 07:52:50.864033 (+ 73us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:50.864154 (+ 121us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:50.864250 (+ 96us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:50.855632 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:50.855785 (+ 153us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:50.855794 (+ 9us) write_op.cc:435] Acquired schema lock
1025 07:52:50.855798 (+ 4us) tablet.cc:660] Decoding operations
1025 07:52:50.855941 (+ 143us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:50.855958 (+ 17us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:50.855965 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:50.856067 (+ 102us) tablet.cc:699] Row locks acquired
1025 07:52:50.856071 (+ 4us) write_op.cc:260] PREPARE: finished
1025 07:52:50.856118 (+ 47us) write_op.cc:270] Start()
1025 07:52:50.856161 (+ 43us) write_op.cc:276] Timestamp: P: 1761378770856098 usec, L: 0
1025 07:52:50.856165 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:50.856394 (+ 229us) log.cc:844] Serialized 433 byte log entry
1025 07:52:50.859945 (+ 3551us) op_driver.cc:464] REPLICATION: finished
1025 07:52:50.860110 (+ 165us) write_op.cc:301] APPLY: starting
1025 07:52:50.860126 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:50.860171 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:50.860196 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:50.860305 (+ 109us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:50.860338 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:50.860346 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:50.860427 (+ 81us) log.cc:844] Serialized 29 byte log entry
1025 07:52:50.860494 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:50.860529 (+ 35us) write_op.cc:454] Released schema lock
1025 07:52:50.860538 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":107,"apply.run_cpu_time_us":493,"apply.run_wall_time_us":492,"num_ops":2,"prepare.queue_time_us":52,"prepare.run_cpu_time_us":926,"prepare.run_wall_time_us":927,"raft.queue_time_us":609,"raft.run_cpu_time_us":340,"raft.run_wall_time_us":340,"replication_time_us":3752}]]}
I20251025 07:52:50.864902 2169 raft_consensus.cc:2243] T 1b5ac7e845a24b2eb5b2429d47361232 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:50.865049 2169 raft_consensus.cc:2272] T 1b5ac7e845a24b2eb5b2429d47361232 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:50.865523 2169 ts_tablet_manager.cc:1916] T 1b5ac7e845a24b2eb5b2429d47361232 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:50.865777 448 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_b7c49ac844ba4fbda183784cdd3982a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:50.866451 448 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7c49ac844ba4fbda183784cdd3982a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:50.869448 1169 tablet_bootstrap.cc:492] T 4f7c15529be54a11961f385a953b0b03 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:50.869637 1169 ts_tablet_manager.cc:1403] T 4f7c15529be54a11961f385a953b0b03 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.003s
I20251025 07:52:50.870483 1169 raft_consensus.cc:359] T 4f7c15529be54a11961f385a953b0b03 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.870743 1169 raft_consensus.cc:385] T 4f7c15529be54a11961f385a953b0b03 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.870859 1169 raft_consensus.cc:740] T 4f7c15529be54a11961f385a953b0b03 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.871116 1169 consensus_queue.cc:260] T 4f7c15529be54a11961f385a953b0b03 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.872973 2146 ts_tablet_manager.cc:1929] T ebf3a2a926754305b5bd95fcebb25d29 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:50.873091 2146 log.cc:1199] T ebf3a2a926754305b5bd95fcebb25d29 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/ebf3a2a926754305b5bd95fcebb25d29
I20251025 07:52:50.873102 448 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 126) took 4290 ms. Trace:
I20251025 07:52:50.873437 2146 ts_tablet_manager.cc:1950] T ebf3a2a926754305b5bd95fcebb25d29 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:50.873226 448 rpcz_store.cc:276] 1025 07:52:46.582981 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:50.865577 (+4282596us) service_pool.cc:225] Handling call
1025 07:52:50.866559 (+ 982us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:50.866989 (+ 430us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:50.872804 (+ 5815us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:50.872809 (+ 5us) write_op.cc:454] Released schema lock
1025 07:52:50.872881 (+ 72us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:50.872980 (+ 99us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:50.873065 (+ 85us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:50.867527 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:50.867630 (+ 103us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:50.867635 (+ 5us) write_op.cc:435] Acquired schema lock
1025 07:52:50.867638 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:50.867716 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:50.867729 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:50.867734 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:50.867800 (+ 66us) tablet.cc:699] Row locks acquired
1025 07:52:50.867803 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:50.867831 (+ 28us) write_op.cc:270] Start()
1025 07:52:50.867858 (+ 27us) write_op.cc:276] Timestamp: P: 1761378770867817 usec, L: 0
1025 07:52:50.867862 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:50.868004 (+ 142us) log.cc:844] Serialized 433 byte log entry
1025 07:52:50.871557 (+ 3553us) op_driver.cc:464] REPLICATION: finished
1025 07:52:50.871749 (+ 192us) write_op.cc:301] APPLY: starting
1025 07:52:50.871770 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:50.871827 (+ 57us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:50.871859 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:50.871956 (+ 97us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:50.871986 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:50.871994 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:50.872062 (+ 68us) log.cc:844] Serialized 29 byte log entry
1025 07:52:50.872115 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:50.872135 (+ 20us) write_op.cc:454] Released schema lock
1025 07:52:50.872141 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":133,"apply.run_cpu_time_us":444,"apply.run_wall_time_us":444,"num_ops":2,"prepare.queue_time_us":117,"prepare.run_cpu_time_us":587,"prepare.run_wall_time_us":587,"raft.queue_time_us":482,"raft.run_cpu_time_us":335,"raft.run_wall_time_us":334,"replication_time_us":3660}]]}
I20251025 07:52:50.874740 448 master_service.cc:954] ReplaceTablet: received request to replace tablet 792f32cbd66449ca98813c669b2a261d from {username='slave'} at 127.0.0.1:47610
I20251025 07:52:50.875133 453 catalog_manager.cc:6194] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: default.Table_750ecb905839434fa9a1aa72f87394be [id=9ce3ba77c56a4ef09cb3b40ea0e488f6] alter complete (version 1)
I20251025 07:52:50.877045 1169 ts_tablet_manager.cc:1434] T 4f7c15529be54a11961f385a953b0b03 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20251025 07:52:50.878058 1169 tablet_bootstrap.cc:492] T 075f0a151e024220b51ba7fcaecdcc3d P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:50.879885 1169 tablet_bootstrap.cc:654] T 075f0a151e024220b51ba7fcaecdcc3d P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:50.863308 3137 raft_consensus.cc:493] T b71bdfd09a27449d98509bb220680147 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:50.881141 3137 raft_consensus.cc:515] T b71bdfd09a27449d98509bb220680147 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.881752 3137 leader_election.cc:290] T b71bdfd09a27449d98509bb220680147 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:50.864516 771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e79d69d5d004cc395b8e35569e25440" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:50.881968 771 raft_consensus.cc:3060] T 3e79d69d5d004cc395b8e35569e25440 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.871130 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ab68e5a89b234064b27aa7cae9467e14" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:50.882176 773 raft_consensus.cc:3060] T ab68e5a89b234064b27aa7cae9467e14 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.871819 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b71bdfd09a27449d98509bb220680147" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:50.874589 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66134661be834f7b811d7927ab629812" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:50.882504 772 raft_consensus.cc:2468] T b71bdfd09a27449d98509bb220680147 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:50.882542 774 raft_consensus.cc:2468] T 66134661be834f7b811d7927ab629812 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:50.884876 771 raft_consensus.cc:2468] T 3e79d69d5d004cc395b8e35569e25440 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:50.885075 773 raft_consensus.cc:2468] T ab68e5a89b234064b27aa7cae9467e14 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
W20251025 07:52:50.876695 712 consensus_peers.cc:597] T 8917a289bd9b4428a4603429edd94581 P 23eeffd3de254ce18db5febfde971c6b -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:50.886415 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 248) took 6306 ms. Trace:
I20251025 07:52:50.886587 2166 rpcz_store.cc:276] 1025 07:52:44.580067 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.580114 (+ 47us) service_pool.cc:225] Handling call
1025 07:52:50.846263 (+6266149us) ts_tablet_manager.cc:1151] Deleting tablet 570201e7bf05406a97c81dde071d894b
1025 07:52:50.846274 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.853811 (+ 7537us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.860220 (+ 6409us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.886395 (+ 26175us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6249638}
I20251025 07:52:50.887351 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 249) took 6306 ms. Trace:
I20251025 07:52:50.887519 2166 rpcz_store.cc:276] 1025 07:52:44.580498 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.580545 (+ 47us) service_pool.cc:225] Handling call
1025 07:52:50.887175 (+6306630us) ts_tablet_manager.cc:1151] Deleting tablet 5cba9c05dd424cc788febc614a34819a
1025 07:52:50.887184 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.887253 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6290504}
I20251025 07:52:50.885926 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b71bdfd09a27449d98509bb220680147" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:50.888149 712 leader_election.cc:343] T b71bdfd09a27449d98509bb220680147 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.888458 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 570201e7bf05406a97c81dde071d894b (table default.Table_c62e0dc1ae504f9bb912013506ff3031 [id=6f395892b09240fea02482b3199cfb5e]) successfully deleted
I20251025 07:52:50.888540 2166 tablet_replica.cc:333] T a7ab3150a35b4caca9119a1374d3f2f7 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:50.888773 2166 raft_consensus.cc:2243] T a7ab3150a35b4caca9119a1374d3f2f7 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
W20251025 07:52:50.888862 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 5cba9c05dd424cc788febc614a34819a because the tablet was not found. No further retry: Not found: Tablet not found: 5cba9c05dd424cc788febc614a34819a
I20251025 07:52:50.888973 2166 raft_consensus.cc:2272] T a7ab3150a35b4caca9119a1374d3f2f7 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:50.889091 439 consensus_peers.cc:597] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944 -> Peer 8603bddd8a084ff7a73a25641768dfa9 (127.0.104.61:42203): Couldn't send request to peer 8603bddd8a084ff7a73a25641768dfa9. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.61:42203: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20251025 07:52:50.889642 2166 ts_tablet_manager.cc:1916] T a7ab3150a35b4caca9119a1374d3f2f7 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:50.891280 850 consensus_peers.cc:597] T 72618756f756484e895e885ef12545c8 P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:50.892091 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8707b93d491e4e4386d148087a2f0870" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:50.892370 773 raft_consensus.cc:2468] T 8707b93d491e4e4386d148087a2f0870 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:50.895890 2169 ts_tablet_manager.cc:1929] T 1b5ac7e845a24b2eb5b2429d47361232 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:50.896011 2169 log.cc:1199] T 1b5ac7e845a24b2eb5b2429d47361232 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/1b5ac7e845a24b2eb5b2429d47361232
I20251025 07:52:50.899348 2169 ts_tablet_manager.cc:1950] T 1b5ac7e845a24b2eb5b2429d47361232 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:50.900704 2169 tablet_replica.cc:333] T fd265cdbf6a8440393a2b5d8240d1872 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:50.900899 2169 raft_consensus.cc:2243] T fd265cdbf6a8440393a2b5d8240d1872 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:50.901039 2169 raft_consensus.cc:2272] T fd265cdbf6a8440393a2b5d8240d1872 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:50.905154 847 leader_election.cc:304] T b71bdfd09a27449d98509bb220680147 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.905539 2686 raft_consensus.cc:2804] T b71bdfd09a27449d98509bb220680147 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:50.905700 2686 raft_consensus.cc:493] T b71bdfd09a27449d98509bb220680147 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:50.905836 2686 raft_consensus.cc:3060] T b71bdfd09a27449d98509bb220680147 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.906015 2169 ts_tablet_manager.cc:1916] T fd265cdbf6a8440393a2b5d8240d1872 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:50.906189 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 1b5ac7e845a24b2eb5b2429d47361232 (table default.Table_f6781cf6e9104f51a278cab3d1781a1a [id=8b4207028ba74a5ebd5d4f6690f46893]) successfully deleted
W20251025 07:52:50.906466 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 9f039d9107ae493486d799ab6a1d731d because the tablet was not found. No further retry: Not found: Tablet not found: 9f039d9107ae493486d799ab6a1d731d
W20251025 07:52:50.906654 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 50ae9bca6bed4f749d54191ac1543410 because the tablet was not found. No further retry: Not found: Tablet not found: 50ae9bca6bed4f749d54191ac1543410
W20251025 07:52:50.906839 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 72dc084f8efc45478771c247e07a16f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72dc084f8efc45478771c247e07a16f4 already in progress: creating tablet
I20251025 07:52:50.908385 450 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet 9979cae195e849908677d78c6c33be71
I20251025 07:52:50.908953 450 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:47610 (request call id 122) took 4820 ms. Trace:
I20251025 07:52:50.909339 2686 raft_consensus.cc:515] T b71bdfd09a27449d98509bb220680147 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.909070 450 rpcz_store.cc:276] 1025 07:52:46.088764 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:50.142208 (+4053444us) service_pool.cc:225] Handling call
1025 07:52:50.143204 (+ 996us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:50.143391 (+ 187us) catalog_manager.cc:2815] Modifying in-memory table state
1025 07:52:50.143466 (+ 75us) catalog_manager.cc:2824] Locking tablets
1025 07:52:50.258451 (+114985us) catalog_manager.cc:2838] Removing table and tablets from system table
1025 07:52:50.264323 (+ 5872us) catalog_manager.cc:2856] Removing table from by-name map
1025 07:52:50.264431 (+ 108us) catalog_manager.cc:2872] Committing in-memory state
1025 07:52:50.908321 (+643890us) catalog_manager.cc:2876] Aborting table tasks
1025 07:52:50.908918 (+ 597us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:50.259322 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:50.259485 (+ 163us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:50.259492 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:50.259494 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:50.259642 (+ 148us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:50.259660 (+ 18us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:50.259667 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:50.259767 (+ 100us) tablet.cc:699] Row locks acquired
1025 07:52:50.259769 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:50.259805 (+ 36us) write_op.cc:270] Start()
1025 07:52:50.259838 (+ 33us) write_op.cc:276] Timestamp: P: 1761378770259786 usec, L: 0
1025 07:52:50.259842 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:50.260012 (+ 170us) log.cc:844] Serialized 759 byte log entry
1025 07:52:50.263631 (+ 3619us) op_driver.cc:464] REPLICATION: finished
1025 07:52:50.263739 (+ 108us) write_op.cc:301] APPLY: starting
1025 07:52:50.263756 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:50.263802 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:50.263848 (+ 46us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:50.263913 (+ 65us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:50.263947 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1025 07:52:50.263954 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:50.264027 (+ 73us) log.cc:844] Serialized 29 byte log entry
1025 07:52:50.264093 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:50.264113 (+ 20us) write_op.cc:454] Released schema lock
1025 07:52:50.264120 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:50.264190 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:50.264194 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":36,"apply.run_cpu_time_us":451,"apply.run_wall_time_us":451,"num_ops":2,"prepare.queue_time_us":74,"prepare.run_cpu_time_us":808,"prepare.run_wall_time_us":807,"raft.queue_time_us":400,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":250,"replication_time_us":3760}]]}
I20251025 07:52:50.910020 450 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_4ee751db640d4715b835ff2a951f515f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:50.910388 2686 leader_election.cc:290] T b71bdfd09a27449d98509bb220680147 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:50.910478 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b71bdfd09a27449d98509bb220680147" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:50.910688 450 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ee751db640d4715b835ff2a951f515f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:50.910975 850 leader_election.cc:343] T b71bdfd09a27449d98509bb220680147 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.912119 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 9979cae195e849908677d78c6c33be71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:50 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:50.913053 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 9979cae195e849908677d78c6c33be71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:50 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:50.918824 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 307) took 5195 ms. Trace:
I20251025 07:52:50.919413 2146 rpcz_store.cc:276] 1025 07:52:45.723410 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.723471 (+ 61us) service_pool.cc:225] Handling call
1025 07:52:50.720141 (+4996670us) ts_tablet_manager.cc:1151] Deleting tablet ebf3a2a926754305b5bd95fcebb25d29
1025 07:52:50.720155 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.724078 (+ 3923us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.872858 (+148780us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.918799 (+ 45941us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4996463}
I20251025 07:52:50.921407 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet 9979cae195e849908677d78c6c33be71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:50 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.921486 450 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 128) took 4310 ms. Trace:
I20251025 07:52:50.921975 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet ebf3a2a926754305b5bd95fcebb25d29 (table default.Table_ac4beca0c3bd4cf29565faf5323d9ff7 [id=857bcd1dbe2e477bbc0d77f79a58f7f7]) successfully deleted
I20251025 07:52:50.921605 450 rpcz_store.cc:276] 1025 07:52:46.610793 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:50.909823 (+4299030us) service_pool.cc:225] Handling call
1025 07:52:50.910783 (+ 960us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:50.911295 (+ 512us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:50.921218 (+ 9923us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:50.921335 (+ 117us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:50.921444 (+ 109us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:50.913835 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:50.913956 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:50.913962 (+ 6us) write_op.cc:435] Acquired schema lock
1025 07:52:50.913965 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:50.914062 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:50.914078 (+ 16us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:50.914083 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:50.914184 (+ 101us) tablet.cc:699] Row locks acquired
1025 07:52:50.914187 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:50.914219 (+ 32us) write_op.cc:270] Start()
1025 07:52:50.914253 (+ 34us) write_op.cc:276] Timestamp: P: 1761378770914204 usec, L: 0
1025 07:52:50.914256 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:50.914482 (+ 226us) log.cc:844] Serialized 433 byte log entry
1025 07:52:50.917890 (+ 3408us) op_driver.cc:464] REPLICATION: finished
1025 07:52:50.918451 (+ 561us) write_op.cc:301] APPLY: starting
1025 07:52:50.918468 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:50.918605 (+ 137us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:50.918658 (+ 53us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:50.918778 (+ 120us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:50.918816 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:50.918823 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:50.918904 (+ 81us) log.cc:844] Serialized 29 byte log entry
1025 07:52:50.918972 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:50.918995 (+ 23us) write_op.cc:454] Released schema lock
1025 07:52:50.919002 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:50.919073 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:50.919077 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":493,"apply.run_cpu_time_us":625,"apply.run_wall_time_us":625,"num_ops":2,"prepare.queue_time_us":2027,"prepare.run_cpu_time_us":746,"prepare.run_wall_time_us":772,"raft.queue_time_us":380,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":224,"replication_time_us":3605}]]}
I20251025 07:52:50.919270 847 leader_election.cc:304] T 66134661be834f7b811d7927ab629812 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.922415 450 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_feeac6a0855742cb8addcbd7508cfec5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:50.923076 847 leader_election.cc:304] T 3e79d69d5d004cc395b8e35569e25440 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.923549 847 leader_election.cc:304] T ab68e5a89b234064b27aa7cae9467e14 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:50.923635 450 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_feeac6a0855742cb8addcbd7508cfec5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:50.923800 3203 raft_consensus.cc:2804] T 66134661be834f7b811d7927ab629812 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:50.923908 3203 raft_consensus.cc:493] T 66134661be834f7b811d7927ab629812 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:50.924052 3203 raft_consensus.cc:3060] T 66134661be834f7b811d7927ab629812 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.924517 3122 raft_consensus.cc:2804] T 3e79d69d5d004cc395b8e35569e25440 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:50.924649 3122 raft_consensus.cc:697] T 3e79d69d5d004cc395b8e35569e25440 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:50.924872 3122 consensus_queue.cc:237] T 3e79d69d5d004cc395b8e35569e25440 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.925043 2169 ts_tablet_manager.cc:1929] T fd265cdbf6a8440393a2b5d8240d1872 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:50.925153 2169 log.cc:1199] T fd265cdbf6a8440393a2b5d8240d1872 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/fd265cdbf6a8440393a2b5d8240d1872
I20251025 07:52:50.925488 2169 ts_tablet_manager.cc:1950] T fd265cdbf6a8440393a2b5d8240d1872 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:50.925907 3204 raft_consensus.cc:2804] T ab68e5a89b234064b27aa7cae9467e14 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:50.926028 3204 raft_consensus.cc:697] T ab68e5a89b234064b27aa7cae9467e14 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:50.926280 3204 consensus_queue.cc:237] T ab68e5a89b234064b27aa7cae9467e14 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.928597 3274 raft_consensus.cc:493] T 294d7d20c2f545b8813e632aa058a60c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:50.928721 3274 raft_consensus.cc:515] T 294d7d20c2f545b8813e632aa058a60c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.929370 3274 leader_election.cc:290] T 294d7d20c2f545b8813e632aa058a60c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:50.929953 1169 tablet_bootstrap.cc:492] T 075f0a151e024220b51ba7fcaecdcc3d P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:50.929967 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "294d7d20c2f545b8813e632aa058a60c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:50.930094 1169 ts_tablet_manager.cc:1403] T 075f0a151e024220b51ba7fcaecdcc3d P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.001s
W20251025 07:52:50.930549 850 leader_election.cc:343] T 294d7d20c2f545b8813e632aa058a60c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.930879 1169 raft_consensus.cc:359] T 075f0a151e024220b51ba7fcaecdcc3d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.931033 1169 raft_consensus.cc:385] T 075f0a151e024220b51ba7fcaecdcc3d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.931085 1169 raft_consensus.cc:740] T 075f0a151e024220b51ba7fcaecdcc3d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.931304 1169 consensus_queue.cc:260] T 075f0a151e024220b51ba7fcaecdcc3d P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.931950 1169 ts_tablet_manager.cc:1434] T 075f0a151e024220b51ba7fcaecdcc3d P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20251025 07:52:50.933547 442 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 6fe43ccecf7047319090ebd035fe24a5 (table default.Table_68b39cfaff194c3d8dbe9eaee7d8b44c [id=c95fc6d40893462eacee1276fb9cacfe], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6fe43ccecf7047319090ebd035fe24a5 consensus state has no leader
I20251025 07:52:50.934007 2166 ts_tablet_manager.cc:1929] T a7ab3150a35b4caca9119a1374d3f2f7 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:50.934125 2166 log.cc:1199] T a7ab3150a35b4caca9119a1374d3f2f7 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/a7ab3150a35b4caca9119a1374d3f2f7
I20251025 07:52:50.934432 2166 ts_tablet_manager.cc:1950] T a7ab3150a35b4caca9119a1374d3f2f7 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:50.935286 450 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 129) took 4322 ms. Trace:
I20251025 07:52:50.935091 1169 tablet_bootstrap.cc:492] T 5dab439bc50d4e869548154e1d222ad6 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:50.935479 450 rpcz_store.cc:276] 1025 07:52:46.613031 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:50.922200 (+4309169us) service_pool.cc:225] Handling call
1025 07:52:50.923840 (+ 1640us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:50.924403 (+ 563us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:50.934862 (+ 10459us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:50.934871 (+ 9us) write_op.cc:454] Released schema lock
1025 07:52:50.934971 (+ 100us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:50.935107 (+ 136us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:50.935231 (+ 124us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:50.924957 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:50.925056 (+ 99us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:50.925061 (+ 5us) write_op.cc:435] Acquired schema lock
1025 07:52:50.925064 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:50.925146 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:50.925159 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:50.925164 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:50.925252 (+ 88us) tablet.cc:699] Row locks acquired
1025 07:52:50.925254 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:50.925287 (+ 33us) write_op.cc:270] Start()
1025 07:52:50.925339 (+ 52us) write_op.cc:276] Timestamp: P: 1761378770925272 usec, L: 0
1025 07:52:50.925342 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:50.925508 (+ 166us) log.cc:844] Serialized 433 byte log entry
1025 07:52:50.930101 (+ 4593us) op_driver.cc:464] REPLICATION: finished
1025 07:52:50.931076 (+ 975us) write_op.cc:301] APPLY: starting
1025 07:52:50.931095 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:50.931147 (+ 52us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:50.931186 (+ 39us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:50.931368 (+ 182us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:50.931416 (+ 48us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:50.931428 (+ 12us) write_op.cc:312] APPLY: finished
1025 07:52:50.931512 (+ 84us) log.cc:844] Serialized 29 byte log entry
1025 07:52:50.931585 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:50.931612 (+ 27us) write_op.cc:454] Released schema lock
1025 07:52:50.931620 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":912,"apply.run_cpu_time_us":611,"apply.run_wall_time_us":611,"num_ops":2,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":665,"prepare.run_wall_time_us":664,"raft.queue_time_us":413,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":240,"replication_time_us":4728}]]}
I20251025 07:52:50.937327 450 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_3053388b8968455a8ea15bba9eab3060"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:50.937567 1169 tablet_bootstrap.cc:654] T 5dab439bc50d4e869548154e1d222ad6 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:50.938269 450 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3053388b8968455a8ea15bba9eab3060 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:50.938949 1039 raft_consensus.cc:3060] T 67cf96a7dd3f42e19b86746c87e689f6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.942263 2991 consensus_queue.cc:1048] T 67cf96a7dd3f42e19b86746c87e689f6 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:50.934737 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 250) took 6353 ms. Trace:
I20251025 07:52:50.945010 2166 rpcz_store.cc:276] 1025 07:52:44.581017 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.581061 (+ 44us) service_pool.cc:225] Handling call
1025 07:52:50.888477 (+6307416us) ts_tablet_manager.cc:1151] Deleting tablet a7ab3150a35b4caca9119a1374d3f2f7
1025 07:52:50.888486 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.892806 (+ 4320us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.933896 (+ 41090us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.934720 (+ 824us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6291759}
I20251025 07:52:50.945744 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet a7ab3150a35b4caca9119a1374d3f2f7 (table default.Table_8129e15bf5c549bd853c04b227a7ef26 [id=e9e8547061ba4ac987d0aa7616c7f3ea]) successfully deleted
I20251025 07:52:50.945982 3203 raft_consensus.cc:515] T 66134661be834f7b811d7927ab629812 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.947089 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66134661be834f7b811d7927ab629812" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:50.947150 3203 leader_election.cc:290] T 66134661be834f7b811d7927ab629812 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
W20251025 07:52:50.947580 850 leader_election.cc:343] T 66134661be834f7b811d7927ab629812 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.947693 450 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 130) took 4287 ms. Trace:
I20251025 07:52:50.947881 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 251) took 6360 ms. Trace:
I20251025 07:52:50.948004 2166 rpcz_store.cc:276] 1025 07:52:44.587774 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.587838 (+ 64us) service_pool.cc:225] Handling call
1025 07:52:50.946196 (+6358358us) ts_tablet_manager.cc:1151] Deleting tablet cb824bd780524832b2b60f5618a39da5
1025 07:52:50.946213 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.947846 (+ 1633us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6349416}
W20251025 07:52:50.948590 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet cb824bd780524832b2b60f5618a39da5 because the tablet was not found. No further retry: Not found: Tablet not found: cb824bd780524832b2b60f5618a39da5
I20251025 07:52:50.949137 2146 tablet_replica.cc:333] T ef1854d35a3047e3a9fbe0166ab1295a P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:50.949226 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 252) took 6361 ms. Trace:
I20251025 07:52:50.947886 450 rpcz_store.cc:276] 1025 07:52:46.660562 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:50.937080 (+4276518us) service_pool.cc:225] Handling call
1025 07:52:50.938407 (+ 1327us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:50.938887 (+ 480us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:50.947361 (+ 8474us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:50.947370 (+ 9us) write_op.cc:454] Released schema lock
1025 07:52:50.947444 (+ 74us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:50.947557 (+ 113us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:50.947652 (+ 95us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:50.939592 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:50.939724 (+ 132us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:50.939733 (+ 9us) write_op.cc:435] Acquired schema lock
1025 07:52:50.939738 (+ 5us) tablet.cc:660] Decoding operations
1025 07:52:50.939881 (+ 143us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:50.939899 (+ 18us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:50.939906 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:50.940007 (+ 101us) tablet.cc:699] Row locks acquired
1025 07:52:50.940011 (+ 4us) write_op.cc:260] PREPARE: finished
1025 07:52:50.940048 (+ 37us) write_op.cc:270] Start()
1025 07:52:50.940086 (+ 38us) write_op.cc:276] Timestamp: P: 1761378770940030 usec, L: 0
1025 07:52:50.940092 (+ 6us) op_driver.cc:348] REPLICATION: starting
1025 07:52:50.940309 (+ 217us) log.cc:844] Serialized 433 byte log entry
1025 07:52:50.944377 (+ 4068us) op_driver.cc:464] REPLICATION: finished
1025 07:52:50.945000 (+ 623us) write_op.cc:301] APPLY: starting
1025 07:52:50.945020 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:50.945065 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:50.945089 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:50.945231 (+ 142us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:50.945270 (+ 39us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:50.945278 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:50.945351 (+ 73us) log.cc:844] Serialized 29 byte log entry
1025 07:52:50.945412 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:50.945435 (+ 23us) write_op.cc:454] Released schema lock
1025 07:52:50.945445 (+ 10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":559,"apply.run_cpu_time_us":505,"apply.run_wall_time_us":505,"num_ops":2,"prepare.queue_time_us":59,"prepare.run_cpu_time_us":859,"prepare.run_wall_time_us":1235,"raft.queue_time_us":98,"raft.run_cpu_time_us":310,"raft.run_wall_time_us":310,"replication_time_us":4249}]]}
I20251025 07:52:50.949324 2166 rpcz_store.cc:276] 1025 07:52:44.587949 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.588081 (+ 132us) service_pool.cc:225] Handling call
1025 07:52:50.948947 (+6360866us) ts_tablet_manager.cc:1151] Deleting tablet f0bdaa8cfd764f1b83f47881353d26b6
1025 07:52:50.948960 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.949205 (+ 245us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6352123}
I20251025 07:52:50.949452 2146 raft_consensus.cc:2243] T ef1854d35a3047e3a9fbe0166ab1295a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251025 07:52:50.949776 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet f0bdaa8cfd764f1b83f47881353d26b6 because the tablet was not found. No further retry: Not found: Tablet not found: f0bdaa8cfd764f1b83f47881353d26b6
I20251025 07:52:50.949950 2146 raft_consensus.cc:2272] T ef1854d35a3047e3a9fbe0166ab1295a P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:50.950273 450 master_service.cc:954] ReplaceTablet: received request to replace tablet 3d5e0dd14e474fe09993a03851f8b1f0 from {username='slave'} at 127.0.0.1:47610
I20251025 07:52:50.950340 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 253) took 6362 ms. Trace:
I20251025 07:52:50.950438 2166 rpcz_store.cc:276] 1025 07:52:44.588090 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.588272 (+ 182us) service_pool.cc:225] Handling call
1025 07:52:50.950247 (+6361975us) ts_tablet_manager.cc:1151] Deleting tablet 0b21b7da1e2f47829c3f6cdaee689e8a
1025 07:52:50.950259 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.950323 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6353173}
I20251025 07:52:50.950604 2146 ts_tablet_manager.cc:1916] T ef1854d35a3047e3a9fbe0166ab1295a P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:50.952368 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 255) took 6364 ms. Trace:
I20251025 07:52:50.952445 2166 rpcz_store.cc:276] 1025 07:52:44.588362 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.588416 (+ 54us) service_pool.cc:225] Handling call
1025 07:52:50.952292 (+6363876us) ts_tablet_manager.cc:1151] Deleting tablet fa77085e81c7440d9e255c44d1e80c26
1025 07:52:50.952301 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.952353 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6355141}
I20251025 07:52:50.952816 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 254) took 6364 ms. Trace:
I20251025 07:52:50.952884 2166 rpcz_store.cc:276] 1025 07:52:44.588242 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.588531 (+ 289us) service_pool.cc:225] Handling call
1025 07:52:50.952745 (+6364214us) ts_tablet_manager.cc:1151] Deleting tablet a6ddefa1738f4ff98586900c344359d2
1025 07:52:50.952753 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.952800 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6355574}
I20251025 07:52:50.953213 2166 tablet_replica.cc:333] T 4b278ce7768349e49dcd1131e404ab34 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:50.953397 2166 raft_consensus.cc:2243] T 4b278ce7768349e49dcd1131e404ab34 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:50.953552 2166 raft_consensus.cc:2272] T 4b278ce7768349e49dcd1131e404ab34 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:50.954057 2166 ts_tablet_manager.cc:1916] T 4b278ce7768349e49dcd1131e404ab34 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:50.954650 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 11bfe0c87ca941a3846462c8a4512d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:44 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:50.954829 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 0551e5d694894e8fbeb66b5e0c1cd75b with delete_type TABLET_DATA_DELETED (replaced by tablet ae835c2078624c42a57ca63b7951dc79) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:50.955253 883 tablet_service.cc:1552] Processing DeleteTablet for tablet cb829649f73d4d3f8862d268c953bd86 with delete_type TABLET_DATA_DELETED (replaced by tablet e330e1b53bed4e7cace1c4fb7fa6b565) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:50.955657 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 110a8e9e4c6c434284c0ae14dd6987f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:45 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:50.956344 883 tablet_service.cc:1552] Processing DeleteTablet for tablet ebf3a2a926754305b5bd95fcebb25d29 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f05894c45154cd0b65f5b9bcef8cd4c) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:50.958508 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b278ce7768349e49dcd1131e404ab34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:44 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:50.960958 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 29dbaf9149134f548a974a5779f7da67 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:44 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:50.962582 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 570201e7bf05406a97c81dde071d894b with delete_type TABLET_DATA_DELETED (replaced by tablet 3a0f50c8000144a4acb5219484b01800) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:50.962741 847 leader_election.cc:304] T 8707b93d491e4e4386d148087a2f0870 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:50.963130 3203 raft_consensus.cc:2804] T 8707b93d491e4e4386d148087a2f0870 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:50.963313 3203 raft_consensus.cc:493] T 8707b93d491e4e4386d148087a2f0870 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:50.963474 3203 raft_consensus.cc:3060] T 8707b93d491e4e4386d148087a2f0870 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.966701 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 623dec4eab1d4666baf2334a01eeb36d with delete_type TABLET_DATA_DELETED (replaced by tablet b2ac640e69794a09811bd93aa02a22ea) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:50.968349 2166 ts_tablet_manager.cc:1929] T 4b278ce7768349e49dcd1131e404ab34 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:50.968497 2166 log.cc:1199] T 4b278ce7768349e49dcd1131e404ab34 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/4b278ce7768349e49dcd1131e404ab34
I20251025 07:52:50.968825 2166 ts_tablet_manager.cc:1950] T 4b278ce7768349e49dcd1131e404ab34 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:50.969688 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 256) took 6269 ms. Trace:
I20251025 07:52:50.969796 2166 rpcz_store.cc:276] 1025 07:52:44.699860 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.699999 (+ 139us) service_pool.cc:225] Handling call
1025 07:52:50.953169 (+6253170us) ts_tablet_manager.cc:1151] Deleting tablet 4b278ce7768349e49dcd1131e404ab34
1025 07:52:50.953177 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.957762 (+ 4585us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.968223 (+ 10461us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.969666 (+ 1443us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6205983}
I20251025 07:52:50.971004 2169 tablet_replica.cc:333] T a7ab3150a35b4caca9119a1374d3f2f7 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:50.971216 2169 raft_consensus.cc:2243] T a7ab3150a35b4caca9119a1374d3f2f7 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:50.971362 2169 raft_consensus.cc:2272] T a7ab3150a35b4caca9119a1374d3f2f7 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:50.971984 2169 ts_tablet_manager.cc:1916] T a7ab3150a35b4caca9119a1374d3f2f7 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:50.971249 847 consensus_peers.cc:597] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382 -> Peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Couldn't send request to peer 23eeffd3de254ce18db5febfde971c6b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e4da89f863cc43f3bf6df908bb9dc6bf. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:50.973424 2166 tablet_replica.cc:333] T 9f039d9107ae493486d799ab6a1d731d P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:50.974956 905 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32a143a715cb485cb4501f4e28c68252" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:50.975135 905 raft_consensus.cc:3060] T 32a143a715cb485cb4501f4e28c68252 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.977517 2166 raft_consensus.cc:2243] T 9f039d9107ae493486d799ab6a1d731d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:50.977723 2166 raft_consensus.cc:2272] T 9f039d9107ae493486d799ab6a1d731d P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:50.977909 905 raft_consensus.cc:2468] T 32a143a715cb485cb4501f4e28c68252 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:50.978267 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b71bdfd09a27449d98509bb220680147" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:50.978340 2166 ts_tablet_manager.cc:1916] T 9f039d9107ae493486d799ab6a1d731d P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:50.978485 912 raft_consensus.cc:2393] T b71bdfd09a27449d98509bb220680147 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 23eeffd3de254ce18db5febfde971c6b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:50.978685 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b22bd172e0e4f86896e9ecdef666b42" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:50.978806 910 raft_consensus.cc:3060] T 1b22bd172e0e4f86896e9ecdef666b42 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:50.981318 910 raft_consensus.cc:2468] T 1b22bd172e0e4f86896e9ecdef666b42 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:50.982576 3203 raft_consensus.cc:515] T 8707b93d491e4e4386d148087a2f0870 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.983603 3203 leader_election.cc:290] T 8707b93d491e4e4386d148087a2f0870 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:50.983582 1039 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8707b93d491e4e4386d148087a2f0870" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:50.984018 850 leader_election.cc:343] T 8707b93d491e4e4386d148087a2f0870 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:50.989780 2169 ts_tablet_manager.cc:1929] T a7ab3150a35b4caca9119a1374d3f2f7 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:50.989899 2169 log.cc:1199] T a7ab3150a35b4caca9119a1374d3f2f7 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/a7ab3150a35b4caca9119a1374d3f2f7
I20251025 07:52:50.990244 2169 ts_tablet_manager.cc:1950] T a7ab3150a35b4caca9119a1374d3f2f7 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:50.991914 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 9f039d9107ae493486d799ab6a1d731d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:44 UTC) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:50.992779 1167 tablet_bootstrap.cc:492] T 60c60a9ee37f43978910bd067a098bb7 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:50.992985 1167 ts_tablet_manager.cc:1403] T 60c60a9ee37f43978910bd067a098bb7 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.164s user 0.021s sys 0.011s
I20251025 07:52:50.993539 2169 tablet_replica.cc:333] T 9979cae195e849908677d78c6c33be71 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:50.994388 1167 raft_consensus.cc:359] T 60c60a9ee37f43978910bd067a098bb7 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.950544 455 catalog_manager.cc:6194] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: default.Table_cea16af97a2242968d72900d3a84a86d [id=d82b3e1317434cf08a0d51e8a9427224] alter complete (version 4)
I20251025 07:52:50.995380 2166 ts_tablet_manager.cc:1929] T 9f039d9107ae493486d799ab6a1d731d P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:50.995498 2166 log.cc:1199] T 9f039d9107ae493486d799ab6a1d731d P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/9f039d9107ae493486d799ab6a1d731d
W20251025 07:52:50.950894 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 0b21b7da1e2f47829c3f6cdaee689e8a because the tablet was not found. No further retry: Not found: Tablet not found: 0b21b7da1e2f47829c3f6cdaee689e8a
I20251025 07:52:50.995854 2166 ts_tablet_manager.cc:1950] T 9f039d9107ae493486d799ab6a1d731d P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
W20251025 07:52:50.995900 439 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:46231, user_credentials={real_user=slave}} blocked reactor thread for 45012.9us
W20251025 07:52:50.996244 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet fa77085e81c7440d9e255c44d1e80c26 because the tablet was not found. No further retry: Not found: Tablet not found: fa77085e81c7440d9e255c44d1e80c26
W20251025 07:52:50.996443 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet a6ddefa1738f4ff98586900c344359d2 because the tablet was not found. No further retry: Not found: Tablet not found: a6ddefa1738f4ff98586900c344359d2
I20251025 07:52:50.996596 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 4b278ce7768349e49dcd1131e404ab34 (table default.Table_309913c03a4d46019d35478363f149e8 [id=bad99c3045fc4b938e20d2dcfdd0b1bc]) successfully deleted
I20251025 07:52:50.996788 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 257) took 6252 ms. Trace:
I20251025 07:52:50.996888 2146 ts_tablet_manager.cc:1929] T ef1854d35a3047e3a9fbe0166ab1295a P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 07:52:50.996977 2146 log.cc:1199] T ef1854d35a3047e3a9fbe0166ab1295a P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/ef1854d35a3047e3a9fbe0166ab1295a
I20251025 07:52:50.996878 2166 rpcz_store.cc:276] 1025 07:52:44.744473 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.744575 (+ 102us) service_pool.cc:225] Handling call
1025 07:52:50.973359 (+6228784us) ts_tablet_manager.cc:1151] Deleting tablet 9f039d9107ae493486d799ab6a1d731d
1025 07:52:50.973370 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.985461 (+ 12091us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.995271 (+ 9810us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.996745 (+ 1474us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6222099}
I20251025 07:52:50.961748 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet fd265cdbf6a8440393a2b5d8240d1872 (table default.Table_434ef1ade0a14af987864563993afae8 [id=3a3af35e11944a6f97fc0144cb9d08d4]) successfully deleted
I20251025 07:52:50.997316 2146 ts_tablet_manager.cc:1950] T ef1854d35a3047e3a9fbe0166ab1295a P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:50.997299 1167 raft_consensus.cc:385] T 60c60a9ee37f43978910bd067a098bb7 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:50.997392 1167 raft_consensus.cc:740] T 60c60a9ee37f43978910bd067a098bb7 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:50.997417 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 9f039d9107ae493486d799ab6a1d731d (table default.Table_3b10c71d2c1b4bd2849e0cad9ec767d7 [id=ee09c3f16a194bbf97dd2768b5231c06]) successfully deleted
I20251025 07:52:50.997697 2166 tablet_replica.cc:333] T 29dbaf9149134f548a974a5779f7da67 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:50.997659 1167 consensus_queue.cc:260] T 60c60a9ee37f43978910bd067a098bb7 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:50.997783 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet 716fb61312e94a4d9eb8ab8336ca140c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:50.997893 2166 raft_consensus.cc:2243] T 29dbaf9149134f548a974a5779f7da67 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:50.998030 2166 pending_rounds.cc:70] T 29dbaf9149134f548a974a5779f7da67 P 23eeffd3de254ce18db5febfde971c6b: Trying to abort 1 pending ops.
I20251025 07:52:50.998011 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet dda0b26e4f1a44928bc4b84f333901c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:44572
W20251025 07:52:50.998011 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 2b594d4718344aafbdaec34040695589 because the tablet was not found. No further retry: Not found: Tablet not found: 2b594d4718344aafbdaec34040695589
I20251025 07:52:50.998095 2166 pending_rounds.cc:77] T 29dbaf9149134f548a974a5779f7da67 P 23eeffd3de254ce18db5febfde971c6b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214607436690837504 op_type: NO_OP noop_request { }
I20251025 07:52:50.998217 2166 raft_consensus.cc:2889] T 29dbaf9149134f548a974a5779f7da67 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
W20251025 07:52:50.998257 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 570201e7bf05406a97c81dde071d894b because the tablet was not found. No further retry: Not found: Tablet not found: 570201e7bf05406a97c81dde071d894b
I20251025 07:52:50.998313 2166 raft_consensus.cc:2272] T 29dbaf9149134f548a974a5779f7da67 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:50.993736 2169 raft_consensus.cc:2243] T 9979cae195e849908677d78c6c33be71 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:50.998459 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 80ee75935d7e461d9d4066fac9819bef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:44572
W20251025 07:52:50.998487 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet fa998b9ca6724c4cb1eb68a3eecb0b11 because the tablet was not found. No further retry: Not found: Tablet not found: fa998b9ca6724c4cb1eb68a3eecb0b11
I20251025 07:52:50.998605 2169 raft_consensus.cc:2272] T 9979cae195e849908677d78c6c33be71 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:50.998670 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet ff3daed6c9af47e3976b82b4b9a41086 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff3daed6c9af47e3976b82b4b9a41086 already in progress: creating tablet
I20251025 07:52:50.998847 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet a7ab3150a35b4caca9119a1374d3f2f7 (table default.Table_8129e15bf5c549bd853c04b227a7ef26 [id=e9e8547061ba4ac987d0aa7616c7f3ea]) successfully deleted
I20251025 07:52:50.998955 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e6544202d4f489d82f22d6073ef240d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
W20251025 07:52:50.999028 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet c88c46e022ff41a58b02538f2151e260 because the tablet was not found. No further retry: Not found: Tablet not found: c88c46e022ff41a58b02538f2151e260
I20251025 07:52:50.999094 2169 ts_tablet_manager.cc:1916] T 9979cae195e849908677d78c6c33be71 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:50.999213 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 53198283718d4b45b30df65b5bde96a6 because the tablet was not found. No further retry: Not found: Tablet not found: 53198283718d4b45b30df65b5bde96a6
I20251025 07:52:50.999269 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet ffe7c8e1d6c44872b7b65e512764a56a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
W20251025 07:52:50.999398 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 08da9d2687704725aacd11cd92c6a9f9 because the tablet was not found. No further retry: Not found: Tablet not found: 08da9d2687704725aacd11cd92c6a9f9
W20251025 07:52:50.999608 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 7fbae8be73bf4e18b5a5c4ec4cdebc12 because the tablet was not found. No further retry: Not found: Tablet not found: 7fbae8be73bf4e18b5a5c4ec4cdebc12
I20251025 07:52:50.999666 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f0c4f0e8f014e59be9e46d2afe0be9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
W20251025 07:52:50.999799 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 3abfd52327a04ba88625f391a8658fbb because the tablet was not found. No further retry: Not found: Tablet not found: 3abfd52327a04ba88625f391a8658fbb
W20251025 07:52:50.999998 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 6ec421eb820b4f40afcff5a5bf80c508 because the tablet was not found. No further retry: Not found: Tablet not found: 6ec421eb820b4f40afcff5a5bf80c508
I20251025 07:52:51.000108 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet 80fe3255ed614270bd8352a2853fbc43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:44572
W20251025 07:52:51.000183 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 9d6a9937d10a4cc2813671dd1bfa1664 because the tablet was not found. No further retry: Not found: Tablet not found: 9d6a9937d10a4cc2813671dd1bfa1664
I20251025 07:52:51.000314 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b6383c99e9e4c90a374da9ccd830a6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
W20251025 07:52:51.000388 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet d5e5e5bbcf9f43118eea502b69880af5 because the tablet was not found. No further retry: Not found: Tablet not found: d5e5e5bbcf9f43118eea502b69880af5
W20251025 07:52:51.000571 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet f3c87a83082b421cb4be546ac4ce7ad8 because the tablet was not found. No further retry: Not found: Tablet not found: f3c87a83082b421cb4be546ac4ce7ad8
I20251025 07:52:51.000608 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet fce535a0c0bd45c6bf17f81add297f40 with delete_type TABLET_DATA_DELETED (replaced by tablet 079f91be4217493d8ab865f5ccce0cb1) from {username='slave'} at 127.0.0.1:44572
W20251025 07:52:51.000782 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 551d07a9242740dcb9366bb2d707a4a8 because the tablet was not found. No further retry: Not found: Tablet not found: 551d07a9242740dcb9366bb2d707a4a8
I20251025 07:52:51.000941 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet ebc824ec4c60411dab1a2ea4a385148e with delete_type TABLET_DATA_DELETED (replaced by tablet 29dbaf9149134f548a974a5779f7da67) from {username='slave'} at 127.0.0.1:44572
W20251025 07:52:51.000960 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 4b28215f61574cf48c447dd01955ac77 because the tablet was not found. No further retry: Not found: Tablet not found: 4b28215f61574cf48c447dd01955ac77
W20251025 07:52:51.001157 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet d66ad049d5484228a268b0facb00ecf1 because the tablet was not found. No further retry: Not found: Tablet not found: d66ad049d5484228a268b0facb00ecf1
W20251025 07:52:51.001330 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet af743ee344e543a0aa6da3cd7a30e467 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af743ee344e543a0aa6da3cd7a30e467 already in progress: creating tablet
W20251025 07:52:51.001554 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 34315f64b89148efbf1f20a43d5c5efb because the tablet was not found. No further retry: Not found: Tablet not found: 34315f64b89148efbf1f20a43d5c5efb
W20251025 07:52:51.001729 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet a4e4b99be0434baaa75d988281797846 because the tablet was not found. No further retry: Not found: Tablet not found: a4e4b99be0434baaa75d988281797846
W20251025 07:52:51.001910 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 92aa605fabbc4257a6a204c087e7cc7f because the tablet was not found. No further retry: Not found: Tablet not found: 92aa605fabbc4257a6a204c087e7cc7f
W20251025 07:52:51.002110 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet b94634ac5077439fadfbbce569f60e8b because the tablet was not found. No further retry: Not found: Tablet not found: b94634ac5077439fadfbbce569f60e8b
W20251025 07:52:51.002957 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet e54fe7cbcffa40c4b04144b47bee2b48 because the tablet was not found. No further retry: Not found: Tablet not found: e54fe7cbcffa40c4b04144b47bee2b48
I20251025 07:52:51.003049 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b71bdfd09a27449d98509bb220680147" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:51.003260 773 raft_consensus.cc:3060] T b71bdfd09a27449d98509bb220680147 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:51.003656 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 0063a0776b2247afaa8be82f6eff26f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0063a0776b2247afaa8be82f6eff26f1 already in progress: creating tablet
I20251025 07:52:51.004415 2166 ts_tablet_manager.cc:1916] T 29dbaf9149134f548a974a5779f7da67 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:50.966804 454 catalog_manager.cc:6194] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: default.Table_76158f8cf82e4cdb9a90580e46aaea48 [id=f249f30497bc450c887d659721469de8] alter complete (version 1)
I20251025 07:52:50.969635 452 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet a6ae1315b28940d49b667e3c80347021
I20251025 07:52:51.005913 452 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:47610 (request call id 123) took 4855 ms. Trace:
I20251025 07:52:51.005936 773 raft_consensus.cc:2468] T b71bdfd09a27449d98509bb220680147 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:51.006229 755 tablet_service.cc:1552] Processing DeleteTablet for tablet a6ae1315b28940d49b667e3c80347021 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:50 UTC) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:51.006004 452 rpcz_store.cc:276] 1025 07:52:46.150857 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:50.296482 (+4145625us) service_pool.cc:225] Handling call
1025 07:52:50.329467 (+ 32985us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1025 07:52:50.448482 (+119015us) catalog_manager.cc:2815] Modifying in-memory table state
1025 07:52:50.448546 (+ 64us) catalog_manager.cc:2824] Locking tablets
1025 07:52:50.538312 (+ 89766us) catalog_manager.cc:2838] Removing table and tablets from system table
1025 07:52:50.546168 (+ 7856us) catalog_manager.cc:2856] Removing table from by-name map
1025 07:52:50.546249 (+ 81us) catalog_manager.cc:2872] Committing in-memory state
1025 07:52:50.969583 (+423334us) catalog_manager.cc:2876] Aborting table tasks
1025 07:52:51.005853 (+ 36270us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:50.539060 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:50.539182 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:50.539190 (+ 8us) write_op.cc:435] Acquired schema lock
1025 07:52:50.539192 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:50.539302 (+ 110us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:50.539316 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:50.539321 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:50.539424 (+ 103us) tablet.cc:699] Row locks acquired
1025 07:52:50.539427 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:50.539461 (+ 34us) write_op.cc:270] Start()
1025 07:52:50.539493 (+ 32us) write_op.cc:276] Timestamp: P: 1761378770539444 usec, L: 0
1025 07:52:50.539496 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:50.539678 (+ 182us) log.cc:844] Serialized 725 byte log entry
1025 07:52:50.543863 (+ 4185us) op_driver.cc:464] REPLICATION: finished
1025 07:52:50.543993 (+ 130us) write_op.cc:301] APPLY: starting
1025 07:52:50.544010 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:50.544054 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:50.544087 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:50.544152 (+ 65us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:50.544183 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1025 07:52:50.544191 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:50.544287 (+ 96us) log.cc:844] Serialized 29 byte log entry
1025 07:52:50.544349 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:50.544371 (+ 22us) write_op.cc:454] Released schema lock
1025 07:52:50.544377 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:50.544442 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:50.544445 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":86,"apply.run_cpu_time_us":439,"apply.run_wall_time_us":439,"num_ops":2,"prepare.queue_time_us":51,"prepare.run_cpu_time_us":728,"prepare.run_wall_time_us":728,"raft.queue_time_us":400,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":246,"replication_time_us":4340}]]}
I20251025 07:52:51.006856 764 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "294d7d20c2f545b8813e632aa058a60c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:51.007032 764 raft_consensus.cc:2468] T 294d7d20c2f545b8813e632aa058a60c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:51.007630 764 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66134661be834f7b811d7927ab629812" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:51.007804 764 raft_consensus.cc:3060] T 66134661be834f7b811d7927ab629812 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:51.009879 1167 ts_tablet_manager.cc:1434] T 60c60a9ee37f43978910bd067a098bb7 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20251025 07:52:51.010577 764 raft_consensus.cc:2468] T 66134661be834f7b811d7927ab629812 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:51.011739 883 tablet_service.cc:1552] Processing DeleteTablet for tablet a6ae1315b28940d49b667e3c80347021 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:50 UTC) from {username='slave'} at 127.0.0.1:59616
W20251025 07:52:51.012197 850 consensus_peers.cc:597] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382 -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:51.013022 2169 ts_tablet_manager.cc:1929] T 9979cae195e849908677d78c6c33be71 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:51.013135 2169 log.cc:1199] T 9979cae195e849908677d78c6c33be71 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/9979cae195e849908677d78c6c33be71
I20251025 07:52:51.013484 2169 ts_tablet_manager.cc:1950] T 9979cae195e849908677d78c6c33be71 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:51.014413 2166 ts_tablet_manager.cc:1929] T 29dbaf9149134f548a974a5779f7da67 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:51.014544 2166 log.cc:1199] T 29dbaf9149134f548a974a5779f7da67 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/29dbaf9149134f548a974a5779f7da67
I20251025 07:52:51.014894 2166 ts_tablet_manager.cc:1950] T 29dbaf9149134f548a974a5779f7da67 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:51.017796 710 leader_election.cc:304] T 32a143a715cb485cb4501f4e28c68252 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:51.018201 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8707b93d491e4e4386d148087a2f0870" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:51.018292 710 leader_election.cc:304] T b71bdfd09a27449d98509bb220680147 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:51.018396 773 raft_consensus.cc:3060] T 8707b93d491e4e4386d148087a2f0870 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:51.018689 710 leader_election.cc:304] T 1b22bd172e0e4f86896e9ecdef666b42 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:51.018900 3236 raft_consensus.cc:2804] T 32a143a715cb485cb4501f4e28c68252 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:51.019037 3142 raft_consensus.cc:2749] T b71bdfd09a27449d98509bb220680147 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:51.019202 3235 raft_consensus.cc:2804] T 1b22bd172e0e4f86896e9ecdef666b42 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:51.019325 3235 raft_consensus.cc:697] T 1b22bd172e0e4f86896e9ecdef666b42 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:51.019052 3236 raft_consensus.cc:697] T 32a143a715cb485cb4501f4e28c68252 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:51.019598 3235 consensus_queue.cc:237] T 1b22bd172e0e4f86896e9ecdef666b42 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.019799 3236 consensus_queue.cc:237] T 32a143a715cb485cb4501f4e28c68252 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.019913 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 259) took 6233 ms. Trace:
I20251025 07:52:51.020015 2166 rpcz_store.cc:276] 1025 07:52:44.786398 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.786482 (+ 84us) service_pool.cc:225] Handling call
1025 07:52:50.997626 (+6211144us) ts_tablet_manager.cc:1151] Deleting tablet 29dbaf9149134f548a974a5779f7da67
1025 07:52:50.997637 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.007843 (+ 10206us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.010717 (+ 2874us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.019869 (+ 9152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6204992}
I20251025 07:52:51.020433 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 29dbaf9149134f548a974a5779f7da67 (table default.Table_c43ac1adc76d4ccf819411ec76775932 [id=106e3728b0b846178eb48ccbf076e24e]) successfully deleted
I20251025 07:52:51.021134 3143 raft_consensus.cc:493] T 243b8828fc344bacab46d7bab6bd2dec P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:51.021279 3143 raft_consensus.cc:515] T 243b8828fc344bacab46d7bab6bd2dec P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.021850 3143 leader_election.cc:290] T 243b8828fc344bacab46d7bab6bd2dec P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:51.022189 2166 tablet_replica.cc:333] T 11bfe0c87ca941a3846462c8a4512d83 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:51.022215 773 raft_consensus.cc:2468] T 8707b93d491e4e4386d148087a2f0870 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:51.022424 2166 raft_consensus.cc:2243] T 11bfe0c87ca941a3846462c8a4512d83 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:51.022428 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "243b8828fc344bacab46d7bab6bd2dec" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:51.022605 2166 raft_consensus.cc:2272] T 11bfe0c87ca941a3846462c8a4512d83 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:51.023192 2166 ts_tablet_manager.cc:1916] T 11bfe0c87ca941a3846462c8a4512d83 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:51.024240 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet a6ae1315b28940d49b667e3c80347021 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:50 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.024471 765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "243b8828fc344bacab46d7bab6bd2dec" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:51.024722 765 raft_consensus.cc:2393] T 243b8828fc344bacab46d7bab6bd2dec P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db42be7ad3c741a0aa427f27e245993b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:51.024673 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 9979cae195e849908677d78c6c33be71 (table default.Table_208cd3f3c07744ed92b1260ebb401310 [id=d6c5598979e84e36a3bf1996f37f9c28]) successfully deleted
W20251025 07:52:51.024936 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 716fb61312e94a4d9eb8ab8336ca140c because the tablet was not found. No further retry: Not found: Tablet not found: 716fb61312e94a4d9eb8ab8336ca140c
I20251025 07:52:51.025013 982 leader_election.cc:304] T 243b8828fc344bacab46d7bab6bd2dec P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
W20251025 07:52:51.025641 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet dda0b26e4f1a44928bc4b84f333901c0 because the tablet was not found. No further retry: Not found: Tablet not found: dda0b26e4f1a44928bc4b84f333901c0
I20251025 07:52:51.025910 3143 raft_consensus.cc:3060] T 243b8828fc344bacab46d7bab6bd2dec P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:51.026254 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 80ee75935d7e461d9d4066fac9819bef because the tablet was not found. No further retry: Not found: Tablet not found: 80ee75935d7e461d9d4066fac9819bef
I20251025 07:52:51.028635 3143 raft_consensus.cc:2749] T 243b8828fc344bacab46d7bab6bd2dec P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251025 07:52:51.026464 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 7e6544202d4f489d82f22d6073ef240d because the tablet was not found. No further retry: Not found: Tablet not found: 7e6544202d4f489d82f22d6073ef240d
W20251025 07:52:51.029067 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet ffe7c8e1d6c44872b7b65e512764a56a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ffe7c8e1d6c44872b7b65e512764a56a already in progress: creating tablet
W20251025 07:52:51.029506 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 4f0c4f0e8f014e59be9e46d2afe0be9d because the tablet was not found. No further retry: Not found: Tablet not found: 4f0c4f0e8f014e59be9e46d2afe0be9d
W20251025 07:52:51.029865 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 80fe3255ed614270bd8352a2853fbc43 because the tablet was not found. No further retry: Not found: Tablet not found: 80fe3255ed614270bd8352a2853fbc43
W20251025 07:52:51.030256 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 5b6383c99e9e4c90a374da9ccd830a6f because the tablet was not found. No further retry: Not found: Tablet not found: 5b6383c99e9e4c90a374da9ccd830a6f
I20251025 07:52:51.030303 2988 consensus_queue.cc:1048] T d0240d20f9904b4bbc8bf7f0f05e8850 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251025 07:52:51.030726 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet fce535a0c0bd45c6bf17f81add297f40 because the tablet was not found. No further retry: Not found: Tablet not found: fce535a0c0bd45c6bf17f81add297f40
W20251025 07:52:51.030915 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet ebc824ec4c60411dab1a2ea4a385148e because the tablet was not found. No further retry: Not found: Tablet not found: ebc824ec4c60411dab1a2ea4a385148e
W20251025 07:52:51.031098 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet a6ae1315b28940d49b667e3c80347021 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6ae1315b28940d49b667e3c80347021 already in progress: creating tablet
I20251025 07:52:51.031985 1166 tablet_bootstrap.cc:492] T f7011f1dcad74fe4ba0356c618890e06 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:51.032202 1166 ts_tablet_manager.cc:1403] T f7011f1dcad74fe4ba0356c618890e06 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.229s user 0.004s sys 0.002s
I20251025 07:52:51.033170 1166 raft_consensus.cc:359] T f7011f1dcad74fe4ba0356c618890e06 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.033411 1166 raft_consensus.cc:385] T f7011f1dcad74fe4ba0356c618890e06 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:51.033512 1166 raft_consensus.cc:740] T f7011f1dcad74fe4ba0356c618890e06 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:51.033792 1166 consensus_queue.cc:260] T f7011f1dcad74fe4ba0356c618890e06 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.034377 1166 ts_tablet_manager.cc:1434] T f7011f1dcad74fe4ba0356c618890e06 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:51.034453 2166 ts_tablet_manager.cc:1929] T 11bfe0c87ca941a3846462c8a4512d83 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:51.034626 2166 log.cc:1199] T 11bfe0c87ca941a3846462c8a4512d83 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/11bfe0c87ca941a3846462c8a4512d83
I20251025 07:52:51.034787 1166 tablet_bootstrap.cc:492] T dbb254115a574bf4976aff0ea56f3bd9 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:51.035305 2166 ts_tablet_manager.cc:1950] T 11bfe0c87ca941a3846462c8a4512d83 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:51.036162 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 262) took 6236 ms. Trace:
I20251025 07:52:51.036275 2166 rpcz_store.cc:276] 1025 07:52:44.799549 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.799822 (+ 273us) service_pool.cc:225] Handling call
1025 07:52:51.022073 (+6222251us) ts_tablet_manager.cc:1151] Deleting tablet 11bfe0c87ca941a3846462c8a4512d83
1025 07:52:51.022082 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.029870 (+ 7788us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.034285 (+ 4415us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.036124 (+ 1839us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6222084}
I20251025 07:52:51.036728 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 11bfe0c87ca941a3846462c8a4512d83 (table default.Table_00b3b394e5ed46faaac0202afe54b437 [id=304ef5bba5ed4e198622b8c8dc17f43a]) successfully deleted
I20251025 07:52:51.037134 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 263) took 6135 ms. Trace:
I20251025 07:52:51.037170 1166 tablet_bootstrap.cc:654] T dbb254115a574bf4976aff0ea56f3bd9 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:51.037285 2166 rpcz_store.cc:276] 1025 07:52:44.901842 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.901903 (+ 61us) service_pool.cc:225] Handling call
1025 07:52:51.037016 (+6135113us) ts_tablet_manager.cc:1151] Deleting tablet 8c76d1b509f04ab1a10b83cff675c11e
1025 07:52:51.037026 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.037091 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6131272}
W20251025 07:52:51.037789 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 8c76d1b509f04ab1a10b83cff675c11e because the tablet was not found. No further retry: Not found: Tablet not found: 8c76d1b509f04ab1a10b83cff675c11e
I20251025 07:52:51.038273 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 264) took 6132 ms. Trace:
I20251025 07:52:51.038472 2166 rpcz_store.cc:276] 1025 07:52:44.905423 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:44.905478 (+ 55us) service_pool.cc:225] Handling call
1025 07:52:51.038155 (+6132677us) ts_tablet_manager.cc:1151] Deleting tablet b772025852794a49b072f37cccbd3291
1025 07:52:51.038169 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.038236 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6132323}
W20251025 07:52:51.038970 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet b772025852794a49b072f37cccbd3291 because the tablet was not found. No further retry: Not found: Tablet not found: b772025852794a49b072f37cccbd3291
I20251025 07:52:51.039348 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 265) took 6034 ms. Trace:
I20251025 07:52:51.039484 2166 rpcz_store.cc:276] 1025 07:52:45.004915 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.004975 (+ 60us) service_pool.cc:225] Handling call
1025 07:52:51.039232 (+6034257us) ts_tablet_manager.cc:1151] Deleting tablet 4037dee385d04e85bb78c8e77cb00379
1025 07:52:51.039240 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.039309 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6034054}
I20251025 07:52:51.039707 1167 tablet_bootstrap.cc:492] T a6587f4be6af44dd944e6b6f9599dfda P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
W20251025 07:52:51.039908 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 4037dee385d04e85bb78c8e77cb00379 because the tablet was not found. No further retry: Not found: Tablet not found: 4037dee385d04e85bb78c8e77cb00379
I20251025 07:52:51.040228 2166 tablet_replica.cc:333] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:51.040539 2166 raft_consensus.cc:2243] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:51.040944 2166 raft_consensus.cc:2272] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:51.041270 960 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
I20251025 07:52:51.041363 1167 tablet_bootstrap.cc:654] T a6587f4be6af44dd944e6b6f9599dfda P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:51.041687 2166 ts_tablet_manager.cc:1916] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:51.049811 2166 ts_tablet_manager.cc:1929] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:51.050232 2166 log.cc:1199] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/0551e5d694894e8fbeb66b5e0c1cd75b
I20251025 07:52:51.050907 2166 ts_tablet_manager.cc:1950] T 0551e5d694894e8fbeb66b5e0c1cd75b P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:51.051142 1167 tablet_bootstrap.cc:492] T a6587f4be6af44dd944e6b6f9599dfda P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:51.051744 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 267) took 5421 ms. Trace:
I20251025 07:52:51.052157 2166 rpcz_store.cc:276] 1025 07:52:45.629782 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.629836 (+ 54us) service_pool.cc:225] Handling call
1025 07:52:51.040156 (+5410320us) ts_tablet_manager.cc:1151] Deleting tablet 0551e5d694894e8fbeb66b5e0c1cd75b
1025 07:52:51.040165 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.044973 (+ 4808us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.049690 (+ 4717us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.051711 (+ 2021us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5410149}
I20251025 07:52:51.052862 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 0551e5d694894e8fbeb66b5e0c1cd75b (table default.Table_2cd70245aa3d4809a0647e7fa499f62f [id=a003268ec3da425989fa4041699de4d3]) successfully deleted
I20251025 07:52:51.053251 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 268) took 5403 ms. Trace:
I20251025 07:52:51.053357 2166 rpcz_store.cc:276] 1025 07:52:45.650191 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.650238 (+ 47us) service_pool.cc:225] Handling call
1025 07:52:51.053119 (+5402881us) ts_tablet_manager.cc:1151] Deleting tablet 110a8e9e4c6c434284c0ae14dd6987f7
1025 07:52:51.053128 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.053219 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5402693}
W20251025 07:52:51.053762 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 110a8e9e4c6c434284c0ae14dd6987f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 110a8e9e4c6c434284c0ae14dd6987f7 already in progress: creating tablet
I20251025 07:52:51.054116 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 269) took 5385 ms. Trace:
I20251025 07:52:51.054240 2166 rpcz_store.cc:276] 1025 07:52:45.668483 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.668538 (+ 55us) service_pool.cc:225] Handling call
1025 07:52:51.053998 (+5385460us) ts_tablet_manager.cc:1151] Deleting tablet 623dec4eab1d4666baf2334a01eeb36d
1025 07:52:51.054007 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.054083 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5385285}
W20251025 07:52:51.054637 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 623dec4eab1d4666baf2334a01eeb36d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 623dec4eab1d4666baf2334a01eeb36d already in progress: creating tablet
I20251025 07:52:51.054913 2166 tablet_replica.cc:333] T cb829649f73d4d3f8862d268c953bd86 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:51.055117 2166 raft_consensus.cc:2243] T cb829649f73d4d3f8862d268c953bd86 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:51.055419 2166 raft_consensus.cc:2272] T cb829649f73d4d3f8862d268c953bd86 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:51.055965 2166 ts_tablet_manager.cc:1916] T cb829649f73d4d3f8862d268c953bd86 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:51.057096 1169 tablet_bootstrap.cc:492] T 5dab439bc50d4e869548154e1d222ad6 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:51.057240 1169 ts_tablet_manager.cc:1403] T 5dab439bc50d4e869548154e1d222ad6 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.122s user 0.005s sys 0.004s
I20251025 07:52:51.057971 1169 raft_consensus.cc:359] T 5dab439bc50d4e869548154e1d222ad6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.058233 1169 raft_consensus.cc:385] T 5dab439bc50d4e869548154e1d222ad6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:51.058300 1169 raft_consensus.cc:740] T 5dab439bc50d4e869548154e1d222ad6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:51.058542 1169 consensus_queue.cc:260] T 5dab439bc50d4e869548154e1d222ad6 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.062266 1167 ts_tablet_manager.cc:1403] T a6587f4be6af44dd944e6b6f9599dfda P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.004s
I20251025 07:52:51.062765 450 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet 3d5e0dd14e474fe09993a03851f8b1f0
I20251025 07:52:51.062965 1167 raft_consensus.cc:359] T a6587f4be6af44dd944e6b6f9599dfda P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.063153 1167 raft_consensus.cc:385] T a6587f4be6af44dd944e6b6f9599dfda P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:51.063259 1167 raft_consensus.cc:740] T a6587f4be6af44dd944e6b6f9599dfda P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:51.063486 1167 consensus_queue.cc:260] T a6587f4be6af44dd944e6b6f9599dfda P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.064803 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 3d5e0dd14e474fe09993a03851f8b1f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 33a8f7c5726844f3930f8fa5983d5f27) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.065066 2169 tablet_replica.cc:333] T 3d5e0dd14e474fe09993a03851f8b1f0 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:51.065271 2169 raft_consensus.cc:2243] T 3d5e0dd14e474fe09993a03851f8b1f0 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:51.065459 2169 raft_consensus.cc:2272] T 3d5e0dd14e474fe09993a03851f8b1f0 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:51.065937 2169 ts_tablet_manager.cc:1916] T 3d5e0dd14e474fe09993a03851f8b1f0 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:51.073388 2169 ts_tablet_manager.cc:1929] T 3d5e0dd14e474fe09993a03851f8b1f0 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251025 07:52:51.073581 2169 log.cc:1199] T 3d5e0dd14e474fe09993a03851f8b1f0 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/3d5e0dd14e474fe09993a03851f8b1f0
I20251025 07:52:51.074009 2169 ts_tablet_manager.cc:1950] T 3d5e0dd14e474fe09993a03851f8b1f0 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:51.064014 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 3d5e0dd14e474fe09993a03851f8b1f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 33a8f7c5726844f3930f8fa5983d5f27) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:51.075949 1167 ts_tablet_manager.cc:1434] T a6587f4be6af44dd944e6b6f9599dfda P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20251025 07:52:51.076277 1167 tablet_bootstrap.cc:492] T 2c4789b9d89f42cb844e6cb595e687db P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:51.076421 3122 raft_consensus.cc:493] T 9903e80a26fd469a8744dc855534ba75 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:51.076622 3122 raft_consensus.cc:515] T 9903e80a26fd469a8744dc855534ba75 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.077234 3122 leader_election.cc:290] T 9903e80a26fd469a8744dc855534ba75 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:51.078063 1167 tablet_bootstrap.cc:654] T 2c4789b9d89f42cb844e6cb595e687db P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:51.078202 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:46231, user_credentials={real_user=slave}} blocked reactor thread for 51528.5us
I20251025 07:52:51.078460 2988 consensus_queue.cc:1048] T 6ce62847e80f48e8807fdbbbdc50803c P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:51.079114 2988 consensus_queue.cc:1048] T 67cf96a7dd3f42e19b86746c87e689f6 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:51.081331 1039 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9903e80a26fd469a8744dc855534ba75" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:51.082172 847 leader_election.cc:304] T b71bdfd09a27449d98509bb220680147 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:51.082692 1169 ts_tablet_manager.cc:1434] T 5dab439bc50d4e869548154e1d222ad6 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20251025 07:52:51.083020 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 3d5e0dd14e474fe09993a03851f8b1f0 (table default.Table_cd9190d6cb714bdbb763dd934b33c10a [id=1b5db2f5053c4cb3a3b40eb109d5d5d9]) successfully deleted
I20251025 07:52:51.083305 2166 ts_tablet_manager.cc:1929] T cb829649f73d4d3f8862d268c953bd86 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 07:52:51.083420 2166 log.cc:1199] T cb829649f73d4d3f8862d268c953bd86 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/cb829649f73d4d3f8862d268c953bd86
I20251025 07:52:51.083777 2166 ts_tablet_manager.cc:1950] T cb829649f73d4d3f8862d268c953bd86 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:51.084502 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 270) took 5372 ms. Trace:
I20251025 07:52:51.084587 2166 rpcz_store.cc:276] 1025 07:52:45.712243 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.712375 (+ 132us) service_pool.cc:225] Handling call
1025 07:52:51.054858 (+5342483us) ts_tablet_manager.cc:1151] Deleting tablet cb829649f73d4d3f8862d268c953bd86
1025 07:52:51.054866 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.059198 (+ 4332us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.083128 (+ 23930us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.084465 (+ 1337us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5342317}
I20251025 07:52:51.085000 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet cb829649f73d4d3f8862d268c953bd86 (table default.Table_a9026f5e095c4063b86d4c544c686e22 [id=7944225d3ce7439b881bfb9289d9f040]) successfully deleted
I20251025 07:52:51.085394 1169 tablet_bootstrap.cc:492] T d46673a283fe42b78ddeb99fbd2678eb P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:51.085672 450 catalog_manager.cc:6719] ReplaceTablet: tablet 3d5e0dd14e474fe09993a03851f8b1f0 deleted and replaced by tablet 33a8f7c5726844f3930f8fa5983d5f27
I20251025 07:52:51.085808 450 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:47610 (request call id 131) took 4291 ms. Trace:
I20251025 07:52:51.085884 450 rpcz_store.cc:276] 1025 07:52:46.794430 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:50.950178 (+4155748us) service_pool.cc:225] Handling call
1025 07:52:51.062388 (+112210us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:51.062393 (+ 5us) write_op.cc:454] Released schema lock
1025 07:52:51.085756 (+ 23363us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:51.054685 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:51.054827 (+ 142us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:51.054834 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:51.054837 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:51.054968 (+ 131us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:51.054983 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:51.054989 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:51.055082 (+ 93us) tablet.cc:699] Row locks acquired
1025 07:52:51.055086 (+ 4us) write_op.cc:260] PREPARE: finished
1025 07:52:51.055120 (+ 34us) write_op.cc:270] Start()
1025 07:52:51.055152 (+ 32us) write_op.cc:276] Timestamp: P: 1761378771055104 usec, L: 0
1025 07:52:51.055157 (+ 5us) op_driver.cc:348] REPLICATION: starting
1025 07:52:51.055354 (+ 197us) log.cc:844] Serialized 656 byte log entry
1025 07:52:51.058923 (+ 3569us) op_driver.cc:464] REPLICATION: finished
1025 07:52:51.059550 (+ 627us) write_op.cc:301] APPLY: starting
1025 07:52:51.059564 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:51.059605 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:51.059636 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:51.059731 (+ 95us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:51.059779 (+ 48us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:51.059786 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:51.059849 (+ 63us) log.cc:844] Serialized 29 byte log entry
1025 07:52:51.059913 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:51.059933 (+ 20us) write_op.cc:454] Released schema lock
1025 07:52:51.059939 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":574,"apply.run_cpu_time_us":440,"apply.run_wall_time_us":440,"num_ops":2,"prepare.queue_time_us":59,"prepare.run_cpu_time_us":799,"prepare.run_wall_time_us":1448,"raft.queue_time_us":67,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":228,"replication_time_us":3740}]]}
I20251025 07:52:51.087117 1169 tablet_bootstrap.cc:654] T d46673a283fe42b78ddeb99fbd2678eb P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:51.088425 2166 tablet_replica.cc:333] T ebf3a2a926754305b5bd95fcebb25d29 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:51.088645 2166 raft_consensus.cc:2243] T ebf3a2a926754305b5bd95fcebb25d29 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:51.088811 2166 raft_consensus.cc:2272] T ebf3a2a926754305b5bd95fcebb25d29 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:51.089335 2166 ts_tablet_manager.cc:1916] T ebf3a2a926754305b5bd95fcebb25d29 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:51.098766 448 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet 792f32cbd66449ca98813c669b2a261d
I20251025 07:52:51.099400 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 792f32cbd66449ca98813c669b2a261d with delete_type TABLET_DATA_DELETED (replaced by tablet bc35a078052749cfa6a61b0ae3f5cac5) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:51.100006 448 catalog_manager.cc:6719] ReplaceTablet: tablet 792f32cbd66449ca98813c669b2a261d deleted and replaced by tablet bc35a078052749cfa6a61b0ae3f5cac5
I20251025 07:52:51.100129 448 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:47610 (request call id 127) took 4508 ms. Trace:
I20251025 07:52:51.100199 448 rpcz_store.cc:276] 1025 07:52:46.591800 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:50.874452 (+4282652us) service_pool.cc:225] Handling call
1025 07:52:51.098390 (+223938us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:51.098395 (+ 5us) write_op.cc:454] Released schema lock
1025 07:52:51.100086 (+ 1691us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:51.090919 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:51.091084 (+ 165us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:51.091093 (+ 9us) write_op.cc:435] Acquired schema lock
1025 07:52:51.091097 (+ 4us) tablet.cc:660] Decoding operations
1025 07:52:51.091250 (+ 153us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:51.091271 (+ 21us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:51.091277 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:51.091378 (+ 101us) tablet.cc:699] Row locks acquired
1025 07:52:51.091382 (+ 4us) write_op.cc:260] PREPARE: finished
1025 07:52:51.091420 (+ 38us) write_op.cc:270] Start()
1025 07:52:51.091456 (+ 36us) write_op.cc:276] Timestamp: P: 1761378771091403 usec, L: 0
1025 07:52:51.091460 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:51.091667 (+ 207us) log.cc:844] Serialized 622 byte log entry
1025 07:52:51.095251 (+ 3584us) op_driver.cc:464] REPLICATION: finished
1025 07:52:51.095402 (+ 151us) write_op.cc:301] APPLY: starting
1025 07:52:51.095417 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:51.095460 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:51.095496 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:51.095594 (+ 98us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:51.095649 (+ 55us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:51.095657 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:51.095728 (+ 71us) log.cc:844] Serialized 29 byte log entry
1025 07:52:51.095785 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:51.095805 (+ 20us) write_op.cc:454] Released schema lock
1025 07:52:51.095811 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":103,"apply.run_cpu_time_us":460,"apply.run_wall_time_us":459,"num_ops":2,"prepare.queue_time_us":58,"prepare.run_cpu_time_us":890,"prepare.run_wall_time_us":1637,"raft.queue_time_us":64,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":254,"replication_time_us":3748}]]}
I20251025 07:52:51.107092 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 3d5e0dd14e474fe09993a03851f8b1f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 33a8f7c5726844f3930f8fa5983d5f27) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:51.108296 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet a77eb82f26ae43b48de1e85693372d34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.108740 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 91728987303b41fd840eb56f3a311c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.109068 1015 tablet_service.cc:1552] Processing DeleteTablet for tablet cb829649f73d4d3f8862d268c953bd86 with delete_type TABLET_DATA_DELETED (replaced by tablet e330e1b53bed4e7cace1c4fb7fa6b565) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.109404 1014 tablet_service.cc:1552] Processing DeleteTablet for tablet 0f5f6ca6c3cd4d369690e425387a4344 with delete_type TABLET_DATA_DELETED (replaced by tablet 22fc90c704f64d2f9f4b8c54dee45576) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.109592 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet 4c155aa7fa5b4d68bbab1918b8432372 with delete_type TABLET_DATA_DELETED (replaced by tablet 110a8e9e4c6c434284c0ae14dd6987f7) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.109746 1012 tablet_service.cc:1552] Processing DeleteTablet for tablet d9bfd7c0c3ad4e5b98b6708793d61d41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.109916 1014 tablet_service.cc:1552] Processing DeleteTablet for tablet 04449cc97f404a19b6a1b20dea074f95 with delete_type TABLET_DATA_DELETED (replaced by tablet f70091024c034c118fb86578db19a31c) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.110093 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet b3a0af893a8947848e12093d0a7f0229 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.110256 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet ce1ebb1ec14b4801930a72765bca1a7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.110285 1012 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b940fa5ae26427dafbcfd26372754b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.110422 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet f43ef51ee45d47a99ec732b49d50d3b4 with delete_type TABLET_DATA_DELETED (replaced by tablet e429b94326cc4827a03968fd92de38e0) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.110512 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 0551e5d694894e8fbeb66b5e0c1cd75b with delete_type TABLET_DATA_DELETED (replaced by tablet ae835c2078624c42a57ca63b7951dc79) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.110570 1015 tablet_service.cc:1552] Processing DeleteTablet for tablet aaef24ccdfba45fe8204327723df834e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.110684 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 050a5d3b5f934dd19bf2e221f2fe0780 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.110761 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 36c8fcafd5464b4e999276a8062a812a with delete_type TABLET_DATA_DELETED (replaced by tablet 181059cc79d54999bf58504c85b739e9) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.110843 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet 110a8e9e4c6c434284c0ae14dd6987f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:45 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.110431 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet 11bfe0c87ca941a3846462c8a4512d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:44 UTC) from {username='slave'} at 127.0.0.1:44572
W20251025 07:52:50.993561 449 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:42387 (request call id 217) took 5167 ms (client timeout 1000 ms). Trace:
W20251025 07:52:51.111429 449 rpcz_store.cc:269] 1025 07:52:45.826448 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.876696 (+ 50248us) service_pool.cc:225] Handling call
1025 07:52:45.955455 (+ 78759us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x616000041254. stack: 00007fb90f7779f6 00007fb90f7777b0 00007fb90e7533ad 0000000000983f04 0000000000983ecd 00007fb91ea756bd 00007fb91ea757b3 00007fb91b640441 00007fb9232f69dd 00007fb92347daa8 00007fb919f96ca6 00007fb919f969d5 00007fb91671327e 00007fb91671215e 00007fb916718395 00007fb916718cbf
1025 07:52:47.348764 (+1393309us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:47.348771 (+ 7us) write_op.cc:454] Released schema lock
1025 07:52:50.993518 (+3644747us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:47.335154 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:47.335274 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:47.335280 (+ 6us) write_op.cc:435] Acquired schema lock
1025 07:52:47.335282 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:47.335428 (+ 146us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:47.335447 (+ 19us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:47.335452 (+ 5us) tablet.cc:683] Acquiring locks for 3 operations
1025 07:52:47.335545 (+ 93us) tablet.cc:699] Row locks acquired
1025 07:52:47.335547 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:47.335579 (+ 32us) write_op.cc:270] Start()
1025 07:52:47.335611 (+ 32us) write_op.cc:276] Timestamp: P: 1761378767335561 usec, L: 0
1025 07:52:47.335614 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:47.335783 (+ 169us) log.cc:844] Serialized 1275 byte log entry
1025 07:52:47.338781 (+ 2998us) op_driver.cc:464] REPLICATION: finished
1025 07:52:47.338884 (+ 103us) write_op.cc:301] APPLY: starting
1025 07:52:47.338898 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:47.338961 (+ 63us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:47.338984 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:47.339066 (+ 82us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:47.339099 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
1025 07:52:47.339106 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:47.339177 (+ 71us) log.cc:844] Serialized 35 byte log entry
1025 07:52:47.339242 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:47.339262 (+ 20us) write_op.cc:454] Released schema lock
1025 07:52:47.339269 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":37321,"reported_tablets":238,"spinlock_wait_cycles":30286208,"child_traces":[["op",{"apply.queue_time_us":62,"apply.run_cpu_time_us":429,"apply.run_wall_time_us":429,"num_ops":3,"prepare.queue_time_us":6158,"prepare.run_cpu_time_us":728,"prepare.run_wall_time_us":728,"raft.queue_time_us":312,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":247,"replication_time_us":3141}]]}
I20251025 07:52:51.112358 449 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_d0dc79aab1e641c48dafba37cdd6acd1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:51.112016 3236 raft_consensus.cc:493] T f7011f1dcad74fe4ba0356c618890e06 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 07:52:51.112964 449 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0dc79aab1e641c48dafba37cdd6acd1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:51.113075 3236 raft_consensus.cc:515] T f7011f1dcad74fe4ba0356c618890e06 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.113960 3236 leader_election.cc:290] T f7011f1dcad74fe4ba0356c618890e06 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:51.114763 450 master_service.cc:954] ReplaceTablet: received request to replace tablet 36c414bc700341188b7d85c14173c3e0 from {username='slave'} at 127.0.0.1:47610
I20251025 07:52:51.116323 1039 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7011f1dcad74fe4ba0356c618890e06" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:51.117491 712 leader_election.cc:343] T f7011f1dcad74fe4ba0356c618890e06 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:51.117754 3142 raft_consensus.cc:493] T 421db850a64048218eecb9de65e2bdbf P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:51.117897 3142 raft_consensus.cc:515] T 421db850a64048218eecb9de65e2bdbf P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.118670 3142 leader_election.cc:290] T 421db850a64048218eecb9de65e2bdbf P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:51.120504 850 leader_election.cc:343] T 9903e80a26fd469a8744dc855534ba75 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:51.120942 1039 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "421db850a64048218eecb9de65e2bdbf" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:51.121470 712 leader_election.cc:343] T 421db850a64048218eecb9de65e2bdbf P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:51.123432 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 792f32cbd66449ca98813c669b2a261d with delete_type TABLET_DATA_DELETED (replaced by tablet bc35a078052749cfa6a61b0ae3f5cac5) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:51.124991 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet f661d5909657455581dcb5e6eadd3c0b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.126559 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet 88eb51447e0f4080baa49dd902deb24f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.126909 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 60c60a9ee37f43978910bd067a098bb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:43 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.127116 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet f97acc9a638b4d5a968dcc3412a384bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.127467 1015 tablet_service.cc:1552] Processing DeleteTablet for tablet ebf3a2a926754305b5bd95fcebb25d29 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f05894c45154cd0b65f5b9bcef8cd4c) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.127724 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 26c849d376d143c6bef0c8e0e1d7d03a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.128063 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet 48ff4ce7846b4e56a37eee9bcb17bd5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.128201 1012 tablet_service.cc:1552] Processing DeleteTablet for tablet e47c4f741f354a19b49777bd06e54764 with delete_type TABLET_DATA_DELETED (replaced by tablet 755f0d6c87044e20a8020f8410940123) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.129645 3143 raft_consensus.cc:493] T 075f0a151e024220b51ba7fcaecdcc3d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:51.129829 3143 raft_consensus.cc:515] T 075f0a151e024220b51ba7fcaecdcc3d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.130717 3143 leader_election.cc:290] T 075f0a151e024220b51ba7fcaecdcc3d P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
W20251025 07:52:51.131106 451 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:38361 (request call id 215) took 4552 ms (client timeout 1000 ms). Trace:
I20251025 07:52:51.131405 764 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "075f0a151e024220b51ba7fcaecdcc3d" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:51.133293 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 308) took 5405 ms. Trace:
I20251025 07:52:51.133424 2146 rpcz_store.cc:276] 1025 07:52:45.727389 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.727661 (+ 272us) service_pool.cc:225] Handling call
1025 07:52:50.949084 (+5221423us) ts_tablet_manager.cc:1151] Deleting tablet ef1854d35a3047e3a9fbe0166ab1295a
1025 07:52:50.949094 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:50.993805 (+ 44711us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:50.996809 (+ 3004us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.133270 (+136461us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5221251}
I20251025 07:52:51.133925 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 309) took 5251 ms. Trace:
I20251025 07:52:51.134027 2146 rpcz_store.cc:276] 1025 07:52:45.882122 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.882555 (+ 433us) service_pool.cc:225] Handling call
1025 07:52:51.133788 (+5251233us) ts_tablet_manager.cc:1151] Deleting tablet f69cdaa7ee094283890036c0189e2c81
1025 07:52:51.133800 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.133909 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5251078}
I20251025 07:52:51.134368 2146 tablet_replica.cc:333] T df1248f6dbd94af68c587da7d149e279 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:51.134624 2146 raft_consensus.cc:2243] T df1248f6dbd94af68c587da7d149e279 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:51.132550 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f7c15529be54a11961f385a953b0b03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.134826 2146 raft_consensus.cc:2272] T df1248f6dbd94af68c587da7d149e279 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:51.135365 2146 ts_tablet_manager.cc:1916] T df1248f6dbd94af68c587da7d149e279 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:51.135895 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet e4da89f863cc43f3bf6df908bb9dc6bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.136068 1014 tablet_service.cc:1552] Processing DeleteTablet for tablet d77704b4fe6c4456a2ba40ffec0c60fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.138065 2169 tablet_replica.cc:333] T 4f7c15529be54a11961f385a953b0b03 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:51.138331 2169 raft_consensus.cc:2243] T 4f7c15529be54a11961f385a953b0b03 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:51.138515 2169 raft_consensus.cc:2272] T 4f7c15529be54a11961f385a953b0b03 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:51.139099 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet 95efd553493e4966bc48a01cac6e1e10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.139086 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "075f0a151e024220b51ba7fcaecdcc3d" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:51.139326 2169 ts_tablet_manager.cc:1916] T 4f7c15529be54a11961f385a953b0b03 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:51.139853 712 consensus_peers.cc:597] T aea8fcf0154a4bd4a05b19e27ab78b79 P 23eeffd3de254ce18db5febfde971c6b -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:51.140009 1166 tablet_bootstrap.cc:492] T dbb254115a574bf4976aff0ea56f3bd9 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:51.140125 1166 ts_tablet_manager.cc:1403] T dbb254115a574bf4976aff0ea56f3bd9 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.105s user 0.005s sys 0.001s
I20251025 07:52:51.140841 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet 792f32cbd66449ca98813c669b2a261d with delete_type TABLET_DATA_DELETED (replaced by tablet bc35a078052749cfa6a61b0ae3f5cac5) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.140848 1166 raft_consensus.cc:359] T dbb254115a574bf4976aff0ea56f3bd9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.141047 1166 raft_consensus.cc:385] T dbb254115a574bf4976aff0ea56f3bd9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:51.141115 1166 raft_consensus.cc:740] T dbb254115a574bf4976aff0ea56f3bd9 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:51.141336 1166 consensus_queue.cc:260] T dbb254115a574bf4976aff0ea56f3bd9 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.142432 1166 ts_tablet_manager.cc:1434] T dbb254115a574bf4976aff0ea56f3bd9 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:51.142472 983 leader_election.cc:304] T 075f0a151e024220b51ba7fcaecdcc3d P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:51.142745 1166 tablet_bootstrap.cc:492] T ffe7c8e1d6c44872b7b65e512764a56a P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:51.143276 3143 raft_consensus.cc:2749] T 075f0a151e024220b51ba7fcaecdcc3d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:51.144245 2166 ts_tablet_manager.cc:1929] T ebf3a2a926754305b5bd95fcebb25d29 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:51.144356 2166 log.cc:1199] T ebf3a2a926754305b5bd95fcebb25d29 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/ebf3a2a926754305b5bd95fcebb25d29
I20251025 07:52:51.144606 1166 tablet_bootstrap.cc:654] T ffe7c8e1d6c44872b7b65e512764a56a P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:51.144685 2166 ts_tablet_manager.cc:1950] T ebf3a2a926754305b5bd95fcebb25d29 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:51.144837 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b278ce7768349e49dcd1131e404ab34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:44 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.145445 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 271) took 5421 ms. Trace:
I20251025 07:52:51.145537 2166 rpcz_store.cc:276] 1025 07:52:45.724268 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.724327 (+ 59us) service_pool.cc:225] Handling call
1025 07:52:51.088350 (+5364023us) ts_tablet_manager.cc:1151] Deleting tablet ebf3a2a926754305b5bd95fcebb25d29
1025 07:52:51.088360 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.132776 (+ 44416us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.144146 (+ 11370us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.145425 (+ 1279us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5363819}
I20251025 07:52:51.145864 2166 tablet_replica.cc:333] T ef1854d35a3047e3a9fbe0166ab1295a P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:51.145857 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet c11f674385184ae6b235f1db8bf1fb91 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.145932 1169 tablet_bootstrap.cc:492] T d46673a283fe42b78ddeb99fbd2678eb P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:51.146045 1169 ts_tablet_manager.cc:1403] T d46673a283fe42b78ddeb99fbd2678eb P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.061s user 0.000s sys 0.003s
I20251025 07:52:51.146067 2166 raft_consensus.cc:2243] T ef1854d35a3047e3a9fbe0166ab1295a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:51.146250 2166 raft_consensus.cc:2272] T ef1854d35a3047e3a9fbe0166ab1295a P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:51.146354 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e098c9c4613459295b6043048cba587 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.146790 2166 ts_tablet_manager.cc:1916] T ef1854d35a3047e3a9fbe0166ab1295a P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:51.146771 1169 raft_consensus.cc:359] T d46673a283fe42b78ddeb99fbd2678eb P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.146857 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet cc407c7eb12e4eadb41fb21385611c94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.146919 1169 raft_consensus.cc:385] T d46673a283fe42b78ddeb99fbd2678eb P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:51.146963 1169 raft_consensus.cc:740] T d46673a283fe42b78ddeb99fbd2678eb P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
W20251025 07:52:51.131206 451 rpcz_store.cc:269] 1025 07:52:46.579102 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:46.599140 (+ 20038us) service_pool.cc:225] Handling call
1025 07:52:51.131042 (+4531902us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14120,"reported_tablets":243}
I20251025 07:52:51.147138 1169 consensus_queue.cc:260] T d46673a283fe42b78ddeb99fbd2678eb P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.147287 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet 6897405bbc1c47a4ab0aa3af80e60e3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.147720 1169 ts_tablet_manager.cc:1434] T d46673a283fe42b78ddeb99fbd2678eb P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:51.148195 450 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet 36c414bc700341188b7d85c14173c3e0
I20251025 07:52:51.148833 450 catalog_manager.cc:6719] ReplaceTablet: tablet 36c414bc700341188b7d85c14173c3e0 deleted and replaced by tablet eced77e6a53447db9fb2df3b670684f2
I20251025 07:52:51.148976 450 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:47610 (request call id 133) took 4291 ms. Trace:
I20251025 07:52:51.149006 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c4970c8a491460e8b4b1c29fde83e4c with delete_type TABLET_DATA_DELETED (replaced by tablet 783487b60c144f719ed2eb3c67b5ccab) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.149441 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet 1912e59548874e0e8e989050ec0064c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.149087 450 rpcz_store.cc:276] 1025 07:52:46.857644 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:51.114666 (+4257022us) service_pool.cc:225] Handling call
1025 07:52:51.147874 (+ 33208us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:51.147883 (+ 9us) write_op.cc:454] Released schema lock
1025 07:52:51.148947 (+ 1064us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:51.115860 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:51.115971 (+ 111us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:51.115978 (+ 7us) write_op.cc:435] Acquired schema lock
1025 07:52:51.115980 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:51.116088 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:51.116102 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:51.116107 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:51.116177 (+ 70us) tablet.cc:699] Row locks acquired
1025 07:52:51.116180 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:51.116214 (+ 34us) write_op.cc:270] Start()
1025 07:52:51.116268 (+ 54us) write_op.cc:276] Timestamp: P: 1761378771116199 usec, L: 0
1025 07:52:51.116271 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:51.116441 (+ 170us) log.cc:844] Serialized 656 byte log entry
1025 07:52:51.120196 (+ 3755us) op_driver.cc:464] REPLICATION: finished
1025 07:52:51.120354 (+ 158us) write_op.cc:301] APPLY: starting
1025 07:52:51.120369 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:51.120413 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:51.120438 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:51.120538 (+ 100us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:51.120608 (+ 70us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:51.120615 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:51.120691 (+ 76us) log.cc:844] Serialized 29 byte log entry
1025 07:52:51.120751 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:51.120772 (+ 21us) write_op.cc:454] Released schema lock
1025 07:52:51.120778 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":109,"apply.run_cpu_time_us":475,"apply.run_wall_time_us":474,"num_ops":2,"prepare.queue_time_us":50,"prepare.run_cpu_time_us":707,"prepare.run_wall_time_us":1314,"raft.queue_time_us":82,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":229,"replication_time_us":3895}]]}
I20251025 07:52:51.149691 450 master_service.cc:954] ReplaceTablet: received request to replace tablet 645263a61b654f04985c40d04506c087 from {username='slave'} at 127.0.0.1:47610
I20251025 07:52:51.150067 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet 0540d6a10cf04fa89756623e9aaaeb68 with delete_type TABLET_DATA_DELETED (replaced by tablet 55a4a524676b4b9aa9464e934b662f30) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.150278 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet ebf3a2a926754305b5bd95fcebb25d29 (table default.Table_ac4beca0c3bd4cf29565faf5323d9ff7 [id=857bcd1dbe2e477bbc0d77f79a58f7f7]) successfully deleted
I20251025 07:52:51.150460 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet f69cdaa7ee094283890036c0189e2c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:45 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.150861 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet 32a143a715cb485cb4501f4e28c68252 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c389d798cb94d8a972683c57b1f9f1f) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.150981 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 36c414bc700341188b7d85c14173c3e0 with delete_type TABLET_DATA_DELETED (replaced by tablet eced77e6a53447db9fb2df3b670684f2) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:51.151238 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet 6dcab53a020847a1aa27930265ba0205 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.154006 2169 ts_tablet_manager.cc:1929] T 4f7c15529be54a11961f385a953b0b03 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:51.155308 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet ef1854d35a3047e3a9fbe0166ab1295a (table default.Table_088aad2a46b041bfa29ed8e6929760b1 [id=834669c54dae449790818684fd8659bc]) successfully deleted
I20251025 07:52:51.155670 2146 ts_tablet_manager.cc:1929] T df1248f6dbd94af68c587da7d149e279 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:51.155709 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 36c414bc700341188b7d85c14173c3e0 with delete_type TABLET_DATA_DELETED (replaced by tablet eced77e6a53447db9fb2df3b670684f2) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:51.155819 2146 log.cc:1199] T df1248f6dbd94af68c587da7d149e279 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/df1248f6dbd94af68c587da7d149e279
W20251025 07:52:51.156131 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet f69cdaa7ee094283890036c0189e2c81 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f69cdaa7ee094283890036c0189e2c81 already in progress: creating tablet
I20251025 07:52:51.156153 2146 ts_tablet_manager.cc:1950] T df1248f6dbd94af68c587da7d149e279 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:51.154109 2169 log.cc:1199] T 4f7c15529be54a11961f385a953b0b03 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/4f7c15529be54a11961f385a953b0b03
I20251025 07:52:51.156864 2169 ts_tablet_manager.cc:1950] T 4f7c15529be54a11961f385a953b0b03 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:51.157444 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 310) took 5234 ms. Trace:
I20251025 07:52:51.158756 2166 ts_tablet_manager.cc:1929] T ef1854d35a3047e3a9fbe0166ab1295a P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251025 07:52:51.158892 2166 log.cc:1199] T ef1854d35a3047e3a9fbe0166ab1295a P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/ef1854d35a3047e3a9fbe0166ab1295a
I20251025 07:52:51.159000 2146 rpcz_store.cc:276] 1025 07:52:45.922514 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.923476 (+ 962us) service_pool.cc:225] Handling call
1025 07:52:51.134322 (+5210846us) ts_tablet_manager.cc:1151] Deleting tablet df1248f6dbd94af68c587da7d149e279
1025 07:52:51.134333 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.144136 (+ 9803us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.155577 (+ 11441us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.157424 (+ 1847us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5210605}
I20251025 07:52:51.159335 2166 ts_tablet_manager.cc:1950] T ef1854d35a3047e3a9fbe0166ab1295a P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:51.160410 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 272) took 5432 ms. Trace:
I20251025 07:52:51.160521 2166 rpcz_store.cc:276] 1025 07:52:45.728088 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.728141 (+ 53us) service_pool.cc:225] Handling call
1025 07:52:51.145814 (+5417673us) ts_tablet_manager.cc:1151] Deleting tablet ef1854d35a3047e3a9fbe0166ab1295a
1025 07:52:51.145825 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.154682 (+ 8857us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.158651 (+ 3969us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.160382 (+ 1731us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5417498}
I20251025 07:52:51.161047 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet ef1854d35a3047e3a9fbe0166ab1295a (table default.Table_088aad2a46b041bfa29ed8e6929760b1 [id=834669c54dae449790818684fd8659bc]) successfully deleted
I20251025 07:52:51.161435 2146 tablet_replica.cc:333] T ab42909afc524f2fa3c1041495d13a59 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:51.161561 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 273) took 5285 ms. Trace:
I20251025 07:52:51.161664 2166 rpcz_store.cc:276] 1025 07:52:45.876252 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:45.876312 (+ 60us) service_pool.cc:225] Handling call
1025 07:52:51.161407 (+5285095us) ts_tablet_manager.cc:1151] Deleting tablet f69cdaa7ee094283890036c0189e2c81
1025 07:52:51.161420 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.161540 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5284879}
W20251025 07:52:51.162212 439 catalog_manager.cc:4965] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet f69cdaa7ee094283890036c0189e2c81 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f69cdaa7ee094283890036c0189e2c81 already in progress: creating tablet
I20251025 07:52:51.162420 2146 raft_consensus.cc:2243] T ab42909afc524f2fa3c1041495d13a59 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:51.162762 2146 raft_consensus.cc:2272] T ab42909afc524f2fa3c1041495d13a59 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:51.164572 2166 tablet_replica.cc:333] T df1248f6dbd94af68c587da7d149e279 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:51.164830 2166 raft_consensus.cc:2243] T df1248f6dbd94af68c587da7d149e279 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:51.165035 2166 raft_consensus.cc:2272] T df1248f6dbd94af68c587da7d149e279 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:51.166812 2166 ts_tablet_manager.cc:1916] T df1248f6dbd94af68c587da7d149e279 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:51.170049 755 tablet_service.cc:1505] Processing CreateTablet for tablet 52ec92092aaa431f8beba2082336d64c (DEFAULT_TABLE table=default.Table_68774a9f47334984a5e9cfae8b304519 [id=c387599e70124eb1891950bc8371e9a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:51.170456 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52ec92092aaa431f8beba2082336d64c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:51.172785 2146 ts_tablet_manager.cc:1916] T ab42909afc524f2fa3c1041495d13a59 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:51.176919 2166 ts_tablet_manager.cc:1929] T df1248f6dbd94af68c587da7d149e279 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:51.177024 2166 log.cc:1199] T df1248f6dbd94af68c587da7d149e279 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/df1248f6dbd94af68c587da7d149e279
I20251025 07:52:51.181315 1166 tablet_bootstrap.cc:492] T ffe7c8e1d6c44872b7b65e512764a56a P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:51.181510 1166 ts_tablet_manager.cc:1403] T ffe7c8e1d6c44872b7b65e512764a56a P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
I20251025 07:52:51.181813 2166 ts_tablet_manager.cc:1950] T df1248f6dbd94af68c587da7d149e279 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:51.182356 1166 raft_consensus.cc:359] T ffe7c8e1d6c44872b7b65e512764a56a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.182582 1166 raft_consensus.cc:385] T ffe7c8e1d6c44872b7b65e512764a56a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:51.182669 1166 raft_consensus.cc:740] T ffe7c8e1d6c44872b7b65e512764a56a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:51.182884 1166 consensus_queue.cc:260] T ffe7c8e1d6c44872b7b65e512764a56a P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.183137 883 tablet_service.cc:1505] Processing CreateTablet for tablet 52ec92092aaa431f8beba2082336d64c (DEFAULT_TABLE table=default.Table_68774a9f47334984a5e9cfae8b304519 [id=c387599e70124eb1891950bc8371e9a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:51.183488 1166 ts_tablet_manager.cc:1434] T ffe7c8e1d6c44872b7b65e512764a56a P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:51.183565 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52ec92092aaa431f8beba2082336d64c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:51.183724 1166 tablet_bootstrap.cc:492] T 0b47d1b0e6a342888135d7e4e5fa1f43 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:51.185180 1166 tablet_bootstrap.cc:654] T 0b47d1b0e6a342888135d7e4e5fa1f43 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:51.185405 2686 raft_consensus.cc:2804] T b71bdfd09a27449d98509bb220680147 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
W20251025 07:52:51.185484 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:46231, user_credentials={real_user=slave}} blocked reactor thread for 103366us
I20251025 07:52:51.185508 2686 raft_consensus.cc:697] T b71bdfd09a27449d98509bb220680147 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:51.185715 847 leader_election.cc:304] T 294d7d20c2f545b8813e632aa058a60c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:51.185791 2686 consensus_queue.cc:237] T b71bdfd09a27449d98509bb220680147 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.186060 847 leader_election.cc:304] T 66134661be834f7b811d7927ab629812 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:51.186340 3203 raft_consensus.cc:2804] T 294d7d20c2f545b8813e632aa058a60c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:51.186405 3203 raft_consensus.cc:493] T 294d7d20c2f545b8813e632aa058a60c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:51.186460 3203 raft_consensus.cc:3060] T 294d7d20c2f545b8813e632aa058a60c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:51.186658 2988 raft_consensus.cc:2804] T 66134661be834f7b811d7927ab629812 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:51.186822 2988 raft_consensus.cc:697] T 66134661be834f7b811d7927ab629812 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:51.187011 2988 consensus_queue.cc:237] T 66134661be834f7b811d7927ab629812 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.188982 3203 raft_consensus.cc:515] T 294d7d20c2f545b8813e632aa058a60c P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.189483 3203 leader_election.cc:290] T 294d7d20c2f545b8813e632aa058a60c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:51.189944 847 leader_election.cc:304] T 8707b93d491e4e4386d148087a2f0870 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:51.190295 3203 raft_consensus.cc:2804] T 8707b93d491e4e4386d148087a2f0870 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:51.190641 3203 raft_consensus.cc:697] T 8707b93d491e4e4386d148087a2f0870 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:51.190896 3203 consensus_queue.cc:237] T 8707b93d491e4e4386d148087a2f0870 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.195816 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a1ec8c2ed7f41abb588f592f62097d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:45 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.196257 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet df1248f6dbd94af68c587da7d149e279 (table default.Table_73279bdee234454e9def34441da9579a [id=6430d82b82c042469a77ea9f4dad15fc]) successfully deleted
I20251025 07:52:51.196759 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet b02f5c0ab0b24288bb50a3ba0e6bd78a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.197650 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet 29dbaf9149134f548a974a5779f7da67 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:44 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.198078 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet ee1c8ea694284d80a30f9c05be4cbf49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.198496 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet b947ecc352e54f4caa73c6ca97372256 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.203480 3235 consensus_queue.cc:1048] T 8917a289bd9b4428a4603429edd94581 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:51.206449 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9903e80a26fd469a8744dc855534ba75" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:51.206632 773 raft_consensus.cc:2468] T 9903e80a26fd469a8744dc855534ba75 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:51.213617 3236 raft_consensus.cc:493] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:51.213768 3236 raft_consensus.cc:515] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.214476 3236 leader_election.cc:290] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:51.215904 1039 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5fc092c2bb2d41ac8c5b4fc07460c8ec" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:51.216368 712 leader_election.cc:343] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:51.220011 3236 raft_consensus.cc:493] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:51.220165 3236 raft_consensus.cc:515] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:51.217711 1169 tablet_bootstrap.cc:492] T 72618756f756484e895e885ef12545c8 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:51.221031 3236 leader_election.cc:290] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:51.221637 1039 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "141e413d2faa4c67b4c822192e631417" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:51.221830 1039 raft_consensus.cc:2468] T 141e413d2faa4c67b4c822192e631417 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:51.221817 771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "294d7d20c2f545b8813e632aa058a60c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:51.221952 771 raft_consensus.cc:3060] T 294d7d20c2f545b8813e632aa058a60c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:51.222261 712 leader_election.cc:304] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:51.222623 3236 raft_consensus.cc:2804] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251025 07:52:51.222738 3236 raft_consensus.cc:493] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:51.222838 3236 raft_consensus.cc:3060] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Advancing to term 2
I20251025 07:52:51.224763 771 raft_consensus.cc:2468] T 294d7d20c2f545b8813e632aa058a60c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:51.225395 3236 raft_consensus.cc:515] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:51.226418 1039 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "141e413d2faa4c67b4c822192e631417" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:51.226630 1039 raft_consensus.cc:3060] T 141e413d2faa4c67b4c822192e631417 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Advancing to term 2
I20251025 07:52:51.227300 1169 tablet_bootstrap.cc:654] T 72618756f756484e895e885ef12545c8 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:51.229195 2686 raft_consensus.cc:493] T 141e413d2faa4c67b4c822192e631417 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:51.229353 2686 raft_consensus.cc:515] T 141e413d2faa4c67b4c822192e631417 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:51.229544 1039 raft_consensus.cc:2468] T 141e413d2faa4c67b4c822192e631417 P db42be7ad3c741a0aa427f27e245993b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 2.
I20251025 07:52:51.229967 712 leader_election.cc:304] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, db42be7ad3c741a0aa427f27e245993b; no voters:
I20251025 07:52:51.230067 2686 leader_election.cc:290] T 141e413d2faa4c67b4c822192e631417 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:51.230196 3235 raft_consensus.cc:2804] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b [term 2 FOLLOWER]: Leader election won for term 2
I20251025 07:52:51.230593 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet 1b5ac7e845a24b2eb5b2429d47361232 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.230998 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet 9f039d9107ae493486d799ab6a1d731d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:44 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.231336 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet 50ae9bca6bed4f749d54191ac1543410 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.231333 3235 raft_consensus.cc:697] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b [term 2 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:51.231614 3235 consensus_queue.cc:237] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } }
I20251025 07:52:51.232317 3236 leader_election.cc:290] T 141e413d2faa4c67b4c822192e631417 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 2 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:51.232892 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7011f1dcad74fe4ba0356c618890e06" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:51.233100 910 raft_consensus.cc:2468] T f7011f1dcad74fe4ba0356c618890e06 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:51.232929 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "421db850a64048218eecb9de65e2bdbf" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:51.233880 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet 72dc084f8efc45478771c247e07a16f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.234462 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet fd265cdbf6a8440393a2b5d8240d1872 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.235607 2146 ts_tablet_manager.cc:1929] T ab42909afc524f2fa3c1041495d13a59 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:51.235774 2146 log.cc:1199] T ab42909afc524f2fa3c1041495d13a59 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/ab42909afc524f2fa3c1041495d13a59
I20251025 07:52:51.236214 2146 ts_tablet_manager.cc:1950] T ab42909afc524f2fa3c1041495d13a59 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:51.236438 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b594d4718344aafbdaec34040695589 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.236495 2686 raft_consensus.cc:493] T 3c4970c8a491460e8b4b1c29fde83e4c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:51.236765 2686 raft_consensus.cc:515] T 3c4970c8a491460e8b4b1c29fde83e4c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.236941 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet 570201e7bf05406a97c81dde071d894b with delete_type TABLET_DATA_DELETED (replaced by tablet 3a0f50c8000144a4acb5219484b01800) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.237040 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 337) took 3365 ms. Trace:
I20251025 07:52:51.237160 2146 rpcz_store.cc:276] 1025 07:52:47.871843 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:47.871910 (+ 67us) service_pool.cc:225] Handling call
1025 07:52:51.161387 (+3289477us) ts_tablet_manager.cc:1151] Deleting tablet ab42909afc524f2fa3c1041495d13a59
1025 07:52:51.161397 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.175866 (+ 14469us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.235493 (+ 59627us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.237023 (+ 1530us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3289256}
I20251025 07:52:51.237416 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet 623dec4eab1d4666baf2334a01eeb36d with delete_type TABLET_DATA_DELETED (replaced by tablet b2ac640e69794a09811bd93aa02a22ea) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.237720 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet ab42909afc524f2fa3c1041495d13a59 (table default.Table_7eb8424d5192485da1a2d67bfdb2bbbf [id=40c29df99590408cbd5c583d4fc741eb]) successfully deleted
I20251025 07:52:51.237913 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet fa998b9ca6724c4cb1eb68a3eecb0b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.237989 2686 leader_election.cc:290] T 3c4970c8a491460e8b4b1c29fde83e4c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:51.239645 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet ff3daed6c9af47e3976b82b4b9a41086 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.240363 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet a7ab3150a35b4caca9119a1374d3f2f7 with delete_type TABLET_DATA_DELETED (replaced by tablet c19c97f881cb45218d4a6f6f231aa96e) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.241041 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet ab42909afc524f2fa3c1041495d13a59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:45 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.241847 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet ef1854d35a3047e3a9fbe0166ab1295a with delete_type TABLET_DATA_DELETED (replaced by tablet 3760348dbe284c41a01a4fe479a2103f) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.242825 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ec421eb820b4f40afcff5a5bf80c508 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.243898 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet 3abfd52327a04ba88625f391a8658fbb with delete_type TABLET_DATA_DELETED (replaced by tablet 04d27d54fdb7483b869408af65ff85c9) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.245033 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b28215f61574cf48c447dd01955ac77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.246011 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet d66ad049d5484228a268b0facb00ecf1 with delete_type TABLET_DATA_DELETED (replaced by tablet 2e098c9c4613459295b6043048cba587) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.247978 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet df1248f6dbd94af68c587da7d149e279 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:45 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.250429 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet af743ee344e543a0aa6da3cd7a30e467 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.251341 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet 34315f64b89148efbf1f20a43d5c5efb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.252305 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet a4e4b99be0434baaa75d988281797846 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.253393 3236 raft_consensus.cc:493] T 3c4970c8a491460e8b4b1c29fde83e4c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:51.253636 3236 raft_consensus.cc:515] T 3c4970c8a491460e8b4b1c29fde83e4c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.254827 1039 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c4970c8a491460e8b4b1c29fde83e4c" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:51.255367 712 leader_election.cc:343] T 3c4970c8a491460e8b4b1c29fde83e4c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:51.255854 3236 leader_election.cc:290] T 3c4970c8a491460e8b4b1c29fde83e4c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:51.256377 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet 92aa605fabbc4257a6a204c087e7cc7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.256476 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5fc092c2bb2d41ac8c5b4fc07460c8ec" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:51.256780 909 raft_consensus.cc:2468] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:51.257120 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet b94634ac5077439fadfbbce569f60e8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.256839 1166 tablet_bootstrap.cc:492] T 0b47d1b0e6a342888135d7e4e5fa1f43 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:51.257616 1166 ts_tablet_manager.cc:1403] T 0b47d1b0e6a342888135d7e4e5fa1f43 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.074s user 0.005s sys 0.007s
I20251025 07:52:51.257650 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "141e413d2faa4c67b4c822192e631417" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:51.257819 909 raft_consensus.cc:2468] T 141e413d2faa4c67b4c822192e631417 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:51.257984 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet e54fe7cbcffa40c4b04144b47bee2b48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.258354 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c4970c8a491460e8b4b1c29fde83e4c" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:51.258423 1166 raft_consensus.cc:359] T 0b47d1b0e6a342888135d7e4e5fa1f43 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.258498 902 raft_consensus.cc:2468] T 3c4970c8a491460e8b4b1c29fde83e4c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:51.258575 1166 raft_consensus.cc:385] T 0b47d1b0e6a342888135d7e4e5fa1f43 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:51.258746 1166 raft_consensus.cc:740] T 0b47d1b0e6a342888135d7e4e5fa1f43 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:51.258945 1166 consensus_queue.cc:260] T 0b47d1b0e6a342888135d7e4e5fa1f43 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.259171 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "141e413d2faa4c67b4c822192e631417" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:51.259334 909 raft_consensus.cc:3060] T 141e413d2faa4c67b4c822192e631417 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Advancing to term 2
I20251025 07:52:51.259433 1166 ts_tablet_manager.cc:1434] T 0b47d1b0e6a342888135d7e4e5fa1f43 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:51.259711 1166 tablet_bootstrap.cc:492] T 19f1b56cdbd24ebe8688cc05fb795d07 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:51.260783 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet f656945454fe4921ab819c772a089eb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.261384 1166 tablet_bootstrap.cc:654] T 19f1b56cdbd24ebe8688cc05fb795d07 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:51.262261 2146 tablet_replica.cc:333] T 4a1ec8c2ed7f41abb588f592f62097d1 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:51.262456 2146 raft_consensus.cc:2243] T 4a1ec8c2ed7f41abb588f592f62097d1 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:51.262765 2146 raft_consensus.cc:2272] T 4a1ec8c2ed7f41abb588f592f62097d1 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:51.263291 2146 ts_tablet_manager.cc:1916] T 4a1ec8c2ed7f41abb588f592f62097d1 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:51.266314 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 274) took 5154 ms. Trace:
I20251025 07:52:51.266405 2166 rpcz_store.cc:276] 1025 07:52:46.112268 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:46.112334 (+ 66us) service_pool.cc:225] Handling call
1025 07:52:51.164508 (+5052174us) ts_tablet_manager.cc:1151] Deleting tablet df1248f6dbd94af68c587da7d149e279
1025 07:52:51.164521 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.173862 (+ 9341us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.176844 (+ 2982us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.266291 (+ 89447us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5051983}
I20251025 07:52:51.266765 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 275) took 5023 ms. Trace:
I20251025 07:52:51.266830 2166 rpcz_store.cc:276] 1025 07:52:46.243231 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:46.243293 (+ 62us) service_pool.cc:225] Handling call
1025 07:52:51.266659 (+5023366us) ts_tablet_manager.cc:1151] Deleting tablet ac0ea298bc114c56a8093e12def21ce1
1025 07:52:51.266668 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.266751 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5022508}
I20251025 07:52:51.267066 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 276) took 4927 ms. Trace:
I20251025 07:52:51.267120 2166 rpcz_store.cc:276] 1025 07:52:46.339260 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:46.339317 (+ 57us) service_pool.cc:225] Handling call
1025 07:52:51.267006 (+4927689us) ts_tablet_manager.cc:1151] Deleting tablet 87a93778620f4c84b3b3f7a3d7d7e87f
1025 07:52:51.267013 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.267055 (+ 42us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4927491}
I20251025 07:52:51.267362 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 279) took 4923 ms. Trace:
I20251025 07:52:51.267424 2166 rpcz_store.cc:276] 1025 07:52:46.343724 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:46.343931 (+ 207us) service_pool.cc:225] Handling call
1025 07:52:51.267290 (+4923359us) ts_tablet_manager.cc:1151] Deleting tablet 2aa522477e934c89834026dc480fe7c0
1025 07:52:51.267298 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.267346 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4923194}
I20251025 07:52:51.267707 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 280) took 4921 ms. Trace:
I20251025 07:52:51.267761 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet df1248f6dbd94af68c587da7d149e279 (table default.Table_73279bdee234454e9def34441da9579a [id=6430d82b82c042469a77ea9f4dad15fc]) successfully deleted
I20251025 07:52:51.267781 2166 rpcz_store.cc:276] 1025 07:52:46.346114 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:46.346191 (+ 77us) service_pool.cc:225] Handling call
1025 07:52:51.267637 (+4921446us) ts_tablet_manager.cc:1151] Deleting tablet 35ba9d23262248cda6fdf03c51be11a4
1025 07:52:51.267647 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.267695 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4921287}
I20251025 07:52:51.268020 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 281) took 4921 ms. Trace:
W20251025 07:52:51.268056 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet ac0ea298bc114c56a8093e12def21ce1 because the tablet was not found. No further retry: Not found: Tablet not found: ac0ea298bc114c56a8093e12def21ce1
I20251025 07:52:51.268091 2166 rpcz_store.cc:276] 1025 07:52:46.346253 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:46.346299 (+ 46us) service_pool.cc:225] Handling call
1025 07:52:51.267953 (+4921654us) ts_tablet_manager.cc:1151] Deleting tablet 74e2d78c024849b0af5c0bde1f48c36f
1025 07:52:51.267961 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.268006 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4921524}
W20251025 07:52:51.268280 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 87a93778620f4c84b3b3f7a3d7d7e87f because the tablet was not found. No further retry: Not found: Tablet not found: 87a93778620f4c84b3b3f7a3d7d7e87f
I20251025 07:52:51.268350 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 282) took 4922 ms. Trace:
I20251025 07:52:51.268412 2166 rpcz_store.cc:276] 1025 07:52:46.346346 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:46.346473 (+ 127us) service_pool.cc:225] Handling call
1025 07:52:51.268265 (+4921792us) ts_tablet_manager.cc:1151] Deleting tablet 5fca357d7e24494fb6af4bd73d98833c
1025 07:52:51.268275 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.268336 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4921639}
I20251025 07:52:51.268663 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 285) took 4922 ms. Trace:
I20251025 07:52:51.268728 2166 rpcz_store.cc:276] 1025 07:52:46.346632 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:46.346712 (+ 80us) service_pool.cc:225] Handling call
1025 07:52:51.268592 (+4921880us) ts_tablet_manager.cc:1151] Deleting tablet 02e4c4d6ffb34cecaf17beb5ba197e81
1025 07:52:51.268602 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.268651 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4921733}
I20251025 07:52:51.268970 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 284) took 4922 ms. Trace:
I20251025 07:52:51.269044 2166 rpcz_store.cc:276] 1025 07:52:46.346504 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:46.346702 (+ 198us) service_pool.cc:225] Handling call
1025 07:52:51.268903 (+4922201us) ts_tablet_manager.cc:1151] Deleting tablet 2b594d4718344aafbdaec34040695589
1025 07:52:51.268910 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.268959 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35,"tablet-delete.queue_time_us":4922042}
W20251025 07:52:51.269290 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 2aa522477e934c89834026dc480fe7c0 because the tablet was not found. No further retry: Not found: Tablet not found: 2aa522477e934c89834026dc480fe7c0
I20251025 07:52:51.269357 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 283) took 4922 ms. Trace:
I20251025 07:52:51.269428 2166 rpcz_store.cc:276] 1025 07:52:46.346430 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:46.346889 (+ 459us) service_pool.cc:225] Handling call
1025 07:52:51.269270 (+4922381us) ts_tablet_manager.cc:1151] Deleting tablet c4610b0c62e741eca28c21f63cc3de81
1025 07:52:51.269297 (+ 27us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.269343 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4922255}
W20251025 07:52:51.269497 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 35ba9d23262248cda6fdf03c51be11a4 because the tablet was not found. No further retry: Not found: Tablet not found: 35ba9d23262248cda6fdf03c51be11a4
I20251025 07:52:51.269656 2166 tablet_replica.cc:333] T ab42909afc524f2fa3c1041495d13a59 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
W20251025 07:52:51.269722 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 74e2d78c024849b0af5c0bde1f48c36f because the tablet was not found. No further retry: Not found: Tablet not found: 74e2d78c024849b0af5c0bde1f48c36f
I20251025 07:52:51.269829 2166 raft_consensus.cc:2243] T ab42909afc524f2fa3c1041495d13a59 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
W20251025 07:52:51.269894 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 5fca357d7e24494fb6af4bd73d98833c because the tablet was not found. No further retry: Not found: Tablet not found: 5fca357d7e24494fb6af4bd73d98833c
I20251025 07:52:51.269985 2166 raft_consensus.cc:2272] T ab42909afc524f2fa3c1041495d13a59 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:51.270112 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 02e4c4d6ffb34cecaf17beb5ba197e81 because the tablet was not found. No further retry: Not found: Tablet not found: 02e4c4d6ffb34cecaf17beb5ba197e81
I20251025 07:52:51.270298 450 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b2dfbd0b44a441c9be9254f660974944: Sending DeleteTablet for 3 replicas of tablet 645263a61b654f04985c40d04506c087
W20251025 07:52:51.270668 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 2b594d4718344aafbdaec34040695589 because the tablet was not found. No further retry: Not found: Tablet not found: 2b594d4718344aafbdaec34040695589
I20251025 07:52:51.270813 450 catalog_manager.cc:6719] ReplaceTablet: tablet 645263a61b654f04985c40d04506c087 deleted and replaced by tablet 3ccc61d97e494ce0ab40b118d6f4fd40
W20251025 07:52:51.270936 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet c4610b0c62e741eca28c21f63cc3de81 because the tablet was not found. No further retry: Not found: Tablet not found: c4610b0c62e741eca28c21f63cc3de81
I20251025 07:52:51.270964 450 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:47610 (request call id 134) took 4382 ms. Trace:
I20251025 07:52:51.271330 710 leader_election.cc:304] T f7011f1dcad74fe4ba0356c618890e06 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:51.271050 450 rpcz_store.cc:276] 1025 07:52:46.888526 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:51.149609 (+4261083us) service_pool.cc:225] Handling call
1025 07:52:51.270900 (+121291us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:51.152043 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:51.152156 (+ 113us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:51.152162 (+ 6us) write_op.cc:435] Acquired schema lock
1025 07:52:51.152164 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:51.152284 (+ 120us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:51.152296 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:51.152301 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:51.152372 (+ 71us) tablet.cc:699] Row locks acquired
1025 07:52:51.152375 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:51.152409 (+ 34us) write_op.cc:270] Start()
1025 07:52:51.152438 (+ 29us) write_op.cc:276] Timestamp: P: 1761378771152399 usec, L: 0
1025 07:52:51.152441 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:51.152622 (+ 181us) log.cc:844] Serialized 656 byte log entry
1025 07:52:51.199208 (+ 46586us) op_driver.cc:464] REPLICATION: finished
1025 07:52:51.199528 (+ 320us) write_op.cc:301] APPLY: starting
1025 07:52:51.199561 (+ 33us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:51.199626 (+ 65us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:51.199717 (+ 91us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:51.199867 (+ 150us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:51.199927 (+ 60us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1025 07:52:51.199934 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:51.200026 (+ 92us) log.cc:844] Serialized 29 byte log entry
1025 07:52:51.200108 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:51.200129 (+ 21us) write_op.cc:454] Released schema lock
1025 07:52:51.200136 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:51.200237 (+ 101us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:51.200243 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":256,"apply.run_cpu_time_us":695,"apply.run_wall_time_us":695,"num_ops":2,"prepare.queue_time_us":50,"prepare.run_cpu_time_us":705,"prepare.run_wall_time_us":1776,"raft.queue_time_us":310,"raft.run_cpu_time_us":8,"raft.run_wall_time_us":8,"replication_time_us":46730}]]}
I20251025 07:52:51.271689 450 master_service.cc:954] ReplaceTablet: received request to replace tablet 9979cae195e849908677d78c6c33be71 from {username='slave'} at 127.0.0.1:47610
I20251025 07:52:51.272043 450 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:47610 (request call id 135) took 4332 ms. Trace:
I20251025 07:52:51.272152 450 rpcz_store.cc:276] 1025 07:52:46.939866 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:51.271613 (+4331747us) service_pool.cc:225] Handling call
1025 07:52:51.272003 (+ 390us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251025 07:52:51.272512 450 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_8b4acedfb7244aa5ae4a0fecd1532d41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251025 07:52:51.272773 3204 consensus_queue.cc:1048] T 72618756f756484e895e885ef12545c8 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251025 07:52:51.273118 450 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b4acedfb7244aa5ae4a0fecd1532d41 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251025 07:52:51.273379 710 leader_election.cc:343] T 421db850a64048218eecb9de65e2bdbf P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:51.273481 710 leader_election.cc:304] T 421db850a64048218eecb9de65e2bdbf P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:51.273562 3204 consensus_queue.cc:1048] T 243b8828fc344bacab46d7bab6bd2dec P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:51.274060 847 leader_election.cc:304] T 9903e80a26fd469a8744dc855534ba75 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:51.274209 3137 raft_consensus.cc:2749] T 421db850a64048218eecb9de65e2bdbf P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:51.274340 3137 consensus_queue.cc:1048] T aea8fcf0154a4bd4a05b19e27ab78b79 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:51.274435 2988 raft_consensus.cc:2804] T 9903e80a26fd469a8744dc855534ba75 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:51.274508 2988 raft_consensus.cc:493] T 9903e80a26fd469a8744dc855534ba75 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:51.274564 2988 raft_consensus.cc:3060] T 9903e80a26fd469a8744dc855534ba75 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:51.274873 3142 raft_consensus.cc:2804] T f7011f1dcad74fe4ba0356c618890e06 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:51.274955 3142 raft_consensus.cc:493] T f7011f1dcad74fe4ba0356c618890e06 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:51.275025 3142 raft_consensus.cc:3060] T f7011f1dcad74fe4ba0356c618890e06 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:51.276860 765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "141e413d2faa4c67b4c822192e631417" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:51.277180 2988 raft_consensus.cc:515] T 9903e80a26fd469a8744dc855534ba75 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.277267 764 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c4970c8a491460e8b4b1c29fde83e4c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:51.277426 764 raft_consensus.cc:2468] T 3c4970c8a491460e8b4b1c29fde83e4c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:51.277478 3142 raft_consensus.cc:515] T f7011f1dcad74fe4ba0356c618890e06 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.277907 449 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 132) took 4468 ms. Trace:
I20251025 07:52:51.278038 449 rpcz_store.cc:276] 1025 07:52:46.809066 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:51.112088 (+4303022us) service_pool.cc:225] Handling call
1025 07:52:51.113065 (+ 977us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:51.113533 (+ 468us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:51.200861 (+ 87328us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:51.277745 (+ 76884us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:51.277873 (+ 128us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:51.162768 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:51.162918 (+ 150us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:51.162924 (+ 6us) write_op.cc:435] Acquired schema lock
1025 07:52:51.162927 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:51.163034 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:51.163046 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:51.163053 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:51.163144 (+ 91us) tablet.cc:699] Row locks acquired
1025 07:52:51.163147 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:51.163181 (+ 34us) write_op.cc:270] Start()
1025 07:52:51.163244 (+ 63us) write_op.cc:276] Timestamp: P: 1761378771163169 usec, L: 0
1025 07:52:51.163248 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:51.163445 (+ 197us) log.cc:844] Serialized 433 byte log entry
1025 07:52:51.199322 (+ 35877us) op_driver.cc:464] REPLICATION: finished
1025 07:52:51.200330 (+ 1008us) write_op.cc:301] APPLY: starting
1025 07:52:51.200345 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:51.200385 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:51.200435 (+ 50us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:51.200540 (+ 105us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:51.200575 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:51.200583 (+ 8us) write_op.cc:312] APPLY: finished
1025 07:52:51.200646 (+ 63us) log.cc:844] Serialized 29 byte log entry
1025 07:52:51.200703 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:51.200720 (+ 17us) write_op.cc:454] Released schema lock
1025 07:52:51.200725 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":964,"apply.run_cpu_time_us":450,"apply.run_wall_time_us":450,"num_ops":2,"prepare.queue_time_us":41,"prepare.run_cpu_time_us":791,"prepare.run_wall_time_us":1147,"raft.queue_time_us":41,"raft.run_cpu_time_us":9,"raft.run_wall_time_us":10,"replication_time_us":36022,"thread_start_us":56760,"threads_started":1}]]}
I20251025 07:52:51.278134 3142 leader_election.cc:290] T f7011f1dcad74fe4ba0356c618890e06 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:51.278860 449 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47610:
name: "default.Table_defef0bab45d4ff8ad287b99761b0fb7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251025 07:52:51.279405 449 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_defef0bab45d4ff8ad287b99761b0fb7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251025 07:52:51.280565 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 645263a61b654f04985c40d04506c087 with delete_type TABLET_DATA_DELETED (replaced by tablet 3ccc61d97e494ce0ab40b118d6f4fd40) from {username='slave'} at 127.0.0.1:58412
I20251025 07:52:51.282047 2988 leader_election.cc:290] T 9903e80a26fd469a8744dc855534ba75 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:51.286801 847 leader_election.cc:304] T 294d7d20c2f545b8813e632aa058a60c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:51.287705 1169 tablet_bootstrap.cc:492] T 72618756f756484e895e885ef12545c8 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:51.287863 1169 ts_tablet_manager.cc:1403] T 72618756f756484e895e885ef12545c8 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.070s user 0.000s sys 0.003s
I20251025 07:52:51.287999 1167 tablet_bootstrap.cc:492] T 2c4789b9d89f42cb844e6cb595e687db P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:51.288125 1167 ts_tablet_manager.cc:1403] T 2c4789b9d89f42cb844e6cb595e687db P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.212s user 0.031s sys 0.019s
I20251025 07:52:51.288172 1039 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7011f1dcad74fe4ba0356c618890e06" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:51.288712 712 leader_election.cc:343] T f7011f1dcad74fe4ba0356c618890e06 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:51.288781 1167 raft_consensus.cc:359] T 2c4789b9d89f42cb844e6cb595e687db P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.288944 1167 raft_consensus.cc:385] T 2c4789b9d89f42cb844e6cb595e687db P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:51.289001 1167 raft_consensus.cc:740] T 2c4789b9d89f42cb844e6cb595e687db P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:51.289207 1167 consensus_queue.cc:260] T 2c4789b9d89f42cb844e6cb595e687db P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.289803 1167 ts_tablet_manager.cc:1434] T 2c4789b9d89f42cb844e6cb595e687db P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251025 07:52:51.290759 1167 tablet_bootstrap.cc:492] T 830fdbc6812e416690d0db42a73af503 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:51.291230 1169 raft_consensus.cc:359] T 72618756f756484e895e885ef12545c8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.291474 1169 raft_consensus.cc:385] T 72618756f756484e895e885ef12545c8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:51.291569 1169 raft_consensus.cc:740] T 72618756f756484e895e885ef12545c8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:51.291570 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 645263a61b654f04985c40d04506c087 with delete_type TABLET_DATA_DELETED (replaced by tablet 3ccc61d97e494ce0ab40b118d6f4fd40) from {username='slave'} at 127.0.0.1:59616
I20251025 07:52:51.291844 1169 consensus_queue.cc:260] T 72618756f756484e895e885ef12545c8 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.292131 2686 raft_consensus.cc:2804] T 294d7d20c2f545b8813e632aa058a60c P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:51.292258 2686 raft_consensus.cc:697] T 294d7d20c2f545b8813e632aa058a60c P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:51.292517 2686 consensus_queue.cc:237] T 294d7d20c2f545b8813e632aa058a60c P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.293828 1166 tablet_bootstrap.cc:492] T 19f1b56cdbd24ebe8688cc05fb795d07 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:51.293964 1166 ts_tablet_manager.cc:1403] T 19f1b56cdbd24ebe8688cc05fb795d07 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.000s
I20251025 07:52:51.294777 1166 raft_consensus.cc:359] T 19f1b56cdbd24ebe8688cc05fb795d07 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.294955 1166 raft_consensus.cc:385] T 19f1b56cdbd24ebe8688cc05fb795d07 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:51.295012 1166 raft_consensus.cc:740] T 19f1b56cdbd24ebe8688cc05fb795d07 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:51.295192 1166 consensus_queue.cc:260] T 19f1b56cdbd24ebe8688cc05fb795d07 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.295794 1166 ts_tablet_manager.cc:1434] T 19f1b56cdbd24ebe8688cc05fb795d07 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:51.296113 1166 tablet_bootstrap.cc:492] T 698e9f8de7db4516a811c9eb9143a268 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:51.301247 2146 ts_tablet_manager.cc:1929] T 4a1ec8c2ed7f41abb588f592f62097d1 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:51.301355 2146 log.cc:1199] T 4a1ec8c2ed7f41abb588f592f62097d1 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/4a1ec8c2ed7f41abb588f592f62097d1
I20251025 07:52:51.301681 2146 ts_tablet_manager.cc:1950] T 4a1ec8c2ed7f41abb588f592f62097d1 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:51.302886 1166 tablet_bootstrap.cc:654] T 698e9f8de7db4516a811c9eb9143a268 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:51.303802 1169 ts_tablet_manager.cc:1434] T 72618756f756484e895e885ef12545c8 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20251025 07:52:51.304306 1169 tablet_bootstrap.cc:492] T 792f32cbd66449ca98813c669b2a261d P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:51.304746 2988 raft_consensus.cc:493] T cc407c7eb12e4eadb41fb21385611c94 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:51.304853 2988 raft_consensus.cc:515] T cc407c7eb12e4eadb41fb21385611c94 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.305349 2988 leader_election.cc:290] T cc407c7eb12e4eadb41fb21385611c94 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:51.312294 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 338) took 3407 ms. Trace:
I20251025 07:52:51.313508 2166 ts_tablet_manager.cc:1916] T ab42909afc524f2fa3c1041495d13a59 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:51.313828 2146 rpcz_store.cc:276] 1025 07:52:47.904351 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:47.904403 (+ 52us) service_pool.cc:225] Handling call
1025 07:52:51.262214 (+3357811us) ts_tablet_manager.cc:1151] Deleting tablet 4a1ec8c2ed7f41abb588f592f62097d1
1025 07:52:51.262225 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.280076 (+ 17851us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.301124 (+ 21048us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.312274 (+ 11150us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3357634}
I20251025 07:52:51.314991 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 4a1ec8c2ed7f41abb588f592f62097d1 (table default.Table_91898f2a551144d18df0309c827855bf [id=aaa734f6549741768b0156bdf5cc25af]) successfully deleted
I20251025 07:52:51.315439 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 341) took 1534 ms. Trace:
I20251025 07:52:51.315621 2146 rpcz_store.cc:276] 1025 07:52:49.781097 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.781172 (+ 75us) service_pool.cc:225] Handling call
1025 07:52:51.315342 (+1534170us) ts_tablet_manager.cc:1151] Deleting tablet 4f0c4f0e8f014e59be9e46d2afe0be9d
1025 07:52:51.315351 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.315424 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1533986}
I20251025 07:52:51.321409 450 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 136) took 4242 ms. Trace:
I20251025 07:52:51.321578 450 rpcz_store.cc:276] 1025 07:52:47.078744 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:51.272329 (+4193585us) service_pool.cc:225] Handling call
1025 07:52:51.273206 (+ 877us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:51.273647 (+ 441us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:51.321096 (+ 47449us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:51.321242 (+ 146us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:51.321360 (+ 118us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:51.284262 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:51.284372 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:51.284378 (+ 6us) write_op.cc:435] Acquired schema lock
1025 07:52:51.284381 (+ 3us) tablet.cc:660] Decoding operations
1025 07:52:51.284483 (+ 102us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:51.284497 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:51.284502 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:51.284573 (+ 71us) tablet.cc:699] Row locks acquired
1025 07:52:51.284575 (+ 2us) write_op.cc:260] PREPARE: finished
1025 07:52:51.284610 (+ 35us) write_op.cc:270] Start()
1025 07:52:51.284645 (+ 35us) write_op.cc:276] Timestamp: P: 1761378771284600 usec, L: 0
1025 07:52:51.284649 (+ 4us) op_driver.cc:348] REPLICATION: starting
1025 07:52:51.284843 (+ 194us) log.cc:844] Serialized 433 byte log entry
1025 07:52:51.319998 (+ 35155us) op_driver.cc:464] REPLICATION: finished
1025 07:52:51.320172 (+ 174us) write_op.cc:301] APPLY: starting
1025 07:52:51.320198 (+ 26us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:51.320257 (+ 59us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:51.320333 (+ 76us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:51.320549 (+ 216us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:51.320598 (+ 49us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:51.320607 (+ 9us) write_op.cc:312] APPLY: finished
1025 07:52:51.320732 (+ 125us) log.cc:844] Serialized 29 byte log entry
1025 07:52:51.320827 (+ 95us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:51.320857 (+ 30us) write_op.cc:454] Released schema lock
1025 07:52:51.320868 (+ 11us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:51.320964 (+ 96us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:51.320969 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":96,"apply.run_cpu_time_us":780,"apply.run_wall_time_us":780,"num_ops":2,"prepare.queue_time_us":10122,"prepare.run_cpu_time_us":718,"prepare.run_wall_time_us":730,"raft.queue_time_us":345,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":211,"replication_time_us":35313}]]}
I20251025 07:52:51.322363 450 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_6fd078e1af7a41f8a24caebf5e12e2b2" } modify_external_catalogs: true
I20251025 07:52:51.322561 450 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_6fd078e1af7a41f8a24caebf5e12e2b2" } modify_external_catalogs: true
I20251025 07:52:51.323966 1166 tablet_bootstrap.cc:492] T 698e9f8de7db4516a811c9eb9143a268 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:51.324108 1166 ts_tablet_manager.cc:1403] T 698e9f8de7db4516a811c9eb9143a268 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.001s
W20251025 07:52:51.324293 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 4f0c4f0e8f014e59be9e46d2afe0be9d because the tablet was not found. No further retry: Not found: Tablet not found: 4f0c4f0e8f014e59be9e46d2afe0be9d
I20251025 07:52:51.324648 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 342) took 1542 ms. Trace:
I20251025 07:52:51.324818 1166 raft_consensus.cc:359] T 698e9f8de7db4516a811c9eb9143a268 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.324972 1166 raft_consensus.cc:385] T 698e9f8de7db4516a811c9eb9143a268 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:51.325095 1166 raft_consensus.cc:740] T 698e9f8de7db4516a811c9eb9143a268 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:51.325260 847 leader_election.cc:304] T 3c4970c8a491460e8b4b1c29fde83e4c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:51.325340 1166 consensus_queue.cc:260] T 698e9f8de7db4516a811c9eb9143a268 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.325846 3143 raft_consensus.cc:493] T 5dab439bc50d4e869548154e1d222ad6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:51.325973 3235 raft_consensus.cc:493] T 6fb1a1d38fb94f07b79ac4f8334fa719 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:51.326093 3235 raft_consensus.cc:515] T 6fb1a1d38fb94f07b79ac4f8334fa719 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.326049 3143 raft_consensus.cc:515] T 5dab439bc50d4e869548154e1d222ad6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.326800 3235 leader_election.cc:290] T 6fb1a1d38fb94f07b79ac4f8334fa719 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:51.326938 3143 leader_election.cc:290] T 5dab439bc50d4e869548154e1d222ad6 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:51.327468 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5dab439bc50d4e869548154e1d222ad6" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:51.327742 764 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5dab439bc50d4e869548154e1d222ad6" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:51.328346 982 leader_election.cc:304] T 5dab439bc50d4e869548154e1d222ad6 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:51.328589 1016 tablet_service.cc:1552] Processing DeleteTablet for tablet 36c414bc700341188b7d85c14173c3e0 with delete_type TABLET_DATA_DELETED (replaced by tablet eced77e6a53447db9fb2df3b670684f2) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.328699 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 52ec92092aaa431f8beba2082336d64c (DEFAULT_TABLE table=default.Table_68774a9f47334984a5e9cfae8b304519 [id=c387599e70124eb1891950bc8371e9a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:51.328851 3143 raft_consensus.cc:2749] T 5dab439bc50d4e869548154e1d222ad6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:51.328886 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet 062c0f3e51a3487880cea373f8cf1f54 with delete_type TABLET_DATA_DELETED (replaced by tablet b56209e4934e4407a8b4518ca4af0d46) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.329137 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52ec92092aaa431f8beba2082336d64c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:51.329388 1013 tablet_service.cc:1552] Processing DeleteTablet for tablet 645263a61b654f04985c40d04506c087 with delete_type TABLET_DATA_DELETED (replaced by tablet 3ccc61d97e494ce0ab40b118d6f4fd40) from {username='slave'} at 127.0.0.1:44572
I20251025 07:52:51.324790 2146 rpcz_store.cc:276] 1025 07:52:49.781768 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.781813 (+ 45us) service_pool.cc:225] Handling call
1025 07:52:51.324555 (+1542742us) ts_tablet_manager.cc:1151] Deleting tablet 80ee75935d7e461d9d4066fac9819bef
1025 07:52:51.324564 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.324633 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1542577}
W20251025 07:52:51.329695 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet a77eb82f26ae43b48de1e85693372d34 because the tablet was not found. No further retry: Not found: Tablet not found: a77eb82f26ae43b48de1e85693372d34
W20251025 07:52:51.329891 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 80ee75935d7e461d9d4066fac9819bef because the tablet was not found. No further retry: Not found: Tablet not found: 80ee75935d7e461d9d4066fac9819bef
W20251025 07:52:51.329952 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 91728987303b41fd840eb56f3a311c5f because the tablet was not found. No further retry: Not found: Tablet not found: 91728987303b41fd840eb56f3a311c5f
W20251025 07:52:51.330116 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet cb829649f73d4d3f8862d268c953bd86 because the tablet was not found. No further retry: Not found: Tablet not found: cb829649f73d4d3f8862d268c953bd86
I20251025 07:52:51.330257 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 343) took 1547 ms. Trace:
W20251025 07:52:51.330355 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 0f5f6ca6c3cd4d369690e425387a4344 because the tablet was not found. No further retry: Not found: Tablet not found: 0f5f6ca6c3cd4d369690e425387a4344
I20251025 07:52:51.330324 2146 rpcz_store.cc:276] 1025 07:52:49.782839 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.782884 (+ 45us) service_pool.cc:225] Handling call
1025 07:52:51.330113 (+1547229us) ts_tablet_manager.cc:1151] Deleting tablet 7e6544202d4f489d82f22d6073ef240d
1025 07:52:51.330124 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.330241 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1547105}
W20251025 07:52:51.330569 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 4c155aa7fa5b4d68bbab1918b8432372 because the tablet was not found. No further retry: Not found: Tablet not found: 4c155aa7fa5b4d68bbab1918b8432372
W20251025 07:52:51.330713 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 7e6544202d4f489d82f22d6073ef240d because the tablet was not found. No further retry: Not found: Tablet not found: 7e6544202d4f489d82f22d6073ef240d
W20251025 07:52:51.330766 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet d9bfd7c0c3ad4e5b98b6708793d61d41 because the tablet was not found. No further retry: Not found: Tablet not found: d9bfd7c0c3ad4e5b98b6708793d61d41
W20251025 07:52:51.330922 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 04449cc97f404a19b6a1b20dea074f95 because the tablet was not found. No further retry: Not found: Tablet not found: 04449cc97f404a19b6a1b20dea074f95
W20251025 07:52:51.331100 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet b3a0af893a8947848e12093d0a7f0229 because the tablet was not found. No further retry: Not found: Tablet not found: b3a0af893a8947848e12093d0a7f0229
W20251025 07:52:51.331305 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet ce1ebb1ec14b4801930a72765bca1a7d because the tablet was not found. No further retry: Not found: Tablet not found: ce1ebb1ec14b4801930a72765bca1a7d
W20251025 07:52:51.331537 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 3b940fa5ae26427dafbcfd26372754b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b940fa5ae26427dafbcfd26372754b3 already in progress: creating tablet
W20251025 07:52:51.331722 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet f43ef51ee45d47a99ec732b49d50d3b4 because the tablet was not found. No further retry: Not found: Tablet not found: f43ef51ee45d47a99ec732b49d50d3b4
W20251025 07:52:51.331918 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 0551e5d694894e8fbeb66b5e0c1cd75b because the tablet was not found. No further retry: Not found: Tablet not found: 0551e5d694894e8fbeb66b5e0c1cd75b
I20251025 07:52:51.333184 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 6d3c9954693d4e8782ee0128ef066aa9 (table default.Table_b936c4aedc5340daa5e60cff2e2ba46e [id=2df257976bfe40a68d54f0cf87dc65a5]). Expected version 1 got 0
I20251025 07:52:51.333359 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 9a8e2658a81e436298fe2190726080d4 (table default.Table_eb2e66417bc9444085fb5ba50ee3853e [id=5c3691c247a448eb985d64e94b1e4ac3]). Expected version 1 got 0
I20251025 07:52:51.333528 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet d46673a283fe42b78ddeb99fbd2678eb (table default.Table_fa83d0528b4b481989bfc0ca35c6de44 [id=bf77b5ff6f1048dd996d948437f9ff5e]). Expected version 3 got 0
I20251025 07:52:51.333794 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 783487b60c144f719ed2eb3c67b5ccab (table default.Table_09095c0829ed4d1cb8b763fc695e7fa8 [id=e372ca4384644779a6b37f389f549dec]). Expected version 1 got 0
I20251025 07:52:51.334265 1167 tablet_bootstrap.cc:654] T 830fdbc6812e416690d0db42a73af503 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:51.334292 2146 tablet_replica.cc:333] T ffe7c8e1d6c44872b7b65e512764a56a P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:51.334483 2146 raft_consensus.cc:2243] T ffe7c8e1d6c44872b7b65e512764a56a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:51.336314 2169 tablet_replica.cc:333] T 36c414bc700341188b7d85c14173c3e0 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:51.336628 2169 raft_consensus.cc:2243] T 36c414bc700341188b7d85c14173c3e0 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:51.334873 2988 raft_consensus.cc:2804] T 3c4970c8a491460e8b4b1c29fde83e4c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:51.336731 2988 raft_consensus.cc:493] T 3c4970c8a491460e8b4b1c29fde83e4c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:51.336830 2988 raft_consensus.cc:3060] T 3c4970c8a491460e8b4b1c29fde83e4c P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:51.336978 2169 raft_consensus.cc:2272] T 36c414bc700341188b7d85c14173c3e0 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:51.337575 2169 ts_tablet_manager.cc:1916] T 36c414bc700341188b7d85c14173c3e0 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:51.339558 2988 raft_consensus.cc:515] T 3c4970c8a491460e8b4b1c29fde83e4c P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.340119 2988 leader_election.cc:290] T 3c4970c8a491460e8b4b1c29fde83e4c P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:51.335427 1166 ts_tablet_manager.cc:1434] T 698e9f8de7db4516a811c9eb9143a268 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20251025 07:52:51.335794 3122 raft_consensus.cc:493] T 2c4789b9d89f42cb844e6cb595e687db P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:51.340404 3122 raft_consensus.cc:515] T 2c4789b9d89f42cb844e6cb595e687db P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.340550 1166 tablet_bootstrap.cc:492] T 7e18611cbc1348eca927162fec70e36f P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:51.340517 1039 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fb1a1d38fb94f07b79ac4f8334fa719" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:51.340916 3122 leader_election.cc:290] T 2c4789b9d89f42cb844e6cb595e687db P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:51.341663 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 72618756f756484e895e885ef12545c8 (table default.Table_8b4581839f31463bb6b7b91a6cf918e8 [id=2965a45ad79e4523ab42d65823f19c8f]). Expected version 1 got 0
I20251025 07:52:51.341979 1166 tablet_bootstrap.cc:654] T 7e18611cbc1348eca927162fec70e36f P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:51.341961 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 30418858bf95438686bdfe64ac733144 (table default.Table_7da05a15ca474e779a81fecd62328481 [id=b4379401507b4a6f916f131a1b3a75d7]). Expected version 1 got 0
I20251025 07:52:51.342396 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 872cb7512d104827b1f1f186d33a4a44 (table default.Table_d5c65dce8b1b4214ba519d964012bfc0 [id=d4be274b36a845a99a4b96b8846248fd]). Expected version 1 got 0
I20251025 07:52:51.343580 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 6fe43ccecf7047319090ebd035fe24a5 (table default.Table_68b39cfaff194c3d8dbe9eaee7d8b44c [id=c95fc6d40893462eacee1276fb9cacfe]). Expected version 1 got 0
I20251025 07:52:51.344452 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 5dab439bc50d4e869548154e1d222ad6 (table default.Table_fb9def2302784c999bf0f5707c60eb4c [id=45325c188eac4d34bb016d01de091f87]). Expected version 3 got 0
I20251025 07:52:51.344648 2146 raft_consensus.cc:2272] T ffe7c8e1d6c44872b7b65e512764a56a P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:51.346886 449 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:47610 (request call id 137) took 4268 ms. Trace:
I20251025 07:52:51.347026 449 rpcz_store.cc:276] 1025 07:52:47.078861 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:51.278661 (+4199800us) service_pool.cc:225] Handling call
1025 07:52:51.279505 (+ 844us) catalog_manager.cc:2101] Acquired catalog manager lock
1025 07:52:51.279986 (+ 481us) catalog_manager.cc:2147] Created new table and tablet info
1025 07:52:51.346634 (+ 66648us) catalog_manager.cc:2207] Wrote table and tablets to system table
1025 07:52:51.346740 (+ 106us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1025 07:52:51.346835 (+ 95us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1025 07:52:51.285446 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1025 07:52:51.285553 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
1025 07:52:51.285558 (+ 5us) write_op.cc:435] Acquired schema lock
1025 07:52:51.285560 (+ 2us) tablet.cc:660] Decoding operations
1025 07:52:51.285633 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
1025 07:52:51.285643 (+ 10us) write_op.cc:641] Partition lock acquired for write op
1025 07:52:51.285647 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1025 07:52:51.285707 (+ 60us) tablet.cc:699] Row locks acquired
1025 07:52:51.285710 (+ 3us) write_op.cc:260] PREPARE: finished
1025 07:52:51.285737 (+ 27us) write_op.cc:270] Start()
1025 07:52:51.285762 (+ 25us) write_op.cc:276] Timestamp: P: 1761378771285727 usec, L: 0
1025 07:52:51.285765 (+ 3us) op_driver.cc:348] REPLICATION: starting
1025 07:52:51.285900 (+ 135us) log.cc:844] Serialized 433 byte log entry
1025 07:52:51.345856 (+ 59956us) op_driver.cc:464] REPLICATION: finished
1025 07:52:51.346015 (+ 159us) write_op.cc:301] APPLY: starting
1025 07:52:51.346031 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1025 07:52:51.346072 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
1025 07:52:51.346103 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
1025 07:52:51.346224 (+ 121us) tablet.cc:1381] finished ApplyRowOperation cycle
1025 07:52:51.346255 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1025 07:52:51.346262 (+ 7us) write_op.cc:312] APPLY: finished
1025 07:52:51.346357 (+ 95us) log.cc:844] Serialized 29 byte log entry
1025 07:52:51.346428 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:51.346453 (+ 25us) write_op.cc:454] Released schema lock
1025 07:52:51.346461 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1025 07:52:51.346533 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
1025 07:52:51.346537 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":108,"apply.run_cpu_time_us":507,"apply.run_wall_time_us":507,"mutex_wait_us":4896,"num_ops":2,"prepare.queue_time_us":4948,"prepare.run_cpu_time_us":533,"prepare.run_wall_time_us":533,"raft.queue_time_us":4140,"raft.run_cpu_time_us":7,"raft.run_wall_time_us":8,"replication_time_us":60052}]]}
I20251025 07:52:51.347813 449 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_45faec266a9b4b1b9f5ef3ac915e3bbc" } modify_external_catalogs: true
I20251025 07:52:51.347929 2146 ts_tablet_manager.cc:1916] T ffe7c8e1d6c44872b7b65e512764a56a P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:51.348002 449 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47610:
table { table_name: "default.Table_45faec266a9b4b1b9f5ef3ac915e3bbc" } modify_external_catalogs: true
I20251025 07:52:51.348260 2169 ts_tablet_manager.cc:1929] T 36c414bc700341188b7d85c14173c3e0 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:51.348356 2169 log.cc:1199] T 36c414bc700341188b7d85c14173c3e0 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/36c414bc700341188b7d85c14173c3e0
I20251025 07:52:51.348737 2169 ts_tablet_manager.cc:1950] T 36c414bc700341188b7d85c14173c3e0 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:51.347029 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 5fc092c2bb2d41ac8c5b4fc07460c8ec (table default.Table_cd20e202397645f8821dcde13b5c0ce3 [id=630c274339664b9db18ab4ef7c29c16f]). Expected version 2 got 0
I20251025 07:52:51.349153 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 3760348dbe284c41a01a4fe479a2103f (table default.Table_088aad2a46b041bfa29ed8e6929760b1 [id=834669c54dae449790818684fd8659bc]). Expected version 2 got 0
I20251025 07:52:51.349335 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 48b73e0c0dce4be686d05f991a21bd19 (table default.Table_1d0f9cc8a4004024af7c174b2d0a72aa [id=4d9e99d6474b4d00bc66e26a33694e69]). Expected version 3 got 0
I20251025 07:52:51.349578 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 22f7db660616457b83b24e947a0b0bae (table default.Table_4a86191163f2408f9bcc795cd8d1bbf6 [id=bb316a7e000748ca902a9bfb494802d7]). Expected version 1 got 0
I20251025 07:52:51.349848 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet e8402fd33ba842e981c6c68787c7c548 (table default.Table_45b5db2f6c964ec583835ac9ca079e81 [id=791652590f3b41849103ebd353672f8e]). Expected version 1 got 0
I20251025 07:52:51.351277 1166 tablet_bootstrap.cc:492] T 7e18611cbc1348eca927162fec70e36f P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:51.351399 1166 ts_tablet_manager.cc:1403] T 7e18611cbc1348eca927162fec70e36f P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20251025 07:52:51.352018 1166 raft_consensus.cc:359] T 7e18611cbc1348eca927162fec70e36f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.352180 1166 raft_consensus.cc:385] T 7e18611cbc1348eca927162fec70e36f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:51.352227 1166 raft_consensus.cc:740] T 7e18611cbc1348eca927162fec70e36f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:51.352404 1166 consensus_queue.cc:260] T 7e18611cbc1348eca927162fec70e36f P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.353332 1166 ts_tablet_manager.cc:1434] T 7e18611cbc1348eca927162fec70e36f P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:51.353596 1166 tablet_bootstrap.cc:492] T a6587f4be6af44dd944e6b6f9599dfda P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:51.354951 1166 tablet_bootstrap.cc:654] T a6587f4be6af44dd944e6b6f9599dfda P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:51.340227 771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9903e80a26fd469a8744dc855534ba75" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:51.360147 771 raft_consensus.cc:3060] T 9903e80a26fd469a8744dc855534ba75 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:51.361294 2169 tablet_replica.cc:333] T 645263a61b654f04985c40d04506c087 P db42be7ad3c741a0aa427f27e245993b: stopping tablet replica
I20251025 07:52:51.361470 2169 raft_consensus.cc:2243] T 645263a61b654f04985c40d04506c087 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:51.361613 2169 raft_consensus.cc:2272] T 645263a61b654f04985c40d04506c087 P db42be7ad3c741a0aa427f27e245993b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:51.362103 2169 ts_tablet_manager.cc:1916] T 645263a61b654f04985c40d04506c087 P db42be7ad3c741a0aa427f27e245993b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:51.341097 712 leader_election.cc:343] T 6fb1a1d38fb94f07b79ac4f8334fa719 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
W20251025 07:52:51.363245 712 consensus_peers.cc:597] T d40519ed9ee746348b17be2786d60e95 P 23eeffd3de254ce18db5febfde971c6b -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:51.364163 771 raft_consensus.cc:2468] T 9903e80a26fd469a8744dc855534ba75 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
W20251025 07:52:51.342684 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:43563, user_credentials={real_user=slave}} blocked reactor thread for 68721.1us
I20251025 07:52:51.364655 710 leader_election.cc:304] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:51.365120 710 leader_election.cc:304] T 3c4970c8a491460e8b4b1c29fde83e4c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:51.365324 3142 raft_consensus.cc:2804] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:51.365387 3142 raft_consensus.cc:493] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:51.365444 3142 raft_consensus.cc:3060] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:51.368223 3142 raft_consensus.cc:515] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.369025 3142 leader_election.cc:290] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:51.369220 3142 raft_consensus.cc:493] T a6587f4be6af44dd944e6b6f9599dfda P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:51.369352 1039 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5fc092c2bb2d41ac8c5b4fc07460c8ec" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:51.369374 3142 raft_consensus.cc:515] T a6587f4be6af44dd944e6b6f9599dfda P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.369853 3142 leader_election.cc:290] T a6587f4be6af44dd944e6b6f9599dfda P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:51.368408 771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc407c7eb12e4eadb41fb21385611c94" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
W20251025 07:52:51.369956 712 leader_election.cc:343] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:51.368494 3236 raft_consensus.cc:2804] T 3c4970c8a491460e8b4b1c29fde83e4c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:51.370100 3236 raft_consensus.cc:493] T 3c4970c8a491460e8b4b1c29fde83e4c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:51.370208 3236 raft_consensus.cc:3060] T 3c4970c8a491460e8b4b1c29fde83e4c P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:51.370512 1039 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6587f4be6af44dd944e6b6f9599dfda" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:51.371085 712 leader_election.cc:343] T a6587f4be6af44dd944e6b6f9599dfda P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:51.373005 3236 raft_consensus.cc:515] T 3c4970c8a491460e8b4b1c29fde83e4c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.374002 3236 leader_election.cc:290] T 3c4970c8a491460e8b4b1c29fde83e4c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:51.374464 2686 raft_consensus.cc:493] T 7e18611cbc1348eca927162fec70e36f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:51.374627 2686 raft_consensus.cc:515] T 7e18611cbc1348eca927162fec70e36f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.377132 2146 ts_tablet_manager.cc:1929] T ffe7c8e1d6c44872b7b65e512764a56a P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:51.378755 1039 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c4970c8a491460e8b4b1c29fde83e4c" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:51.378926 2166 ts_tablet_manager.cc:1929] T ab42909afc524f2fa3c1041495d13a59 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251025 07:52:51.379273 712 leader_election.cc:343] T 3c4970c8a491460e8b4b1c29fde83e4c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:51.373303 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 830fdbc6812e416690d0db42a73af503 (table default.Table_92824b3f2a774564b3aea04aaa25d204 [id=fb21a45b86ab446e8c3dae4512a9598f]). Expected version 1 got 0
I20251025 07:52:51.379940 2166 log.cc:1199] T ab42909afc524f2fa3c1041495d13a59 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/ab42909afc524f2fa3c1041495d13a59
I20251025 07:52:51.380370 2166 ts_tablet_manager.cc:1950] T ab42909afc524f2fa3c1041495d13a59 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:51.381111 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 306) took 3509 ms. Trace:
I20251025 07:52:51.381239 2166 rpcz_store.cc:276] 1025 07:52:47.872148 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:47.872220 (+ 72us) service_pool.cc:225] Handling call
1025 07:52:51.269613 (+3397393us) ts_tablet_manager.cc:1151] Deleting tablet ab42909afc524f2fa3c1041495d13a59
1025 07:52:51.269621 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.324421 (+ 54800us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.378829 (+ 54408us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.381095 (+ 2266us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3397232}
I20251025 07:52:51.381830 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet ab42909afc524f2fa3c1041495d13a59 (table default.Table_7eb8424d5192485da1a2d67bfdb2bbbf [id=40c29df99590408cbd5c583d4fc741eb]) successfully deleted
I20251025 07:52:51.382283 2166 tablet_replica.cc:333] T 4a1ec8c2ed7f41abb588f592f62097d1 P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:51.382535 2166 raft_consensus.cc:2243] T 4a1ec8c2ed7f41abb588f592f62097d1 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:51.382766 2166 raft_consensus.cc:2272] T 4a1ec8c2ed7f41abb588f592f62097d1 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:51.383344 2166 ts_tablet_manager.cc:1916] T 4a1ec8c2ed7f41abb588f592f62097d1 P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:51.383503 2146 log.cc:1199] T ffe7c8e1d6c44872b7b65e512764a56a P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/ffe7c8e1d6c44872b7b65e512764a56a
I20251025 07:52:51.383803 2146 ts_tablet_manager.cc:1950] T ffe7c8e1d6c44872b7b65e512764a56a P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:51.384508 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 344) took 1600 ms. Trace:
I20251025 07:52:51.384619 2146 rpcz_store.cc:276] 1025 07:52:49.783725 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.783769 (+ 44us) service_pool.cc:225] Handling call
1025 07:52:51.334248 (+1550479us) ts_tablet_manager.cc:1151] Deleting tablet ffe7c8e1d6c44872b7b65e512764a56a
1025 07:52:51.334257 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.353351 (+ 19094us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.377029 (+ 23678us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.384490 (+ 7461us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1550337}
I20251025 07:52:51.385093 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 345) took 1599 ms. Trace:
I20251025 07:52:51.385188 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet ffe7c8e1d6c44872b7b65e512764a56a (table default.Table_d9880621c8ca4025b89b6c1ec47f5cb4 [id=be817d9282bd4fbf84adf11e25f06b5d]) successfully deleted
I20251025 07:52:51.385179 2146 rpcz_store.cc:276] 1025 07:52:49.785263 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.785312 (+ 49us) service_pool.cc:225] Handling call
1025 07:52:51.384976 (+1599664us) ts_tablet_manager.cc:1151] Deleting tablet dda0b26e4f1a44928bc4b84f333901c0
1025 07:52:51.384990 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.385077 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1599539}
I20251025 07:52:51.385527 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 346) took 1599 ms. Trace:
I20251025 07:52:51.385588 2146 rpcz_store.cc:276] 1025 07:52:49.785806 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.785851 (+ 45us) service_pool.cc:225] Handling call
1025 07:52:51.385449 (+1599598us) ts_tablet_manager.cc:1151] Deleting tablet 80fe3255ed614270bd8352a2853fbc43
1025 07:52:51.385459 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.385514 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1599461}
W20251025 07:52:51.385766 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet dda0b26e4f1a44928bc4b84f333901c0 because the tablet was not found. No further retry: Not found: Tablet not found: dda0b26e4f1a44928bc4b84f333901c0
I20251025 07:52:51.385813 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 347) took 1599 ms. Trace:
I20251025 07:52:51.385834 909 raft_consensus.cc:2468] T 141e413d2faa4c67b4c822192e631417 P 3aea2869ff814cc7b142012faa274382 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 2.
I20251025 07:52:51.385864 2146 rpcz_store.cc:276] 1025 07:52:49.786396 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.786446 (+ 50us) service_pool.cc:225] Handling call
1025 07:52:51.385759 (+1599313us) ts_tablet_manager.cc:1151] Deleting tablet 5b6383c99e9e4c90a374da9ccd830a6f
1025 07:52:51.385766 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.385804 (+ 38us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1599182}
I20251025 07:52:51.386066 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 348) took 1599 ms. Trace:
I20251025 07:52:51.382421 2686 leader_election.cc:290] T 7e18611cbc1348eca927162fec70e36f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
W20251025 07:52:51.386507 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 80fe3255ed614270bd8352a2853fbc43 because the tablet was not found. No further retry: Not found: Tablet not found: 80fe3255ed614270bd8352a2853fbc43
I20251025 07:52:51.386118 2146 rpcz_store.cc:276] 1025 07:52:49.786988 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.787040 (+ 52us) service_pool.cc:225] Handling call
1025 07:52:51.386018 (+1598978us) ts_tablet_manager.cc:1151] Deleting tablet fce535a0c0bd45c6bf17f81add297f40
1025 07:52:51.386024 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.386057 (+ 33us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1598847}
I20251025 07:52:51.388046 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 349) took 1600 ms. Trace:
I20251025 07:52:51.387955 771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c4789b9d89f42cb844e6cb595e687db" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:51.388110 2146 rpcz_store.cc:276] 1025 07:52:49.787585 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.787655 (+ 70us) service_pool.cc:225] Handling call
1025 07:52:51.387972 (+1600317us) ts_tablet_manager.cc:1151] Deleting tablet ce1ebb1ec14b4801930a72765bca1a7d
1025 07:52:51.387980 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.388033 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1600162}
I20251025 07:52:51.388281 771 raft_consensus.cc:2468] T 2c4789b9d89f42cb844e6cb595e687db P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:51.388351 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 350) took 1600 ms. Trace:
I20251025 07:52:51.388411 2146 rpcz_store.cc:276] 1025 07:52:49.788273 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.788367 (+ 94us) service_pool.cc:225] Handling call
1025 07:52:51.388286 (+1599919us) ts_tablet_manager.cc:1151] Deleting tablet 91728987303b41fd840eb56f3a311c5f
1025 07:52:51.388293 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.388339 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1599760}
I20251025 07:52:51.388639 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 351) took 1599 ms. Trace:
I20251025 07:52:51.388692 2146 rpcz_store.cc:276] 1025 07:52:49.789029 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.789096 (+ 67us) service_pool.cc:225] Handling call
1025 07:52:51.388579 (+1599483us) ts_tablet_manager.cc:1151] Deleting tablet ebc824ec4c60411dab1a2ea4a385148e
1025 07:52:51.388586 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.388628 (+ 42us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1599349}
I20251025 07:52:51.388907 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 352) took 1599 ms. Trace:
I20251025 07:52:51.388958 2146 rpcz_store.cc:276] 1025 07:52:49.789894 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.789938 (+ 44us) service_pool.cc:225] Handling call
1025 07:52:51.388851 (+1598913us) ts_tablet_manager.cc:1151] Deleting tablet a77eb82f26ae43b48de1e85693372d34
1025 07:52:51.388858 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.388897 (+ 39us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1598781}
I20251025 07:52:51.389186 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 353) took 1598 ms. Trace:
I20251025 07:52:51.389261 2146 rpcz_store.cc:276] 1025 07:52:49.790525 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.790568 (+ 43us) service_pool.cc:225] Handling call
1025 07:52:51.389122 (+1598554us) ts_tablet_manager.cc:1151] Deleting tablet 050a5d3b5f934dd19bf2e221f2fe0780
1025 07:52:51.389130 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.389176 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1598409}
I20251025 07:52:51.389490 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 354) took 1598 ms. Trace:
I20251025 07:52:51.389542 2146 rpcz_store.cc:276] 1025 07:52:49.791259 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.792480 (+ 1221us) service_pool.cc:225] Handling call
1025 07:52:51.389430 (+1596950us) ts_tablet_manager.cc:1151] Deleting tablet 36c8fcafd5464b4e999276a8062a812a
1025 07:52:51.389438 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.389479 (+ 41us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1596814}
I20251025 07:52:51.395388 1169 tablet_bootstrap.cc:654] T 792f32cbd66449ca98813c669b2a261d P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:51.396313 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 5b6383c99e9e4c90a374da9ccd830a6f because the tablet was not found. No further retry: Not found: Tablet not found: 5b6383c99e9e4c90a374da9ccd830a6f
W20251025 07:52:51.396561 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet fce535a0c0bd45c6bf17f81add297f40 because the tablet was not found. No further retry: Not found: Tablet not found: fce535a0c0bd45c6bf17f81add297f40
W20251025 07:52:51.396731 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet ce1ebb1ec14b4801930a72765bca1a7d because the tablet was not found. No further retry: Not found: Tablet not found: ce1ebb1ec14b4801930a72765bca1a7d
W20251025 07:52:51.396888 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 91728987303b41fd840eb56f3a311c5f because the tablet was not found. No further retry: Not found: Tablet not found: 91728987303b41fd840eb56f3a311c5f
W20251025 07:52:51.397019 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet ebc824ec4c60411dab1a2ea4a385148e because the tablet was not found. No further retry: Not found: Tablet not found: ebc824ec4c60411dab1a2ea4a385148e
W20251025 07:52:51.397161 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet a77eb82f26ae43b48de1e85693372d34 because the tablet was not found. No further retry: Not found: Tablet not found: a77eb82f26ae43b48de1e85693372d34
W20251025 07:52:51.397315 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 050a5d3b5f934dd19bf2e221f2fe0780 because the tablet was not found. No further retry: Not found: Tablet not found: 050a5d3b5f934dd19bf2e221f2fe0780
W20251025 07:52:51.397513 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 36c8fcafd5464b4e999276a8062a812a because the tablet was not found. No further retry: Not found: Tablet not found: 36c8fcafd5464b4e999276a8062a812a
I20251025 07:52:51.398034 764 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c4970c8a491460e8b4b1c29fde83e4c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:51.398294 764 raft_consensus.cc:2393] T 3c4970c8a491460e8b4b1c29fde83e4c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3aea2869ff814cc7b142012faa274382 in current term 1: Already voted for candidate 23eeffd3de254ce18db5febfde971c6b in this term.
I20251025 07:52:51.399780 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7011f1dcad74fe4ba0356c618890e06" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:51.399959 902 raft_consensus.cc:3060] T f7011f1dcad74fe4ba0356c618890e06 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:51.400233 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6587f4be6af44dd944e6b6f9599dfda" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:51.400619 910 raft_consensus.cc:1275] T 6d3c9954693d4e8782ee0128ef066aa9 P 3aea2869ff814cc7b142012faa274382 [term 2 FOLLOWER]: Refusing update from remote peer 23eeffd3de254ce18db5febfde971c6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251025 07:52:51.400868 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c4970c8a491460e8b4b1c29fde83e4c" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:51.401033 912 raft_consensus.cc:2393] T 3c4970c8a491460e8b4b1c29fde83e4c P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 23eeffd3de254ce18db5febfde971c6b in current term 1: Already voted for candidate 3aea2869ff814cc7b142012faa274382 in this term.
I20251025 07:52:51.401568 847 leader_election.cc:304] T 9903e80a26fd469a8744dc855534ba75 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:51.401782 2988 raft_consensus.cc:2804] T 9903e80a26fd469a8744dc855534ba75 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:51.401870 2988 raft_consensus.cc:697] T 9903e80a26fd469a8744dc855534ba75 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:51.402076 2988 consensus_queue.cc:237] T 9903e80a26fd469a8744dc855534ba75 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.402863 764 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e18611cbc1348eca927162fec70e36f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:51.403067 764 raft_consensus.cc:2468] T 7e18611cbc1348eca927162fec70e36f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:51.406792 902 raft_consensus.cc:2468] T f7011f1dcad74fe4ba0356c618890e06 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:51.410826 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 141e413d2faa4c67b4c822192e631417 (table default.Table_25a54dc35a8a4e15aff640922ebf13cf [id=8c364800b3f34fa39a375f2f8196b525]). Expected version 1 got 0
I20251025 07:52:51.412420 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 755f0d6c87044e20a8020f8410940123 (table default.Table_5344c3d93df647aba4baf40f77b9191b [id=fa9b211cb1cb437992aef4ed8f2e7439]). Expected version 4 got 0
I20251025 07:52:51.415048 1167 tablet_bootstrap.cc:492] T 830fdbc6812e416690d0db42a73af503 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:51.415858 907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5fc092c2bb2d41ac8c5b4fc07460c8ec" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:51.390651 1166 tablet_bootstrap.cc:492] T a6587f4be6af44dd944e6b6f9599dfda P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:51.416043 907 raft_consensus.cc:3060] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:51.416177 1166 ts_tablet_manager.cc:1403] T a6587f4be6af44dd944e6b6f9599dfda P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.003s
I20251025 07:52:51.417025 1166 raft_consensus.cc:359] T a6587f4be6af44dd944e6b6f9599dfda P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.417173 1166 raft_consensus.cc:385] T a6587f4be6af44dd944e6b6f9599dfda P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:51.417233 1166 raft_consensus.cc:740] T a6587f4be6af44dd944e6b6f9599dfda P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:51.418185 905 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fb1a1d38fb94f07b79ac4f8334fa719" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:51.418315 1167 ts_tablet_manager.cc:1403] T 830fdbc6812e416690d0db42a73af503 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.128s user 0.001s sys 0.004s
I20251025 07:52:51.418780 1166 consensus_queue.cc:260] T a6587f4be6af44dd944e6b6f9599dfda P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.419013 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet e330e1b53bed4e7cace1c4fb7fa6b565 (table default.Table_a9026f5e095c4063b86d4c544c686e22 [id=7944225d3ce7439b881bfb9289d9f040]). Expected version 2 got 0
I20251025 07:52:51.419231 1166 ts_tablet_manager.cc:1434] T a6587f4be6af44dd944e6b6f9599dfda P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251025 07:52:51.419312 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet f70091024c034c118fb86578db19a31c (table default.Table_b7da40b96d6c4359bed6f898e8cf6275 [id=e041946c093b4c789ce175c736cc230b]). Expected version 2 got 0
I20251025 07:52:51.419477 1166 tablet_bootstrap.cc:492] T 0540d6a10cf04fa89756623e9aaaeb68 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:51.419909 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 3a0f50c8000144a4acb5219484b01800 (table default.Table_c62e0dc1ae504f9bb912013506ff3031 [id=6f395892b09240fea02482b3199cfb5e]). Expected version 1 got 0
I20251025 07:52:51.421159 907 raft_consensus.cc:2468] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
W20251025 07:52:51.417112 847 leader_election.cc:343] T cc407c7eb12e4eadb41fb21385611c94 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:51.426782 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 355) took 1628 ms. Trace:
I20251025 07:52:51.426946 2146 rpcz_store.cc:276] 1025 07:52:49.797939 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.797994 (+ 55us) service_pool.cc:225] Handling call
1025 07:52:51.391007 (+1593013us) ts_tablet_manager.cc:1151] Deleting tablet 04449cc97f404a19b6a1b20dea074f95
1025 07:52:51.407553 (+ 16546us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.426763 (+ 19210us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1592863}
I20251025 07:52:51.427474 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 356) took 1627 ms. Trace:
I20251025 07:52:51.427613 2146 rpcz_store.cc:276] 1025 07:52:49.799541 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.799586 (+ 45us) service_pool.cc:225] Handling call
1025 07:52:51.427375 (+1627789us) ts_tablet_manager.cc:1151] Deleting tablet f43ef51ee45d47a99ec732b49d50d3b4
1025 07:52:51.427385 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.427458 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1627632}
I20251025 07:52:51.427614 1167 raft_consensus.cc:359] T 830fdbc6812e416690d0db42a73af503 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.427770 1167 raft_consensus.cc:385] T 830fdbc6812e416690d0db42a73af503 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:51.427842 1167 raft_consensus.cc:740] T 830fdbc6812e416690d0db42a73af503 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:51.427987 2146 tablet_replica.cc:333] T 3b940fa5ae26427dafbcfd26372754b3 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:51.428126 1167 consensus_queue.cc:260] T 830fdbc6812e416690d0db42a73af503 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.428218 2146 raft_consensus.cc:2243] T 3b940fa5ae26427dafbcfd26372754b3 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:51.428354 2146 raft_consensus.cc:2272] T 3b940fa5ae26427dafbcfd26372754b3 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:51.428947 2146 ts_tablet_manager.cc:1916] T 3b940fa5ae26427dafbcfd26372754b3 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:51.429667 1167 ts_tablet_manager.cc:1434] T 830fdbc6812e416690d0db42a73af503 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20251025 07:52:51.430612 1167 tablet_bootstrap.cc:492] T 698e9f8de7db4516a811c9eb9143a268 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:51.431977 1167 tablet_bootstrap.cc:654] T 698e9f8de7db4516a811c9eb9143a268 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:51.432389 1166 tablet_bootstrap.cc:654] T 0540d6a10cf04fa89756623e9aaaeb68 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:51.426856 847 leader_election.cc:304] T 2c4789b9d89f42cb844e6cb595e687db P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
W20251025 07:52:51.428857 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 04449cc97f404a19b6a1b20dea074f95 because the tablet was not found. No further retry: Not found: Tablet not found: 04449cc97f404a19b6a1b20dea074f95
I20251025 07:52:51.442399 2990 raft_consensus.cc:2804] T 2c4789b9d89f42cb844e6cb595e687db P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:51.442921 2990 raft_consensus.cc:493] T 2c4789b9d89f42cb844e6cb595e687db P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251025 07:52:51.442916 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet f43ef51ee45d47a99ec732b49d50d3b4 because the tablet was not found. No further retry: Not found: Tablet not found: f43ef51ee45d47a99ec732b49d50d3b4
I20251025 07:52:51.443014 2990 raft_consensus.cc:3060] T 2c4789b9d89f42cb844e6cb595e687db P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:51.443252 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 9903e80a26fd469a8744dc855534ba75 (table default.Table_3297a14e9a534d31bbe2239df6f6ee37 [id=f6e8d725be3e47d3a3f682510f90b236]). Expected version 1 got 0
I20251025 07:52:51.443404 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 8707b93d491e4e4386d148087a2f0870 (table default.Table_799dcb1cef2f4fe6b9b8f17f9f1026c5 [id=59840a567e5244e8a9024cbad42471e4]). Expected version 1 got 0
I20251025 07:52:51.444006 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet df0d86ad301245c1a6bcb44805936d25 (table default.Table_67b99749f3bf43228b40d8b9807e69b4 [id=fc8180c09d7b4e74b2e161f7acb8fe0e]). Expected version 2 got 0
I20251025 07:52:51.444320 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet ed243540f2d746b980c25687fbca7414 (table default.Table_616a4d314c7445b0b4f4283d76c0bea8 [id=7507e4bd03ca4dd2a93655b138e8d49c]). Expected version 1 got 0
I20251025 07:52:51.444561 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet b56209e4934e4407a8b4518ca4af0d46 (table default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de]). Expected version 1 got 0
I20251025 07:52:51.444831 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet fa826b85f0c94150a17d4e417dd3d62f (table default.Table_54ea01577e6f4faea38bddfc2d04aeb2 [id=b2486c1de45d47eeb272a605ce7fdf84]). Expected version 1 got 0
I20251025 07:52:51.445021 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet c501ef5d70e742598c1a902efe47feb1 (table default.Table_d11bba62f65740319701b4337a5c255a [id=ad812862698d421184ebf501ce79b7be]). Expected version 3 got 0
I20251025 07:52:51.445600 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 288e543bb279410c91c01dd1623e110f (table default.Table_6e242ba55b194c23a6701f4b064fdbe5 [id=4c34b64b299f4162a53866cf01c5298f]). Expected version 1 got 0
I20251025 07:52:51.446954 2990 raft_consensus.cc:515] T 2c4789b9d89f42cb844e6cb595e687db P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:51.447168 712 consensus_peers.cc:597] T 22f7db660616457b83b24e947a0b0bae P 23eeffd3de254ce18db5febfde971c6b -> Peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Couldn't send request to peer db42be7ad3c741a0aa427f27e245993b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:51.447513 2169 ts_tablet_manager.cc:1929] T 645263a61b654f04985c40d04506c087 P db42be7ad3c741a0aa427f27e245993b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:51.447646 2169 log.cc:1199] T 645263a61b654f04985c40d04506c087 P db42be7ad3c741a0aa427f27e245993b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-2/wal/wals/645263a61b654f04985c40d04506c087
I20251025 07:52:51.448122 2169 ts_tablet_manager.cc:1950] T 645263a61b654f04985c40d04506c087 P db42be7ad3c741a0aa427f27e245993b: Deleting consensus metadata
I20251025 07:52:51.448411 847 leader_election.cc:304] T 7e18611cbc1348eca927162fec70e36f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:51.448783 2990 leader_election.cc:290] T 2c4789b9d89f42cb844e6cb595e687db P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:51.448982 2988 raft_consensus.cc:2804] T 7e18611cbc1348eca927162fec70e36f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:51.449043 2988 raft_consensus.cc:493] T 7e18611cbc1348eca927162fec70e36f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:51.449095 2988 raft_consensus.cc:3060] T 7e18611cbc1348eca927162fec70e36f P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:51.451766 2988 raft_consensus.cc:515] T 7e18611cbc1348eca927162fec70e36f P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.452483 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 37c7938e8c5e41f08506aff6747801c9 (table default.Table_df5799a9181a48bb84af0f3c767e7c29 [id=4b4b515be910449395ba4de41a681ec3]). Expected version 2 got 0
I20251025 07:52:51.453316 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet 075f0a151e024220b51ba7fcaecdcc3d (table default.Table_cb2c8357cfa2432cbc3d708c62222479 [id=b064a22cfdc04a0793ffd858b3fdc45e]). Expected version 1 got 0
I20251025 07:52:51.453416 2988 leader_election.cc:290] T 7e18611cbc1348eca927162fec70e36f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
W20251025 07:52:51.455384 710 leader_election.cc:343] T a6587f4be6af44dd944e6b6f9599dfda P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:51.455510 710 leader_election.cc:304] T a6587f4be6af44dd944e6b6f9599dfda P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:51.455731 3235 raft_consensus.cc:2749] T a6587f4be6af44dd944e6b6f9599dfda P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:51.456421 3235 consensus_queue.cc:1048] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:51.456640 710 leader_election.cc:304] T 3c4970c8a491460e8b4b1c29fde83e4c P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:51.456947 3235 raft_consensus.cc:2749] T 3c4970c8a491460e8b4b1c29fde83e4c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251025 07:52:51.457440 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet d0240d20f9904b4bbc8bf7f0f05e8850 (table default.Table_41c7a8a9dce040e790987150ecf0c1c1 [id=03f5437c8d3d4650bd465f154cc97fc7]). Expected version 1 got 0
I20251025 07:52:51.458438 3235 raft_consensus.cc:493] T 830fdbc6812e416690d0db42a73af503 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:51.458563 3235 raft_consensus.cc:515] T 830fdbc6812e416690d0db42a73af503 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.459075 3235 leader_election.cc:290] T 830fdbc6812e416690d0db42a73af503 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:51.460391 710 leader_election.cc:304] T f7011f1dcad74fe4ba0356c618890e06 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
W20251025 07:52:51.460758 710 leader_election.cc:343] T 6fb1a1d38fb94f07b79ac4f8334fa719 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:51.460852 710 leader_election.cc:304] T 6fb1a1d38fb94f07b79ac4f8334fa719 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:51.461222 710 leader_election.cc:304] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:51.461549 3236 raft_consensus.cc:2804] T f7011f1dcad74fe4ba0356c618890e06 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:51.461654 3236 raft_consensus.cc:697] T f7011f1dcad74fe4ba0356c618890e06 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:51.461653 1039 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "830fdbc6812e416690d0db42a73af503" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:51.461884 3236 consensus_queue.cc:237] T f7011f1dcad74fe4ba0356c618890e06 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:51.462164 712 leader_election.cc:343] T 830fdbc6812e416690d0db42a73af503 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:51.462225 3137 raft_consensus.cc:2749] T 6fb1a1d38fb94f07b79ac4f8334fa719 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:51.462323 3142 raft_consensus.cc:2804] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:51.462414 3142 raft_consensus.cc:697] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:51.462631 3142 consensus_queue.cc:237] T 5fc092c2bb2d41ac8c5b4fc07460c8ec P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.463732 765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c4789b9d89f42cb844e6cb595e687db" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:51.463969 765 raft_consensus.cc:3060] T 2c4789b9d89f42cb844e6cb595e687db P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:51.464134 3235 raft_consensus.cc:493] T 60c60a9ee37f43978910bd067a098bb7 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:51.464272 3235 raft_consensus.cc:515] T 60c60a9ee37f43978910bd067a098bb7 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.464836 3235 leader_election.cc:290] T 60c60a9ee37f43978910bd067a098bb7 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:51.465405 1039 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60c60a9ee37f43978910bd067a098bb7" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:51.465907 712 leader_election.cc:343] T 60c60a9ee37f43978910bd067a098bb7 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:51.467690 765 raft_consensus.cc:2468] T 2c4789b9d89f42cb844e6cb595e687db P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:51.470767 765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e18611cbc1348eca927162fec70e36f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:51.470973 765 raft_consensus.cc:3060] T 7e18611cbc1348eca927162fec70e36f P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:51.473693 765 raft_consensus.cc:2468] T 7e18611cbc1348eca927162fec70e36f P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:51.475150 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60c60a9ee37f43978910bd067a098bb7" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:51.476706 1166 tablet_bootstrap.cc:492] T 0540d6a10cf04fa89756623e9aaaeb68 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:51.476842 1166 ts_tablet_manager.cc:1403] T 0540d6a10cf04fa89756623e9aaaeb68 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.057s user 0.001s sys 0.005s
I20251025 07:52:51.477026 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "830fdbc6812e416690d0db42a73af503" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:51.479223 2988 consensus_queue.cc:1048] T a6ae1315b28940d49b667e3c80347021 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:51.480459 847 leader_election.cc:304] T 2c4789b9d89f42cb844e6cb595e687db P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:51.481338 2988 raft_consensus.cc:2804] T 2c4789b9d89f42cb844e6cb595e687db P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:51.481532 2988 raft_consensus.cc:697] T 2c4789b9d89f42cb844e6cb595e687db P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:51.481767 2988 consensus_queue.cc:237] T 2c4789b9d89f42cb844e6cb595e687db P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.490433 2990 consensus_queue.cc:1048] T 792f32cbd66449ca98813c669b2a261d P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:51.490605 452 catalog_manager.cc:5741] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417) does not have the latest schema for tablet c0c6324cbae34dcebade3ef9f89a7150 (table default.Table_e2adb17a63724d56b3ad83d8daa4c30c [id=7d3e8beaac8c40c09cd20b9a476f3049]). Expected version 1 got 0
I20251025 07:52:51.491109 1166 raft_consensus.cc:359] T 0540d6a10cf04fa89756623e9aaaeb68 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.491310 1166 raft_consensus.cc:385] T 0540d6a10cf04fa89756623e9aaaeb68 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:51.491379 1166 raft_consensus.cc:740] T 0540d6a10cf04fa89756623e9aaaeb68 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:51.491623 1166 consensus_queue.cc:260] T 0540d6a10cf04fa89756623e9aaaeb68 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.492781 2990 consensus_queue.cc:1048] T 48b73e0c0dce4be686d05f991a21bd19 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:51.481988 1167 tablet_bootstrap.cc:492] T 698e9f8de7db4516a811c9eb9143a268 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:51.493583 847 leader_election.cc:304] T 7e18611cbc1348eca927162fec70e36f P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:51.493754 1167 ts_tablet_manager.cc:1403] T 698e9f8de7db4516a811c9eb9143a268 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.063s user 0.002s sys 0.002s
I20251025 07:52:51.493839 2990 raft_consensus.cc:2804] T 7e18611cbc1348eca927162fec70e36f P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:51.493954 2990 raft_consensus.cc:697] T 7e18611cbc1348eca927162fec70e36f P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:51.494200 2990 consensus_queue.cc:237] T 7e18611cbc1348eca927162fec70e36f P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.494632 1167 raft_consensus.cc:359] T 698e9f8de7db4516a811c9eb9143a268 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.494807 1167 raft_consensus.cc:385] T 698e9f8de7db4516a811c9eb9143a268 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:51.494870 1167 raft_consensus.cc:740] T 698e9f8de7db4516a811c9eb9143a268 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
W20251025 07:52:51.500380 710 leader_election.cc:343] T 60c60a9ee37f43978910bd067a098bb7 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:51.500514 710 leader_election.cc:304] T 60c60a9ee37f43978910bd067a098bb7 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:51.502653 3325 raft_consensus.cc:2749] T 60c60a9ee37f43978910bd067a098bb7 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:51.503355 1166 ts_tablet_manager.cc:1434] T 0540d6a10cf04fa89756623e9aaaeb68 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.023s user 0.000s sys 0.001s
I20251025 07:52:51.503636 1166 tablet_bootstrap.cc:492] T 60c60a9ee37f43978910bd067a098bb7 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
W20251025 07:52:51.504649 710 leader_election.cc:343] T 830fdbc6812e416690d0db42a73af503 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:51.504777 710 leader_election.cc:304] T 830fdbc6812e416690d0db42a73af503 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b; no voters: 3aea2869ff814cc7b142012faa274382, db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:51.505246 3236 raft_consensus.cc:2749] T 830fdbc6812e416690d0db42a73af503 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:51.508533 1039 raft_consensus.cc:1275] T 6d3c9954693d4e8782ee0128ef066aa9 P db42be7ad3c741a0aa427f27e245993b [term 2 FOLLOWER]: Refusing update from remote peer 23eeffd3de254ce18db5febfde971c6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251025 07:52:51.509078 3137 consensus_queue.cc:1048] T 6d3c9954693d4e8782ee0128ef066aa9 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:51.512650 1167 consensus_queue.cc:260] T 698e9f8de7db4516a811c9eb9143a268 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.513345 1167 ts_tablet_manager.cc:1434] T 698e9f8de7db4516a811c9eb9143a268 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20251025 07:52:51.531426 1166 tablet_bootstrap.cc:654] T 60c60a9ee37f43978910bd067a098bb7 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:51.531490 1167 tablet_bootstrap.cc:492] T ffe7c8e1d6c44872b7b65e512764a56a P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:51.533082 1167 tablet_bootstrap.cc:654] T ffe7c8e1d6c44872b7b65e512764a56a P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:51.564831 1166 tablet_bootstrap.cc:492] T 60c60a9ee37f43978910bd067a098bb7 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:51.565023 1166 ts_tablet_manager.cc:1403] T 60c60a9ee37f43978910bd067a098bb7 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.061s user 0.001s sys 0.003s
I20251025 07:52:51.565543 2988 consensus_queue.cc:1048] T df9db9dd31d840e0b1394db04d662a62 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:51.547530 3137 raft_consensus.cc:493] T 698e9f8de7db4516a811c9eb9143a268 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:51.566056 1166 raft_consensus.cc:359] T 60c60a9ee37f43978910bd067a098bb7 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.566217 1166 raft_consensus.cc:385] T 60c60a9ee37f43978910bd067a098bb7 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:51.566265 1166 raft_consensus.cc:740] T 60c60a9ee37f43978910bd067a098bb7 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:51.566237 3137 raft_consensus.cc:515] T 698e9f8de7db4516a811c9eb9143a268 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.566418 1166 consensus_queue.cc:260] T 60c60a9ee37f43978910bd067a098bb7 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.566975 3137 leader_election.cc:290] T 698e9f8de7db4516a811c9eb9143a268 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:51.567065 1166 ts_tablet_manager.cc:1434] T 60c60a9ee37f43978910bd067a098bb7 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:51.567525 1169 tablet_bootstrap.cc:492] T 792f32cbd66449ca98813c669b2a261d P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:51.567714 1169 ts_tablet_manager.cc:1403] T 792f32cbd66449ca98813c669b2a261d P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.264s user 0.015s sys 0.008s
I20251025 07:52:51.567888 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "698e9f8de7db4516a811c9eb9143a268" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:51.568471 712 leader_election.cc:343] T 698e9f8de7db4516a811c9eb9143a268 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:51.568476 1169 raft_consensus.cc:359] T 792f32cbd66449ca98813c669b2a261d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.568619 1169 raft_consensus.cc:385] T 792f32cbd66449ca98813c669b2a261d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:51.568711 1169 raft_consensus.cc:740] T 792f32cbd66449ca98813c669b2a261d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:51.568991 1169 consensus_queue.cc:260] T 792f32cbd66449ca98813c669b2a261d P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.570501 1169 ts_tablet_manager.cc:1434] T 792f32cbd66449ca98813c669b2a261d P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251025 07:52:51.571843 1169 tablet_bootstrap.cc:492] T df0d86ad301245c1a6bcb44805936d25 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:51.572361 1166 tablet_bootstrap.cc:492] T de8da6abeda74a2a96d94ce5ac8fb7e5 P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:51.574369 1166 tablet_bootstrap.cc:654] T de8da6abeda74a2a96d94ce5ac8fb7e5 P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:51.578627 3143 raft_consensus.cc:493] T 792f32cbd66449ca98813c669b2a261d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:51.578740 3143 raft_consensus.cc:515] T 792f32cbd66449ca98813c669b2a261d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.579250 3143 leader_election.cc:290] T 792f32cbd66449ca98813c669b2a261d P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:51.579824 764 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "792f32cbd66449ca98813c669b2a261d" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:51.582216 3307 consensus_queue.cc:1048] T 22f7db660616457b83b24e947a0b0bae P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:51.581166 905 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "792f32cbd66449ca98813c669b2a261d" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:51.585628 983 leader_election.cc:304] T 792f32cbd66449ca98813c669b2a261d P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:51.585863 3143 raft_consensus.cc:2749] T 792f32cbd66449ca98813c669b2a261d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:51.582451 2146 ts_tablet_manager.cc:1929] T 3b940fa5ae26427dafbcfd26372754b3 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:51.586182 2146 log.cc:1199] T 3b940fa5ae26427dafbcfd26372754b3 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/3b940fa5ae26427dafbcfd26372754b3
I20251025 07:52:51.588548 2146 ts_tablet_manager.cc:1950] T 3b940fa5ae26427dafbcfd26372754b3 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:51.589166 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 357) took 1788 ms. Trace:
I20251025 07:52:51.590796 2166 ts_tablet_manager.cc:1929] T 4a1ec8c2ed7f41abb588f592f62097d1 P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:51.591229 1169 tablet_bootstrap.cc:654] T df0d86ad301245c1a6bcb44805936d25 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:51.594803 905 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "698e9f8de7db4516a811c9eb9143a268" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:51.597061 905 raft_consensus.cc:2468] T 698e9f8de7db4516a811c9eb9143a268 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 0.
I20251025 07:52:51.597296 2146 rpcz_store.cc:276] 1025 07:52:49.800356 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.800402 (+ 46us) service_pool.cc:225] Handling call
1025 07:52:51.427939 (+1627537us) ts_tablet_manager.cc:1151] Deleting tablet 3b940fa5ae26427dafbcfd26372754b3
1025 07:52:51.427949 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.452977 (+ 25028us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.582324 (+129347us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.589151 (+ 6827us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1627403}
I20251025 07:52:51.598728 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 358) took 1796 ms. Trace:
I20251025 07:52:51.598819 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 3b940fa5ae26427dafbcfd26372754b3 (table default.Table_d57e7ea8e59641f781c564e5b01479bc [id=fbaf826daa7c448eb92c7f93826260c8]) successfully deleted
I20251025 07:52:51.601024 2146 rpcz_store.cc:276] 1025 07:52:49.802127 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.802192 (+ 65us) service_pool.cc:225] Handling call
1025 07:52:51.598630 (+1796438us) ts_tablet_manager.cc:1151] Deleting tablet 0f5f6ca6c3cd4d369690e425387a4344
1025 07:52:51.598641 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.598712 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1796306}
W20251025 07:52:51.601934 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 0f5f6ca6c3cd4d369690e425387a4344 because the tablet was not found. No further retry: Not found: Tablet not found: 0f5f6ca6c3cd4d369690e425387a4344
I20251025 07:52:51.595484 1166 tablet_bootstrap.cc:492] T de8da6abeda74a2a96d94ce5ac8fb7e5 P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:51.602340 1166 ts_tablet_manager.cc:1403] T de8da6abeda74a2a96d94ce5ac8fb7e5 P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20251025 07:52:51.603189 1166 raft_consensus.cc:359] T de8da6abeda74a2a96d94ce5ac8fb7e5 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.604149 2166 log.cc:1199] T 4a1ec8c2ed7f41abb588f592f62097d1 P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/4a1ec8c2ed7f41abb588f592f62097d1
I20251025 07:52:51.604785 2166 ts_tablet_manager.cc:1950] T 4a1ec8c2ed7f41abb588f592f62097d1 P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:51.607882 1166 raft_consensus.cc:385] T de8da6abeda74a2a96d94ce5ac8fb7e5 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:51.608031 710 leader_election.cc:304] T 698e9f8de7db4516a811c9eb9143a268 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:51.608458 1166 raft_consensus.cc:740] T de8da6abeda74a2a96d94ce5ac8fb7e5 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:51.603966 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 359) took 1801 ms. Trace:
I20251025 07:52:51.608644 2146 rpcz_store.cc:276] 1025 07:52:49.802865 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.802915 (+ 50us) service_pool.cc:225] Handling call
1025 07:52:51.603873 (+1800958us) ts_tablet_manager.cc:1151] Deleting tablet 4c155aa7fa5b4d68bbab1918b8432372
1025 07:52:51.603884 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.603950 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1800833}
W20251025 07:52:51.609090 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 4c155aa7fa5b4d68bbab1918b8432372 because the tablet was not found. No further retry: Not found: Tablet not found: 4c155aa7fa5b4d68bbab1918b8432372
I20251025 07:52:51.609457 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 361) took 1804 ms. Trace:
I20251025 07:52:51.609539 2146 rpcz_store.cc:276] 1025 07:52:49.805290 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.805334 (+ 44us) service_pool.cc:225] Handling call
1025 07:52:51.609355 (+1804021us) ts_tablet_manager.cc:1151] Deleting tablet d9bfd7c0c3ad4e5b98b6708793d61d41
1025 07:52:51.609368 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.609440 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1803883}
W20251025 07:52:51.609965 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet d9bfd7c0c3ad4e5b98b6708793d61d41 because the tablet was not found. No further retry: Not found: Tablet not found: d9bfd7c0c3ad4e5b98b6708793d61d41
I20251025 07:52:51.610630 3236 raft_consensus.cc:2804] T 698e9f8de7db4516a811c9eb9143a268 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:51.610599 1166 consensus_queue.cc:260] T de8da6abeda74a2a96d94ce5ac8fb7e5 P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.611125 1166 ts_tablet_manager.cc:1434] T de8da6abeda74a2a96d94ce5ac8fb7e5 P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20251025 07:52:51.611394 1166 tablet_bootstrap.cc:492] T 421db850a64048218eecb9de65e2bdbf P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:51.611565 3236 raft_consensus.cc:493] T 698e9f8de7db4516a811c9eb9143a268 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:51.611903 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 362) took 1805 ms. Trace:
I20251025 07:52:51.612010 2146 rpcz_store.cc:276] 1025 07:52:49.806312 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.806358 (+ 46us) service_pool.cc:225] Handling call
1025 07:52:51.611829 (+1805471us) ts_tablet_manager.cc:1151] Deleting tablet aaef24ccdfba45fe8204327723df834e
1025 07:52:51.611838 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.611890 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1805342}
I20251025 07:52:51.612109 3236 raft_consensus.cc:3060] T 698e9f8de7db4516a811c9eb9143a268 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:51.612376 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet aaef24ccdfba45fe8204327723df834e because the tablet was not found. No further retry: Not found: Tablet not found: aaef24ccdfba45fe8204327723df834e
I20251025 07:52:51.613507 2146 tablet_replica.cc:333] T 60c60a9ee37f43978910bd067a098bb7 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:51.613682 2146 raft_consensus.cc:2243] T 60c60a9ee37f43978910bd067a098bb7 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:51.614836 2146 raft_consensus.cc:2272] T 60c60a9ee37f43978910bd067a098bb7 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:51.619024 1166 tablet_bootstrap.cc:654] T 421db850a64048218eecb9de65e2bdbf P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
W20251025 07:52:51.630151 847 consensus_peers.cc:597] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382 -> Peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Couldn't send request to peer 23eeffd3de254ce18db5febfde971c6b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ee1c8ea694284d80a30f9c05be4cbf49. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:51.633569 3122 raft_consensus.cc:493] T de8da6abeda74a2a96d94ce5ac8fb7e5 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:51.633618 3236 raft_consensus.cc:515] T 698e9f8de7db4516a811c9eb9143a268 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.633708 3122 raft_consensus.cc:515] T de8da6abeda74a2a96d94ce5ac8fb7e5 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.634287 3122 leader_election.cc:290] T de8da6abeda74a2a96d94ce5ac8fb7e5 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417)
I20251025 07:52:51.634696 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 307) took 3739 ms. Trace:
I20251025 07:52:51.634846 3143 raft_consensus.cc:493] T 72618756f756484e895e885ef12545c8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:51.634799 2166 rpcz_store.cc:276] 1025 07:52:47.895186 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:47.895253 (+ 67us) service_pool.cc:225] Handling call
1025 07:52:51.382239 (+3486986us) ts_tablet_manager.cc:1151] Deleting tablet 4a1ec8c2ed7f41abb588f592f62097d1
1025 07:52:51.382249 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.508889 (+126640us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.590687 (+ 81798us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.634671 (+ 43984us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3486755}
I20251025 07:52:51.634970 3143 raft_consensus.cc:515] T 72618756f756484e895e885ef12545c8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.635039 1039 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "698e9f8de7db4516a811c9eb9143a268" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:51.635346 439 catalog_manager.cc:4985] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): tablet 4a1ec8c2ed7f41abb588f592f62097d1 (table default.Table_91898f2a551144d18df0309c827855bf [id=aaa734f6549741768b0156bdf5cc25af]) successfully deleted
I20251025 07:52:51.635540 3143 leader_election.cc:290] T 72618756f756484e895e885ef12545c8 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:51.635864 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 315) took 1851 ms. Trace:
I20251025 07:52:51.635948 905 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72618756f756484e895e885ef12545c8" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:51.635942 2166 rpcz_store.cc:276] 1025 07:52:49.783922 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.783977 (+ 55us) service_pool.cc:225] Handling call
1025 07:52:51.635757 (+1851780us) ts_tablet_manager.cc:1151] Deleting tablet ce1ebb1ec14b4801930a72765bca1a7d
1025 07:52:51.635770 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.635842 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1851551}
I20251025 07:52:51.636067 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72618756f756484e895e885ef12545c8" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:51.636227 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 310) took 1852 ms. Trace:
I20251025 07:52:51.636291 2166 rpcz_store.cc:276] 1025 07:52:49.783416 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.784334 (+ 918us) service_pool.cc:225] Handling call
1025 07:52:51.636144 (+1851810us) ts_tablet_manager.cc:1151] Deleting tablet 5b6383c99e9e4c90a374da9ccd830a6f
1025 07:52:51.636152 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.636215 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1851672}
W20251025 07:52:51.636452 712 leader_election.cc:343] T 698e9f8de7db4516a811c9eb9143a268 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:51.636555 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 311) took 1853 ms. Trace:
I20251025 07:52:51.636639 2166 rpcz_store.cc:276] 1025 07:52:49.783553 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.784504 (+ 951us) service_pool.cc:225] Handling call
1025 07:52:51.636478 (+1851974us) ts_tablet_manager.cc:1151] Deleting tablet 80fe3255ed614270bd8352a2853fbc43
1025 07:52:51.636486 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.636538 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1851851}
W20251025 07:52:51.636894 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet ce1ebb1ec14b4801930a72765bca1a7d because the tablet was not found. No further retry: Not found: Tablet not found: ce1ebb1ec14b4801930a72765bca1a7d
I20251025 07:52:51.636951 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 312) took 1853 ms. Trace:
I20251025 07:52:51.637022 2166 rpcz_store.cc:276] 1025 07:52:49.783666 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.784658 (+ 992us) service_pool.cc:225] Handling call
1025 07:52:51.636877 (+1852219us) ts_tablet_manager.cc:1151] Deleting tablet ebc824ec4c60411dab1a2ea4a385148e
1025 07:52:51.636886 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.636937 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1852088}
W20251025 07:52:51.637142 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 5b6383c99e9e4c90a374da9ccd830a6f because the tablet was not found. No further retry: Not found: Tablet not found: 5b6383c99e9e4c90a374da9ccd830a6f
I20251025 07:52:51.637315 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 313) took 1853 ms. Trace:
W20251025 07:52:51.637590 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 80fe3255ed614270bd8352a2853fbc43 because the tablet was not found. No further retry: Not found: Tablet not found: 80fe3255ed614270bd8352a2853fbc43
I20251025 07:52:51.637629 2166 rpcz_store.cc:276] 1025 07:52:49.783753 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.784806 (+ 1053us) service_pool.cc:225] Handling call
1025 07:52:51.637245 (+1852439us) ts_tablet_manager.cc:1151] Deleting tablet 91728987303b41fd840eb56f3a311c5f
1025 07:52:51.637255 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.637300 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1852333}
W20251025 07:52:51.637825 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet ebc824ec4c60411dab1a2ea4a385148e because the tablet was not found. No further retry: Not found: Tablet not found: ebc824ec4c60411dab1a2ea4a385148e
W20251025 07:52:51.638023 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 91728987303b41fd840eb56f3a311c5f because the tablet was not found. No further retry: Not found: Tablet not found: 91728987303b41fd840eb56f3a311c5f
I20251025 07:52:51.638083 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 314) took 1854 ms. Trace:
I20251025 07:52:51.638165 2166 rpcz_store.cc:276] 1025 07:52:49.783838 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.784929 (+ 1091us) service_pool.cc:225] Handling call
1025 07:52:51.638010 (+1853081us) ts_tablet_manager.cc:1151] Deleting tablet a77eb82f26ae43b48de1e85693372d34
1025 07:52:51.638021 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.638069 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1852966}
I20251025 07:52:51.638348 983 leader_election.cc:304] T 72618756f756484e895e885ef12545c8 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:51.638494 2146 ts_tablet_manager.cc:1916] T 60c60a9ee37f43978910bd067a098bb7 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:51.638532 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet a77eb82f26ae43b48de1e85693372d34 because the tablet was not found. No further retry: Not found: Tablet not found: a77eb82f26ae43b48de1e85693372d34
I20251025 07:52:51.638793 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 316) took 1854 ms. Trace:
I20251025 07:52:51.638810 3143 raft_consensus.cc:2749] T 72618756f756484e895e885ef12545c8 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:51.638872 2166 rpcz_store.cc:276] 1025 07:52:49.784002 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.785069 (+ 1067us) service_pool.cc:225] Handling call
1025 07:52:51.638723 (+1853654us) ts_tablet_manager.cc:1151] Deleting tablet 0f5f6ca6c3cd4d369690e425387a4344
1025 07:52:51.638732 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.638779 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1853507}
W20251025 07:52:51.639225 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 0f5f6ca6c3cd4d369690e425387a4344 because the tablet was not found. No further retry: Not found: Tablet not found: 0f5f6ca6c3cd4d369690e425387a4344
I20251025 07:52:51.639807 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 317) took 1855 ms. Trace:
I20251025 07:52:51.642275 3347 tablet.cc:1722] T 5dab439bc50d4e869548154e1d222ad6 P 3aea2869ff814cc7b142012faa274382: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251025 07:52:51.658352 3353 tablet.cc:1722] T 288e543bb279410c91c01dd1623e110f P 3aea2869ff814cc7b142012faa274382: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 07:52:51.661695 905 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "698e9f8de7db4516a811c9eb9143a268" candidate_uuid: "23eeffd3de254ce18db5febfde971c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382"
I20251025 07:52:51.661906 905 raft_consensus.cc:3060] T 698e9f8de7db4516a811c9eb9143a268 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:51.643965 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de8da6abeda74a2a96d94ce5ac8fb7e5" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:51.657908 3351 tablet.cc:1722] T 5dab439bc50d4e869548154e1d222ad6 P 23eeffd3de254ce18db5febfde971c6b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20251025 07:52:51.664722 847 leader_election.cc:343] T de8da6abeda74a2a96d94ce5ac8fb7e5 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:51.664752 905 raft_consensus.cc:2468] T 698e9f8de7db4516a811c9eb9143a268 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eeffd3de254ce18db5febfde971c6b in term 1.
I20251025 07:52:51.665282 710 leader_election.cc:304] T 698e9f8de7db4516a811c9eb9143a268 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters: db42be7ad3c741a0aa427f27e245993b
I20251025 07:52:51.665539 3142 raft_consensus.cc:2804] T 698e9f8de7db4516a811c9eb9143a268 P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:51.665652 3142 raft_consensus.cc:697] T 698e9f8de7db4516a811c9eb9143a268 P 23eeffd3de254ce18db5febfde971c6b [term 1 LEADER]: Becoming Leader. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Running, Role: LEADER
I20251025 07:52:51.665866 3142 consensus_queue.cc:237] T 698e9f8de7db4516a811c9eb9143a268 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.667471 2166 rpcz_store.cc:276] 1025 07:52:49.784080 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.785246 (+ 1166us) service_pool.cc:225] Handling call
1025 07:52:51.639738 (+1854492us) ts_tablet_manager.cc:1151] Deleting tablet 36c8fcafd5464b4e999276a8062a812a
1025 07:52:51.639746 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.639793 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1854337}
I20251025 07:52:51.667815 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 319) took 1883 ms. Trace:
I20251025 07:52:51.667837 2146 ts_tablet_manager.cc:1929] T 60c60a9ee37f43978910bd067a098bb7 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:51.667922 2146 log.cc:1199] T 60c60a9ee37f43978910bd067a098bb7 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/60c60a9ee37f43978910bd067a098bb7
I20251025 07:52:51.667884 2166 rpcz_store.cc:276] 1025 07:52:49.784288 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.785436 (+ 1148us) service_pool.cc:225] Handling call
1025 07:52:51.667723 (+1882287us) ts_tablet_manager.cc:1151] Deleting tablet aaef24ccdfba45fe8204327723df834e
1025 07:52:51.667732 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.667800 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1882165}
W20251025 07:52:51.668007 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 36c8fcafd5464b4e999276a8062a812a because the tablet was not found. No further retry: Not found: Tablet not found: 36c8fcafd5464b4e999276a8062a812a
I20251025 07:52:51.668243 2146 ts_tablet_manager.cc:1950] T 60c60a9ee37f43978910bd067a098bb7 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
W20251025 07:52:51.668289 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet aaef24ccdfba45fe8204327723df834e because the tablet was not found. No further retry: Not found: Tablet not found: aaef24ccdfba45fe8204327723df834e
I20251025 07:52:51.668334 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 320) took 1876 ms. Trace:
I20251025 07:52:51.668401 2166 rpcz_store.cc:276] 1025 07:52:49.791460 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.791707 (+ 247us) service_pool.cc:225] Handling call
1025 07:52:51.668251 (+1876544us) ts_tablet_manager.cc:1151] Deleting tablet 050a5d3b5f934dd19bf2e221f2fe0780
1025 07:52:51.668261 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.668318 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1876389}
W20251025 07:52:51.668781 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 050a5d3b5f934dd19bf2e221f2fe0780 because the tablet was not found. No further retry: Not found: Tablet not found: 050a5d3b5f934dd19bf2e221f2fe0780
I20251025 07:52:51.668965 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 363) took 1861 ms. Trace:
I20251025 07:52:51.669076 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 322) took 1877 ms. Trace:
I20251025 07:52:51.669036 2146 rpcz_store.cc:276] 1025 07:52:49.807114 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.807159 (+ 45us) service_pool.cc:225] Handling call
1025 07:52:51.613466 (+1806307us) ts_tablet_manager.cc:1151] Deleting tablet 60c60a9ee37f43978910bd067a098bb7
1025 07:52:51.613474 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.641724 (+ 28250us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.667758 (+ 26034us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.668950 (+ 1192us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1806176}
I20251025 07:52:51.669133 2166 rpcz_store.cc:276] 1025 07:52:49.791659 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.791893 (+ 234us) service_pool.cc:225] Handling call
1025 07:52:51.669004 (+1877111us) ts_tablet_manager.cc:1151] Deleting tablet 062c0f3e51a3487880cea373f8cf1f54
1025 07:52:51.669012 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.669063 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1876990}
I20251025 07:52:51.669371 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 364) took 1861 ms. Trace:
I20251025 07:52:51.669440 2146 rpcz_store.cc:276] 1025 07:52:49.808005 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.808048 (+ 43us) service_pool.cc:225] Handling call
1025 07:52:51.669284 (+1861236us) ts_tablet_manager.cc:1151] Deleting tablet f97acc9a638b4d5a968dcc3412a384bc
1025 07:52:51.669294 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.669356 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1861110}
I20251025 07:52:51.669495 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 60c60a9ee37f43978910bd067a098bb7 (table default.Table_173708dfd0274b34a7db6fa2b239a8b0 [id=6bc5ca21f8c54249a5150f0fea2686c7]) successfully deleted
I20251025 07:52:51.669664 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 321) took 1878 ms. Trace:
I20251025 07:52:51.669724 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 365) took 1860 ms. Trace:
I20251025 07:52:51.669728 2166 rpcz_store.cc:276] 1025 07:52:49.791557 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.792040 (+ 483us) service_pool.cc:225] Handling call
1025 07:52:51.669591 (+1877551us) ts_tablet_manager.cc:1151] Deleting tablet 04449cc97f404a19b6a1b20dea074f95
1025 07:52:51.669600 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.669650 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1877439}
I20251025 07:52:51.669790 2146 rpcz_store.cc:276] 1025 07:52:49.808805 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.808848 (+ 43us) service_pool.cc:225] Handling call
1025 07:52:51.669656 (+1860808us) ts_tablet_manager.cc:1151] Deleting tablet b3a0af893a8947848e12093d0a7f0229
1025 07:52:51.669664 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.669711 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1860672}
W20251025 07:52:51.669907 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet f97acc9a638b4d5a968dcc3412a384bc because the tablet was not found. No further retry: Not found: Tablet not found: f97acc9a638b4d5a968dcc3412a384bc
I20251025 07:52:51.670037 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 366) took 1860 ms. Trace:
I20251025 07:52:51.670090 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 323) took 1875 ms. Trace:
W20251025 07:52:51.669515 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 062c0f3e51a3487880cea373f8cf1f54 because the tablet was not found. No further retry: Not found: Tablet not found: 062c0f3e51a3487880cea373f8cf1f54
I20251025 07:52:51.670109 2146 rpcz_store.cc:276] 1025 07:52:49.809893 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.810120 (+ 227us) service_pool.cc:225] Handling call
1025 07:52:51.669966 (+1859846us) ts_tablet_manager.cc:1151] Deleting tablet f661d5909657455581dcb5e6eadd3c0b
1025 07:52:51.669975 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.670025 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1859686}
I20251025 07:52:51.670174 2166 rpcz_store.cc:276] 1025 07:52:49.794921 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.794982 (+ 61us) service_pool.cc:225] Handling call
1025 07:52:51.670029 (+1875047us) ts_tablet_manager.cc:1151] Deleting tablet f43ef51ee45d47a99ec732b49d50d3b4
1025 07:52:51.670036 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.670078 (+ 42us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1874861}
I20251025 07:52:51.670393 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 367) took 1859 ms. Trace:
W20251025 07:52:51.670408 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 04449cc97f404a19b6a1b20dea074f95 because the tablet was not found. No further retry: Not found: Tablet not found: 04449cc97f404a19b6a1b20dea074f95
I20251025 07:52:51.670532 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 324) took 1874 ms. Trace:
I20251025 07:52:51.670476 2146 rpcz_store.cc:276] 1025 07:52:49.811103 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.811146 (+ 43us) service_pool.cc:225] Handling call
1025 07:52:51.670322 (+1859176us) ts_tablet_manager.cc:1151] Deleting tablet 88eb51447e0f4080baa49dd902deb24f
1025 07:52:51.670330 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.670380 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1859051}
I20251025 07:52:51.670593 2166 rpcz_store.cc:276] 1025 07:52:49.796164 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.796221 (+ 57us) service_pool.cc:225] Handling call
1025 07:52:51.670467 (+1874246us) ts_tablet_manager.cc:1151] Deleting tablet 3b940fa5ae26427dafbcfd26372754b3
1025 07:52:51.670475 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.670520 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1874046}
W20251025 07:52:51.670637 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet f43ef51ee45d47a99ec732b49d50d3b4 because the tablet was not found. No further retry: Not found: Tablet not found: f43ef51ee45d47a99ec732b49d50d3b4
I20251025 07:52:51.670691 2146 tablet_replica.cc:333] T 062c0f3e51a3487880cea373f8cf1f54 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:51.670869 2146 raft_consensus.cc:2243] T 062c0f3e51a3487880cea373f8cf1f54 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:51.670938 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 325) took 1871 ms. Trace:
I20251025 07:52:51.671005 2146 raft_consensus.cc:2272] T 062c0f3e51a3487880cea373f8cf1f54 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:51.671001 2166 rpcz_store.cc:276] 1025 07:52:49.799192 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.799242 (+ 50us) service_pool.cc:225] Handling call
1025 07:52:51.670868 (+1871626us) ts_tablet_manager.cc:1151] Deleting tablet f97acc9a638b4d5a968dcc3412a384bc
1025 07:52:51.670875 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.670926 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1871468}
W20251025 07:52:51.671057 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet b3a0af893a8947848e12093d0a7f0229 because the tablet was not found. No further retry: Not found: Tablet not found: b3a0af893a8947848e12093d0a7f0229
W20251025 07:52:51.671265 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet f661d5909657455581dcb5e6eadd3c0b because the tablet was not found. No further retry: Not found: Tablet not found: f661d5909657455581dcb5e6eadd3c0b
I20251025 07:52:51.671363 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 326) took 1871 ms. Trace:
I20251025 07:52:51.671487 2146 ts_tablet_manager.cc:1916] T 062c0f3e51a3487880cea373f8cf1f54 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:51.671432 2166 rpcz_store.cc:276] 1025 07:52:49.799921 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.799978 (+ 57us) service_pool.cc:225] Handling call
1025 07:52:51.671298 (+1871320us) ts_tablet_manager.cc:1151] Deleting tablet 88eb51447e0f4080baa49dd902deb24f
1025 07:52:51.671306 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.671351 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1871188}
W20251025 07:52:51.671489 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 88eb51447e0f4080baa49dd902deb24f because the tablet was not found. No further retry: Not found: Tablet not found: 88eb51447e0f4080baa49dd902deb24f
W20251025 07:52:51.671716 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 3b940fa5ae26427dafbcfd26372754b3 because the tablet was not found. No further retry: Not found: Tablet not found: 3b940fa5ae26427dafbcfd26372754b3
I20251025 07:52:51.671761 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 329) took 1861 ms. Trace:
I20251025 07:52:51.671828 2166 rpcz_store.cc:276] 1025 07:52:49.810666 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.810823 (+ 157us) service_pool.cc:225] Handling call
1025 07:52:51.671694 (+1860871us) ts_tablet_manager.cc:1151] Deleting tablet 48ff4ce7846b4e56a37eee9bcb17bd5e
1025 07:52:51.671702 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.671747 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1860715}
W20251025 07:52:51.671916 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet f97acc9a638b4d5a968dcc3412a384bc because the tablet was not found. No further retry: Not found: Tablet not found: f97acc9a638b4d5a968dcc3412a384bc
W20251025 07:52:51.672106 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 88eb51447e0f4080baa49dd902deb24f because the tablet was not found. No further retry: Not found: Tablet not found: 88eb51447e0f4080baa49dd902deb24f
W20251025 07:52:51.672298 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 48ff4ce7846b4e56a37eee9bcb17bd5e because the tablet was not found. No further retry: Not found: Tablet not found: 48ff4ce7846b4e56a37eee9bcb17bd5e
I20251025 07:52:51.672541 2990 raft_consensus.cc:493] T 0540d6a10cf04fa89756623e9aaaeb68 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:51.672696 2990 raft_consensus.cc:515] T 0540d6a10cf04fa89756623e9aaaeb68 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.673528 2990 leader_election.cc:290] T 0540d6a10cf04fa89756623e9aaaeb68 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:51.675088 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0540d6a10cf04fa89756623e9aaaeb68" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
W20251025 07:52:51.675581 847 leader_election.cc:343] T 0540d6a10cf04fa89756623e9aaaeb68 P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): Illegal state: must be running to vote when last-logged opid is not known
I20251025 07:52:51.659997 3354 tablet.cc:1722] T 288e543bb279410c91c01dd1623e110f P 23eeffd3de254ce18db5febfde971c6b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251025 07:52:51.661602 3236 leader_election.cc:290] T 698e9f8de7db4516a811c9eb9143a268 P 23eeffd3de254ce18db5febfde971c6b [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:51.684487 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 328) took 1874 ms. Trace:
I20251025 07:52:51.684441 3236 consensus_queue.cc:1048] T d40519ed9ee746348b17be2786d60e95 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:51.685096 2166 rpcz_store.cc:276] 1025 07:52:49.810535 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.811099 (+ 564us) service_pool.cc:225] Handling call
1025 07:52:51.684384 (+1873285us) ts_tablet_manager.cc:1151] Deleting tablet 26c849d376d143c6bef0c8e0e1d7d03a
1025 07:52:51.684408 (+ 24us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.684473 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1873129}
I20251025 07:52:51.686688 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 330) took 1841 ms. Trace:
W20251025 07:52:51.686754 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet 26c849d376d143c6bef0c8e0e1d7d03a because the tablet was not found. No further retry: Not found: Tablet not found: 26c849d376d143c6bef0c8e0e1d7d03a
I20251025 07:52:51.687357 2166 rpcz_store.cc:276] 1025 07:52:49.845013 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.845074 (+ 61us) service_pool.cc:225] Handling call
1025 07:52:51.686606 (+1841532us) ts_tablet_manager.cc:1151] Deleting tablet d9bfd7c0c3ad4e5b98b6708793d61d41
1025 07:52:51.686617 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.686675 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1841349}
I20251025 07:52:51.687644 2146 ts_tablet_manager.cc:1929] T 062c0f3e51a3487880cea373f8cf1f54 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:51.687752 2146 log.cc:1199] T 062c0f3e51a3487880cea373f8cf1f54 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/062c0f3e51a3487880cea373f8cf1f54
W20251025 07:52:51.687767 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet d9bfd7c0c3ad4e5b98b6708793d61d41 because the tablet was not found. No further retry: Not found: Tablet not found: d9bfd7c0c3ad4e5b98b6708793d61d41
I20251025 07:52:51.687812 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 331) took 1842 ms. Trace:
I20251025 07:52:51.687880 2166 rpcz_store.cc:276] 1025 07:52:49.845487 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.845530 (+ 43us) service_pool.cc:225] Handling call
1025 07:52:51.687729 (+1842199us) ts_tablet_manager.cc:1151] Deleting tablet a4e4b99be0434baaa75d988281797846
1025 07:52:51.687739 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.687797 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1842037}
I20251025 07:52:51.688035 2146 ts_tablet_manager.cc:1950] T 062c0f3e51a3487880cea373f8cf1f54 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
W20251025 07:52:51.688256 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet a4e4b99be0434baaa75d988281797846 because the tablet was not found. No further retry: Not found: Tablet not found: a4e4b99be0434baaa75d988281797846
I20251025 07:52:51.688730 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 368) took 1875 ms. Trace:
I20251025 07:52:51.688822 2146 rpcz_store.cc:276] 1025 07:52:49.812859 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.812908 (+ 49us) service_pool.cc:225] Handling call
1025 07:52:51.670653 (+1857745us) ts_tablet_manager.cc:1151] Deleting tablet 062c0f3e51a3487880cea373f8cf1f54
1025 07:52:51.670660 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.675927 (+ 5267us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.687551 (+ 11624us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.688710 (+ 1159us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1857548}
I20251025 07:52:51.689206 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 062c0f3e51a3487880cea373f8cf1f54 (table default.Table_6597694b643c4ef2b46ca84afc10f4b0 [id=0624650d8b8f40d2a6a4e1e7278968de]) successfully deleted
I20251025 07:52:51.689561 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 369) took 1875 ms. Trace:
I20251025 07:52:51.689646 2146 rpcz_store.cc:276] 1025 07:52:49.813780 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.813830 (+ 50us) service_pool.cc:225] Handling call
1025 07:52:51.689467 (+1875637us) ts_tablet_manager.cc:1151] Deleting tablet 26c849d376d143c6bef0c8e0e1d7d03a
1025 07:52:51.689476 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.689543 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1875436}
W20251025 07:52:51.690162 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 26c849d376d143c6bef0c8e0e1d7d03a because the tablet was not found. No further retry: Not found: Tablet not found: 26c849d376d143c6bef0c8e0e1d7d03a
I20251025 07:52:51.690400 1167 tablet_bootstrap.cc:492] T ffe7c8e1d6c44872b7b65e512764a56a P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:51.690538 1167 ts_tablet_manager.cc:1403] T ffe7c8e1d6c44872b7b65e512764a56a P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.177s user 0.001s sys 0.003s
I20251025 07:52:51.690526 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 370) took 1875 ms. Trace:
I20251025 07:52:51.690606 2146 rpcz_store.cc:276] 1025 07:52:49.814808 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.814854 (+ 46us) service_pool.cc:225] Handling call
1025 07:52:51.690421 (+1875567us) ts_tablet_manager.cc:1151] Deleting tablet f656945454fe4921ab819c772a089eb3
1025 07:52:51.690431 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.690507 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1875402}
W20251025 07:52:51.691031 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet f656945454fe4921ab819c772a089eb3 because the tablet was not found. No further retry: Not found: Tablet not found: f656945454fe4921ab819c772a089eb3
I20251025 07:52:51.691283 1167 raft_consensus.cc:359] T ffe7c8e1d6c44872b7b65e512764a56a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.691376 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 371) took 1872 ms. Trace:
I20251025 07:52:51.691449 1167 raft_consensus.cc:385] T ffe7c8e1d6c44872b7b65e512764a56a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:51.691443 2146 rpcz_store.cc:276] 1025 07:52:49.818854 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.818902 (+ 48us) service_pool.cc:225] Handling call
1025 07:52:51.691291 (+1872389us) ts_tablet_manager.cc:1151] Deleting tablet e54fe7cbcffa40c4b04144b47bee2b48
1025 07:52:51.691301 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.691360 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1872208}
I20251025 07:52:51.691515 1167 raft_consensus.cc:740] T ffe7c8e1d6c44872b7b65e512764a56a P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
W20251025 07:52:51.691783 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet e54fe7cbcffa40c4b04144b47bee2b48 because the tablet was not found. No further retry: Not found: Tablet not found: e54fe7cbcffa40c4b04144b47bee2b48
I20251025 07:52:51.691717 1167 consensus_queue.cc:260] T ffe7c8e1d6c44872b7b65e512764a56a P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.691846 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 372) took 1871 ms. Trace:
I20251025 07:52:51.691906 2146 rpcz_store.cc:276] 1025 07:52:49.819910 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.819954 (+ 44us) service_pool.cc:225] Handling call
1025 07:52:51.691779 (+1871825us) ts_tablet_manager.cc:1151] Deleting tablet 48ff4ce7846b4e56a37eee9bcb17bd5e
1025 07:52:51.691786 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.691834 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1871670}
W20251025 07:52:51.692235 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 48ff4ce7846b4e56a37eee9bcb17bd5e because the tablet was not found. No further retry: Not found: Tablet not found: 48ff4ce7846b4e56a37eee9bcb17bd5e
I20251025 07:52:51.692286 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 373) took 1869 ms. Trace:
I20251025 07:52:51.692346 2146 rpcz_store.cc:276] 1025 07:52:49.822875 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.822917 (+ 42us) service_pool.cc:225] Handling call
1025 07:52:51.692218 (+1869301us) ts_tablet_manager.cc:1151] Deleting tablet d77704b4fe6c4456a2ba40ffec0c60fc
1025 07:52:51.692226 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.692272 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1869168}
W20251025 07:52:51.692775 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet d77704b4fe6c4456a2ba40ffec0c60fc because the tablet was not found. No further retry: Not found: Tablet not found: d77704b4fe6c4456a2ba40ffec0c60fc
I20251025 07:52:51.693027 2146 tablet_replica.cc:333] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:51.693210 2146 raft_consensus.cc:2243] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:51.693477 2146 pending_rounds.cc:70] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382: Trying to abort 1 pending ops.
I20251025 07:52:51.693529 2146 pending_rounds.cc:77] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214607440984297472 op_type: NO_OP noop_request { }
I20251025 07:52:51.693614 2146 raft_consensus.cc:2889] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 07:52:51.693691 2146 raft_consensus.cc:2272] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:51.693722 1167 ts_tablet_manager.cc:1434] T ffe7c8e1d6c44872b7b65e512764a56a P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251025 07:52:51.694247 2146 ts_tablet_manager.cc:1916] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:51.695358 1167 tablet_bootstrap.cc:492] T 0063a0776b2247afaa8be82f6eff26f1 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:51.699659 3143 raft_consensus.cc:493] T d46673a283fe42b78ddeb99fbd2678eb P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:51.699832 3143 raft_consensus.cc:515] T d46673a283fe42b78ddeb99fbd2678eb P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.700553 3143 leader_election.cc:290] T d46673a283fe42b78ddeb99fbd2678eb P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:51.700899 2990 consensus_queue.cc:1048] T 3e79d69d5d004cc395b8e35569e25440 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:51.701642 2146 ts_tablet_manager.cc:1929] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:51.701777 2146 log.cc:1199] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/e4da89f863cc43f3bf6df908bb9dc6bf
I20251025 07:52:51.702288 2146 ts_tablet_manager.cc:1950] T e4da89f863cc43f3bf6df908bb9dc6bf P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:51.701637 905 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d46673a283fe42b78ddeb99fbd2678eb" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:51.703094 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 374) took 1876 ms. Trace:
I20251025 07:52:51.703202 2146 rpcz_store.cc:276] 1025 07:52:49.826695 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.826749 (+ 54us) service_pool.cc:225] Handling call
1025 07:52:51.692984 (+1866235us) ts_tablet_manager.cc:1151] Deleting tablet e4da89f863cc43f3bf6df908bb9dc6bf
1025 07:52:51.692992 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.697739 (+ 4747us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.701557 (+ 3818us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.703075 (+ 1518us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1866054}
I20251025 07:52:51.703632 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet e4da89f863cc43f3bf6df908bb9dc6bf (table default.Table_3d995c9c75174000b6cb98779cc0bb2f [id=6c1ffab0043545c2b7cce93a641eaa1d]) successfully deleted
I20251025 07:52:51.703967 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 375) took 1872 ms. Trace:
I20251025 07:52:51.704043 2146 rpcz_store.cc:276] 1025 07:52:49.831503 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.832823 (+ 1320us) service_pool.cc:225] Handling call
1025 07:52:51.703887 (+1871064us) ts_tablet_manager.cc:1151] Deleting tablet 4f7c15529be54a11961f385a953b0b03
1025 07:52:51.703896 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.703952 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1870900}
W20251025 07:52:51.704466 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 4f7c15529be54a11961f385a953b0b03 because the tablet was not found. No further retry: Not found: Tablet not found: 4f7c15529be54a11961f385a953b0b03
I20251025 07:52:51.704761 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 376) took 1871 ms. Trace:
I20251025 07:52:51.704829 2146 rpcz_store.cc:276] 1025 07:52:49.833521 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.833564 (+ 43us) service_pool.cc:225] Handling call
1025 07:52:51.704687 (+1871123us) ts_tablet_manager.cc:1151] Deleting tablet e47c4f741f354a19b49777bd06e54764
1025 07:52:51.704696 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.704746 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1870968}
W20251025 07:52:51.705173 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet e47c4f741f354a19b49777bd06e54764 because the tablet was not found. No further retry: Not found: Tablet not found: e47c4f741f354a19b49777bd06e54764
I20251025 07:52:51.705458 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 377) took 1871 ms. Trace:
I20251025 07:52:51.705533 2146 rpcz_store.cc:276] 1025 07:52:49.834203 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.834252 (+ 49us) service_pool.cc:225] Handling call
1025 07:52:51.705381 (+1871129us) ts_tablet_manager.cc:1151] Deleting tablet 95efd553493e4966bc48a01cac6e1e10
1025 07:52:51.705390 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.705435 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1870901}
W20251025 07:52:51.705901 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 95efd553493e4966bc48a01cac6e1e10 because the tablet was not found. No further retry: Not found: Tablet not found: 95efd553493e4966bc48a01cac6e1e10
I20251025 07:52:51.706192 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 382) took 1817 ms. Trace:
I20251025 07:52:51.706260 2146 rpcz_store.cc:276] 1025 07:52:49.888438 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.888487 (+ 49us) service_pool.cc:225] Handling call
1025 07:52:51.706103 (+1817616us) ts_tablet_manager.cc:1151] Deleting tablet c11f674385184ae6b235f1db8bf1fb91
1025 07:52:51.706111 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.706178 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1817458}
W20251025 07:52:51.706629 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet c11f674385184ae6b235f1db8bf1fb91 because the tablet was not found. No further retry: Not found: Tablet not found: c11f674385184ae6b235f1db8bf1fb91
I20251025 07:52:51.706857 2146 tablet_replica.cc:333] T 2e098c9c4613459295b6043048cba587 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:51.707039 2146 raft_consensus.cc:2243] T 2e098c9c4613459295b6043048cba587 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:51.707158 2146 raft_consensus.cc:2272] T 2e098c9c4613459295b6043048cba587 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:51.707616 2146 ts_tablet_manager.cc:1916] T 2e098c9c4613459295b6043048cba587 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:51.712540 1169 tablet_bootstrap.cc:492] T df0d86ad301245c1a6bcb44805936d25 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
W20251025 07:52:51.712532 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet aaef24ccdfba45fe8204327723df834e because the tablet was not found. No further retry: Not found: Tablet not found: aaef24ccdfba45fe8204327723df834e
I20251025 07:52:51.712692 1169 ts_tablet_manager.cc:1403] T df0d86ad301245c1a6bcb44805936d25 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.142s user 0.011s sys 0.007s
I20251025 07:52:51.713822 1169 raft_consensus.cc:359] T df0d86ad301245c1a6bcb44805936d25 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.713994 1169 raft_consensus.cc:385] T df0d86ad301245c1a6bcb44805936d25 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:51.714046 1169 raft_consensus.cc:740] T df0d86ad301245c1a6bcb44805936d25 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:51.714114 2146 ts_tablet_manager.cc:1929] T 2e098c9c4613459295b6043048cba587 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:51.714262 2146 log.cc:1199] T 2e098c9c4613459295b6043048cba587 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/2e098c9c4613459295b6043048cba587
I20251025 07:52:51.715939 2146 ts_tablet_manager.cc:1950] T 2e098c9c4613459295b6043048cba587 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:51.716748 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 383) took 1827 ms. Trace:
I20251025 07:52:51.717664 2146 rpcz_store.cc:276] 1025 07:52:49.888803 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.888848 (+ 45us) service_pool.cc:225] Handling call
1025 07:52:51.706815 (+1817967us) ts_tablet_manager.cc:1151] Deleting tablet 2e098c9c4613459295b6043048cba587
1025 07:52:51.706823 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.711047 (+ 4224us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.714019 (+ 2972us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.716731 (+ 2712us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1817826}
I20251025 07:52:51.718124 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 2e098c9c4613459295b6043048cba587 (table default.Table_9f52d94037424befbdea5c711068396a [id=55440328803042a28df1b6869d279f59]) successfully deleted
W20251025 07:52:51.718357 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 050a5d3b5f934dd19bf2e221f2fe0780 because the tablet was not found. No further retry: Not found: Tablet not found: 050a5d3b5f934dd19bf2e221f2fe0780
I20251025 07:52:51.719422 2146 tablet_replica.cc:333] T 3c4970c8a491460e8b4b1c29fde83e4c P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:51.719632 2146 raft_consensus.cc:2243] T 3c4970c8a491460e8b4b1c29fde83e4c P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:51.719776 2146 raft_consensus.cc:2272] T 3c4970c8a491460e8b4b1c29fde83e4c P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:51.719765 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 36c8fcafd5464b4e999276a8062a812a because the tablet was not found. No further retry: Not found: Tablet not found: 36c8fcafd5464b4e999276a8062a812a
I20251025 07:52:51.720264 2146 ts_tablet_manager.cc:1916] T 3c4970c8a491460e8b4b1c29fde83e4c P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:51.721731 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 110a8e9e4c6c434284c0ae14dd6987f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 110a8e9e4c6c434284c0ae14dd6987f7 already in progress: creating tablet
W20251025 07:52:51.722685 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 11bfe0c87ca941a3846462c8a4512d83 because the tablet was not found. No further retry: Not found: Tablet not found: 11bfe0c87ca941a3846462c8a4512d83
I20251025 07:52:51.722939 1169 consensus_queue.cc:260] T df0d86ad301245c1a6bcb44805936d25 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
W20251025 07:52:51.723275 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet f661d5909657455581dcb5e6eadd3c0b because the tablet was not found. No further retry: Not found: Tablet not found: f661d5909657455581dcb5e6eadd3c0b
W20251025 07:52:51.724553 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 88eb51447e0f4080baa49dd902deb24f because the tablet was not found. No further retry: Not found: Tablet not found: 88eb51447e0f4080baa49dd902deb24f
W20251025 07:52:51.725114 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet e4da89f863cc43f3bf6df908bb9dc6bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4da89f863cc43f3bf6df908bb9dc6bf already in progress: creating tablet
W20251025 07:52:51.725657 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet d77704b4fe6c4456a2ba40ffec0c60fc because the tablet was not found. No further retry: Not found: Tablet not found: d77704b4fe6c4456a2ba40ffec0c60fc
W20251025 07:52:51.726387 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 60c60a9ee37f43978910bd067a098bb7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60c60a9ee37f43978910bd067a098bb7 already in progress: creating tablet
W20251025 07:52:51.726838 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet ebf3a2a926754305b5bd95fcebb25d29 because the tablet was not found. No further retry: Not found: Tablet not found: ebf3a2a926754305b5bd95fcebb25d29
W20251025 07:52:51.728590 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 26c849d376d143c6bef0c8e0e1d7d03a because the tablet was not found. No further retry: Not found: Tablet not found: 26c849d376d143c6bef0c8e0e1d7d03a
W20251025 07:52:51.733081 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 48ff4ce7846b4e56a37eee9bcb17bd5e because the tablet was not found. No further retry: Not found: Tablet not found: 48ff4ce7846b4e56a37eee9bcb17bd5e
I20251025 07:52:51.738976 2146 ts_tablet_manager.cc:1929] T 3c4970c8a491460e8b4b1c29fde83e4c P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:51.739094 2146 log.cc:1199] T 3c4970c8a491460e8b4b1c29fde83e4c P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/3c4970c8a491460e8b4b1c29fde83e4c
I20251025 07:52:51.739385 2146 ts_tablet_manager.cc:1950] T 3c4970c8a491460e8b4b1c29fde83e4c P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:51.739861 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 384) took 1850 ms. Trace:
I20251025 07:52:51.739945 2146 rpcz_store.cc:276] 1025 07:52:49.889170 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.889213 (+ 43us) service_pool.cc:225] Handling call
1025 07:52:51.719377 (+1830164us) ts_tablet_manager.cc:1151] Deleting tablet 3c4970c8a491460e8b4b1c29fde83e4c
1025 07:52:51.719386 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.735673 (+ 16287us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.738872 (+ 3199us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.739842 (+ 970us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1830027}
I20251025 07:52:51.740116 1169 ts_tablet_manager.cc:1434] T df0d86ad301245c1a6bcb44805936d25 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.027s user 0.003s sys 0.000s
I20251025 07:52:51.740370 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 3c4970c8a491460e8b4b1c29fde83e4c (table default.Table_09095c0829ed4d1cb8b763fc695e7fa8 [id=e372ca4384644779a6b37f389f549dec]) successfully deleted
I20251025 07:52:51.740808 2146 tablet_replica.cc:333] T 32a143a715cb485cb4501f4e28c68252 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:51.741030 2146 raft_consensus.cc:2243] T 32a143a715cb485cb4501f4e28c68252 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:51.741215 2146 raft_consensus.cc:2272] T 32a143a715cb485cb4501f4e28c68252 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:51.741758 2146 ts_tablet_manager.cc:1916] T 32a143a715cb485cb4501f4e28c68252 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:51.742704 3203 consensus_queue.cc:1048] T 0582ee42ab564c07ae3facfc5935617c P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251025 07:52:51.746825 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet e47c4f741f354a19b49777bd06e54764 because the tablet was not found. No further retry: Not found: Tablet not found: e47c4f741f354a19b49777bd06e54764
I20251025 07:52:51.753501 883 tablet_service.cc:1505] Processing CreateTablet for tablet 05b675d7b06e4f07a37cfede382fc3b6 (DEFAULT_TABLE table=default.Table_feeac6a0855742cb8addcbd7508cfec5 [id=8d2a359292fb45f7bd401f0b540acf7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:51.753857 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05b675d7b06e4f07a37cfede382fc3b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:51.757140 2146 ts_tablet_manager.cc:1929] T 32a143a715cb485cb4501f4e28c68252 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:51.757227 2146 log.cc:1199] T 32a143a715cb485cb4501f4e28c68252 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/32a143a715cb485cb4501f4e28c68252
I20251025 07:52:51.757515 2146 ts_tablet_manager.cc:1950] T 32a143a715cb485cb4501f4e28c68252 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:51.760326 1169 tablet_bootstrap.cc:492] T 437da5ded9b648abb668a8cb4896452c P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:51.761703 1169 tablet_bootstrap.cc:654] T 437da5ded9b648abb668a8cb4896452c P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:51.762852 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 385) took 1873 ms. Trace:
I20251025 07:52:51.762945 2146 rpcz_store.cc:276] 1025 07:52:49.889529 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.889569 (+ 40us) service_pool.cc:225] Handling call
1025 07:52:51.740758 (+1851189us) ts_tablet_manager.cc:1151] Deleting tablet 32a143a715cb485cb4501f4e28c68252
1025 07:52:51.740769 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.747980 (+ 7211us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.757050 (+ 9070us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.762834 (+ 5784us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4631,"tablet-delete.queue_time_us":1850965}
I20251025 07:52:51.763415 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 32a143a715cb485cb4501f4e28c68252 (table default.Table_e386bc02f89041418ff3ddf8ac224eea [id=e800c96844e843b7a6fd3145907e5503]) successfully deleted
I20251025 07:52:51.763733 2146 tablet_replica.cc:333] T cc407c7eb12e4eadb41fb21385611c94 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:51.763912 2146 raft_consensus.cc:2243] T cc407c7eb12e4eadb41fb21385611c94 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:51.764045 2146 raft_consensus.cc:2272] T cc407c7eb12e4eadb41fb21385611c94 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:51.764823 2146 ts_tablet_manager.cc:1916] T cc407c7eb12e4eadb41fb21385611c94 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:51.766353 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet f97acc9a638b4d5a968dcc3412a384bc because the tablet was not found. No further retry: Not found: Tablet not found: f97acc9a638b4d5a968dcc3412a384bc
I20251025 07:52:51.767063 442 catalog_manager.cc:4985] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): tablet 4f7c15529be54a11961f385a953b0b03 (table default.Table_06053a14dcc84e79917273c10b06b751 [id=c42ad6bf27aa4744b66b4462ac97e57d]) successfully deleted
W20251025 07:52:51.768671 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 95efd553493e4966bc48a01cac6e1e10 because the tablet was not found. No further retry: Not found: Tablet not found: 95efd553493e4966bc48a01cac6e1e10
W20251025 07:52:51.770496 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 792f32cbd66449ca98813c669b2a261d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 792f32cbd66449ca98813c669b2a261d already in progress: creating tablet
W20251025 07:52:51.771128 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 4b278ce7768349e49dcd1131e404ab34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b278ce7768349e49dcd1131e404ab34 already in progress: creating tablet
I20251025 07:52:51.774415 883 tablet_service.cc:1505] Processing CreateTablet for tablet 50cc270e0ebb4fd1a10f3bf2d627cb46 (DEFAULT_TABLE table=default.Table_4ee751db640d4715b835ff2a951f515f [id=08858292b06c4e929666fa16536fe9b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:51.775401 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50cc270e0ebb4fd1a10f3bf2d627cb46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:51.775738 2146 ts_tablet_manager.cc:1929] T cc407c7eb12e4eadb41fb21385611c94 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:51.775851 2146 log.cc:1199] T cc407c7eb12e4eadb41fb21385611c94 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/cc407c7eb12e4eadb41fb21385611c94
I20251025 07:52:51.776139 2146 ts_tablet_manager.cc:1950] T cc407c7eb12e4eadb41fb21385611c94 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:51.776533 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 386) took 1886 ms. Trace:
I20251025 07:52:51.776608 2146 rpcz_store.cc:276] 1025 07:52:49.889884 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.889925 (+ 41us) service_pool.cc:225] Handling call
1025 07:52:51.763690 (+1873765us) ts_tablet_manager.cc:1151] Deleting tablet cc407c7eb12e4eadb41fb21385611c94
1025 07:52:51.763699 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.767957 (+ 4258us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.775650 (+ 7693us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.776519 (+ 869us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1873634}
I20251025 07:52:51.776935 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 387) took 1886 ms. Trace:
I20251025 07:52:51.777000 2146 rpcz_store.cc:276] 1025 07:52:49.890360 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.890404 (+ 44us) service_pool.cc:225] Handling call
1025 07:52:51.776855 (+1886451us) ts_tablet_manager.cc:1151] Deleting tablet 6897405bbc1c47a4ab0aa3af80e60e3d
1025 07:52:51.776863 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.776923 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1886299}
I20251025 07:52:51.777053 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet cc407c7eb12e4eadb41fb21385611c94 (table default.Table_50fe33e9d42343ff9194780a28b1b1b7 [id=01b6f41ccb23455b87ed38ed96184aa5]) successfully deleted
I20251025 07:52:51.777245 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 389) took 1886 ms. Trace:
I20251025 07:52:51.777314 2146 rpcz_store.cc:276] 1025 07:52:49.891098 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.891139 (+ 41us) service_pool.cc:225] Handling call
1025 07:52:51.777180 (+1886041us) ts_tablet_manager.cc:1151] Deleting tablet 1912e59548874e0e8e989050ec0064c5
1025 07:52:51.777187 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.777233 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1885903}
I20251025 07:52:51.777608 2146 tablet_replica.cc:333] T 0540d6a10cf04fa89756623e9aaaeb68 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
W20251025 07:52:51.777697 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 6897405bbc1c47a4ab0aa3af80e60e3d because the tablet was not found. No further retry: Not found: Tablet not found: 6897405bbc1c47a4ab0aa3af80e60e3d
I20251025 07:52:51.777773 2146 raft_consensus.cc:2243] T 0540d6a10cf04fa89756623e9aaaeb68 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:51.777892 2146 raft_consensus.cc:2272] T 0540d6a10cf04fa89756623e9aaaeb68 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251025 07:52:51.778110 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 1912e59548874e0e8e989050ec0064c5 because the tablet was not found. No further retry: Not found: Tablet not found: 1912e59548874e0e8e989050ec0064c5
W20251025 07:52:51.784324 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet c11f674385184ae6b235f1db8bf1fb91 because the tablet was not found. No further retry: Not found: Tablet not found: c11f674385184ae6b235f1db8bf1fb91
W20251025 07:52:51.784698 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 2e098c9c4613459295b6043048cba587 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e098c9c4613459295b6043048cba587 already in progress: creating tablet
W20251025 07:52:51.784987 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet cc407c7eb12e4eadb41fb21385611c94 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc407c7eb12e4eadb41fb21385611c94 already in progress: creating tablet
W20251025 07:52:51.785246 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 6897405bbc1c47a4ab0aa3af80e60e3d because the tablet was not found. No further retry: Not found: Tablet not found: 6897405bbc1c47a4ab0aa3af80e60e3d
W20251025 07:52:51.786110 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 3c4970c8a491460e8b4b1c29fde83e4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c4970c8a491460e8b4b1c29fde83e4c already in progress: creating tablet
W20251025 07:52:51.786410 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 1912e59548874e0e8e989050ec0064c5 because the tablet was not found. No further retry: Not found: Tablet not found: 1912e59548874e0e8e989050ec0064c5
I20251025 07:52:51.786628 2146 ts_tablet_manager.cc:1916] T 0540d6a10cf04fa89756623e9aaaeb68 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:51.786669 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 0540d6a10cf04fa89756623e9aaaeb68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0540d6a10cf04fa89756623e9aaaeb68 already in progress: creating tablet
W20251025 07:52:51.786967 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet f69cdaa7ee094283890036c0189e2c81 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f69cdaa7ee094283890036c0189e2c81 already in progress: creating tablet
W20251025 07:52:51.787246 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 32a143a715cb485cb4501f4e28c68252 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32a143a715cb485cb4501f4e28c68252 already in progress: creating tablet
W20251025 07:52:51.787499 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 6dcab53a020847a1aa27930265ba0205 because the tablet was not found. No further retry: Not found: Tablet not found: 6dcab53a020847a1aa27930265ba0205
W20251025 07:52:51.787765 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 4a1ec8c2ed7f41abb588f592f62097d1 because the tablet was not found. No further retry: Not found: Tablet not found: 4a1ec8c2ed7f41abb588f592f62097d1
W20251025 07:52:51.788013 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet b02f5c0ab0b24288bb50a3ba0e6bd78a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b02f5c0ab0b24288bb50a3ba0e6bd78a already in progress: creating tablet
W20251025 07:52:51.788285 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 29dbaf9149134f548a974a5779f7da67 because the tablet was not found. No further retry: Not found: Tablet not found: 29dbaf9149134f548a974a5779f7da67
W20251025 07:52:51.788499 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet ee1c8ea694284d80a30f9c05be4cbf49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee1c8ea694284d80a30f9c05be4cbf49 already in progress: creating tablet
W20251025 07:52:51.788743 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet b947ecc352e54f4caa73c6ca97372256 because the tablet was not found. No further retry: Not found: Tablet not found: b947ecc352e54f4caa73c6ca97372256
W20251025 07:52:51.788985 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 1b5ac7e845a24b2eb5b2429d47361232 because the tablet was not found. No further retry: Not found: Tablet not found: 1b5ac7e845a24b2eb5b2429d47361232
I20251025 07:52:51.789119 1039 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "294d7d20c2f545b8813e632aa058a60c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
W20251025 07:52:51.789256 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 9f039d9107ae493486d799ab6a1d731d because the tablet was not found. No further retry: Not found: Tablet not found: 9f039d9107ae493486d799ab6a1d731d
W20251025 07:52:51.789433 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 50ae9bca6bed4f749d54191ac1543410 because the tablet was not found. No further retry: Not found: Tablet not found: 50ae9bca6bed4f749d54191ac1543410
W20251025 07:52:51.789642 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 72dc084f8efc45478771c247e07a16f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72dc084f8efc45478771c247e07a16f4 already in progress: creating tablet
W20251025 07:52:51.789870 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet fd265cdbf6a8440393a2b5d8240d1872 because the tablet was not found. No further retry: Not found: Tablet not found: fd265cdbf6a8440393a2b5d8240d1872
I20251025 07:52:51.789929 1039 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "141e413d2faa4c67b4c822192e631417" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:51.790092 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 2b594d4718344aafbdaec34040695589 because the tablet was not found. No further retry: Not found: Tablet not found: 2b594d4718344aafbdaec34040695589
I20251025 07:52:51.790203 1039 raft_consensus.cc:2393] T 141e413d2faa4c67b4c822192e631417 P db42be7ad3c741a0aa427f27e245993b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3aea2869ff814cc7b142012faa274382 in current term 2: Already voted for candidate 23eeffd3de254ce18db5febfde971c6b in this term.
W20251025 07:52:51.790325 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 570201e7bf05406a97c81dde071d894b because the tablet was not found. No further retry: Not found: Tablet not found: 570201e7bf05406a97c81dde071d894b
W20251025 07:52:51.790548 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 623dec4eab1d4666baf2334a01eeb36d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 623dec4eab1d4666baf2334a01eeb36d already in progress: creating tablet
W20251025 07:52:51.790805 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet fa998b9ca6724c4cb1eb68a3eecb0b11 because the tablet was not found. No further retry: Not found: Tablet not found: fa998b9ca6724c4cb1eb68a3eecb0b11
W20251025 07:52:51.791055 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet ff3daed6c9af47e3976b82b4b9a41086 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff3daed6c9af47e3976b82b4b9a41086 already in progress: creating tablet
W20251025 07:52:51.791289 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet a7ab3150a35b4caca9119a1374d3f2f7 because the tablet was not found. No further retry: Not found: Tablet not found: a7ab3150a35b4caca9119a1374d3f2f7
W20251025 07:52:51.794919 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet ab42909afc524f2fa3c1041495d13a59 because the tablet was not found. No further retry: Not found: Tablet not found: ab42909afc524f2fa3c1041495d13a59
W20251025 07:52:51.795218 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet ef1854d35a3047e3a9fbe0166ab1295a because the tablet was not found. No further retry: Not found: Tablet not found: ef1854d35a3047e3a9fbe0166ab1295a
I20251025 07:52:51.795239 1039 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c4970c8a491460e8b4b1c29fde83e4c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
W20251025 07:52:51.795545 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 6ec421eb820b4f40afcff5a5bf80c508 because the tablet was not found. No further retry: Not found: Tablet not found: 6ec421eb820b4f40afcff5a5bf80c508
W20251025 07:52:51.795742 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 3abfd52327a04ba88625f391a8658fbb because the tablet was not found. No further retry: Not found: Tablet not found: 3abfd52327a04ba88625f391a8658fbb
W20251025 07:52:51.795945 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 4b28215f61574cf48c447dd01955ac77 because the tablet was not found. No further retry: Not found: Tablet not found: 4b28215f61574cf48c447dd01955ac77
W20251025 07:52:51.796221 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet d66ad049d5484228a268b0facb00ecf1 because the tablet was not found. No further retry: Not found: Tablet not found: d66ad049d5484228a268b0facb00ecf1
I20251025 07:52:51.697975 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 332) took 1852 ms. Trace:
I20251025 07:52:51.796518 2166 rpcz_store.cc:276] 1025 07:52:49.845995 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.846044 (+ 49us) service_pool.cc:225] Handling call
1025 07:52:51.697882 (+1851838us) ts_tablet_manager.cc:1151] Deleting tablet af743ee344e543a0aa6da3cd7a30e467
1025 07:52:51.697892 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.697952 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1851689}
I20251025 07:52:51.796702 2146 ts_tablet_manager.cc:1929] T 0540d6a10cf04fa89756623e9aaaeb68 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:51.796818 2146 log.cc:1199] T 0540d6a10cf04fa89756623e9aaaeb68 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/0540d6a10cf04fa89756623e9aaaeb68
W20251025 07:52:51.796479 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet df1248f6dbd94af68c587da7d149e279 because the tablet was not found. No further retry: Not found: Tablet not found: df1248f6dbd94af68c587da7d149e279
I20251025 07:52:51.772385 755 tablet_service.cc:1505] Processing CreateTablet for tablet 05b675d7b06e4f07a37cfede382fc3b6 (DEFAULT_TABLE table=default.Table_feeac6a0855742cb8addcbd7508cfec5 [id=8d2a359292fb45f7bd401f0b540acf7e]), partition=RANGE (key) PARTITION UNBOUNDED
W20251025 07:52:51.797097 442 catalog_manager.cc:4965] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet af743ee344e543a0aa6da3cd7a30e467 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af743ee344e543a0aa6da3cd7a30e467 already in progress: creating tablet
W20251025 07:52:51.797340 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 34315f64b89148efbf1f20a43d5c5efb because the tablet was not found. No further retry: Not found: Tablet not found: 34315f64b89148efbf1f20a43d5c5efb
W20251025 07:52:51.797418 439 catalog_manager.cc:4952] TS 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231): delete failed for tablet af743ee344e543a0aa6da3cd7a30e467 because the tablet was not found. No further retry: Not found: Tablet not found: af743ee344e543a0aa6da3cd7a30e467
W20251025 07:52:51.797518 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet a4e4b99be0434baaa75d988281797846 because the tablet was not found. No further retry: Not found: Tablet not found: a4e4b99be0434baaa75d988281797846
I20251025 07:52:51.702075 771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d46673a283fe42b78ddeb99fbd2678eb" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
W20251025 07:52:51.797722 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet 92aa605fabbc4257a6a204c087e7cc7f because the tablet was not found. No further retry: Not found: Tablet not found: 92aa605fabbc4257a6a204c087e7cc7f
I20251025 07:52:51.697172 1167 tablet_bootstrap.cc:654] T 0063a0776b2247afaa8be82f6eff26f1 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:51.798321 982 leader_election.cc:304] T d46673a283fe42b78ddeb99fbd2678eb P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:51.798768 3011 raft_consensus.cc:2749] T d46673a283fe42b78ddeb99fbd2678eb P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:51.799687 1169 tablet_bootstrap.cc:492] T 437da5ded9b648abb668a8cb4896452c P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:51.799821 1169 ts_tablet_manager.cc:1403] T 437da5ded9b648abb668a8cb4896452c P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.040s user 0.007s sys 0.002s
I20251025 07:52:51.800290 753 tablet_service.cc:1505] Processing CreateTablet for tablet 50cc270e0ebb4fd1a10f3bf2d627cb46 (DEFAULT_TABLE table=default.Table_4ee751db640d4715b835ff2a951f515f [id=08858292b06c4e929666fa16536fe9b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:51.800509 1169 raft_consensus.cc:359] T 437da5ded9b648abb668a8cb4896452c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.800655 1169 raft_consensus.cc:385] T 437da5ded9b648abb668a8cb4896452c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:51.800702 1169 raft_consensus.cc:740] T 437da5ded9b648abb668a8cb4896452c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:51.800884 2146 ts_tablet_manager.cc:1950] T 0540d6a10cf04fa89756623e9aaaeb68 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:51.800889 1169 consensus_queue.cc:260] T 437da5ded9b648abb668a8cb4896452c P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.801342 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 390) took 1909 ms. Trace:
I20251025 07:52:51.801440 2146 rpcz_store.cc:276] 1025 07:52:49.891440 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.891481 (+ 41us) service_pool.cc:225] Handling call
1025 07:52:51.777571 (+1886090us) ts_tablet_manager.cc:1151] Deleting tablet 0540d6a10cf04fa89756623e9aaaeb68
1025 07:52:51.777578 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.790466 (+ 12888us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.796592 (+ 6126us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.801325 (+ 4733us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1885954}
W20251025 07:52:51.797902 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet b94634ac5077439fadfbbce569f60e8b because the tablet was not found. No further retry: Not found: Tablet not found: b94634ac5077439fadfbbce569f60e8b
I20251025 07:52:51.801882 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 0540d6a10cf04fa89756623e9aaaeb68 (table default.Table_f16b7a3b7d91423f8502d73e4234ab4a [id=cc7db0a7eaea4caa8c17281a8079c372]) successfully deleted
W20251025 07:52:51.802225 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet e54fe7cbcffa40c4b04144b47bee2b48 because the tablet was not found. No further retry: Not found: Tablet not found: e54fe7cbcffa40c4b04144b47bee2b48
I20251025 07:52:51.802320 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 391) took 1910 ms. Trace:
I20251025 07:52:51.802394 1169 ts_tablet_manager.cc:1434] T 437da5ded9b648abb668a8cb4896452c P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:51.802403 2146 rpcz_store.cc:276] 1025 07:52:49.891789 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.891825 (+ 36us) service_pool.cc:225] Handling call
1025 07:52:51.802223 (+1910398us) ts_tablet_manager.cc:1151] Deleting tablet 716fb61312e94a4d9eb8ab8336ca140c
1025 07:52:51.802233 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.802303 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1910260}
I20251025 07:52:51.802742 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 392) took 1910 ms. Trace:
W20251025 07:52:51.802816 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 716fb61312e94a4d9eb8ab8336ca140c because the tablet was not found. No further retry: Not found: Tablet not found: 716fb61312e94a4d9eb8ab8336ca140c
I20251025 07:52:51.802814 2146 rpcz_store.cc:276] 1025 07:52:49.892102 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.892143 (+ 41us) service_pool.cc:225] Handling call
1025 07:52:51.802652 (+1910509us) ts_tablet_manager.cc:1151] Deleting tablet b02f5c0ab0b24288bb50a3ba0e6bd78a
1025 07:52:51.802660 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.802730 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1910383}
I20251025 07:52:51.802989 3011 raft_consensus.cc:493] T 437da5ded9b648abb668a8cb4896452c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251025 07:52:51.803097 442 catalog_manager.cc:4952] TS db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417): delete failed for tablet f656945454fe4921ab819c772a089eb3 because the tablet was not found. No further retry: Not found: Tablet not found: f656945454fe4921ab819c772a089eb3
I20251025 07:52:51.803094 3011 raft_consensus.cc:515] T 437da5ded9b648abb668a8cb4896452c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.803516 2146 tablet_replica.cc:333] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
W20251025 07:52:51.803331 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet b02f5c0ab0b24288bb50a3ba0e6bd78a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b02f5c0ab0b24288bb50a3ba0e6bd78a already in progress: creating tablet
I20251025 07:52:51.803683 3011 leader_election.cc:290] T 437da5ded9b648abb668a8cb4896452c P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:51.803859 2146 raft_consensus.cc:2243] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:51.804009 741 tablet_service.cc:1505] Processing CreateTablet for tablet a52f80e7c6a147c6a17ad7d67fc66490 (DEFAULT_TABLE table=default.Table_7a900ed2ccce48ab9116f9e8f381abc6 [id=889bee06a41a4febb421857cb1841a59]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:51.804088 2146 pending_rounds.cc:70] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382: Trying to abort 1 pending ops.
I20251025 07:52:51.804140 2146 pending_rounds.cc:77] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214607444338241536 op_type: NO_OP noop_request { }
I20251025 07:52:51.804214 2146 raft_consensus.cc:2889] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251025 07:52:51.804289 2146 raft_consensus.cc:2272] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:51.804436 771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "437da5ded9b648abb668a8cb4896452c" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:51.804801 2146 ts_tablet_manager.cc:1916] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:51.805588 1169 tablet_bootstrap.cc:492] T 1c40b2e06d894a06af9e58285892ea8d P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:51.806296 878 tablet_service.cc:1505] Processing CreateTablet for tablet d7bfe43d889a4f09b878d91f5e67e4fd (DEFAULT_TABLE table=default.Table_3053388b8968455a8ea15bba9eab3060 [id=22e1f91ad1274a608e83f9d22022ee44]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:51.806663 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7bfe43d889a4f09b878d91f5e67e4fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:51.806991 1169 tablet_bootstrap.cc:654] T 1c40b2e06d894a06af9e58285892ea8d P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:51.813520 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "437da5ded9b648abb668a8cb4896452c" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
I20251025 07:52:51.814066 983 leader_election.cc:304] T 437da5ded9b648abb668a8cb4896452c P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:51.814296 883 tablet_service.cc:1505] Processing CreateTablet for tablet a52f80e7c6a147c6a17ad7d67fc66490 (DEFAULT_TABLE table=default.Table_7a900ed2ccce48ab9116f9e8f381abc6 [id=889bee06a41a4febb421857cb1841a59]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:51.814772 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a52f80e7c6a147c6a17ad7d67fc66490. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:51.814826 3143 raft_consensus.cc:2749] T 437da5ded9b648abb668a8cb4896452c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:51.822741 1166 tablet_bootstrap.cc:492] T 421db850a64048218eecb9de65e2bdbf P 3aea2869ff814cc7b142012faa274382: No bootstrap required, opened a new log
I20251025 07:52:51.822871 1166 ts_tablet_manager.cc:1403] T 421db850a64048218eecb9de65e2bdbf P 3aea2869ff814cc7b142012faa274382: Time spent bootstrapping tablet: real 0.212s user 0.003s sys 0.016s
I20251025 07:52:51.823467 1169 tablet_bootstrap.cc:492] T 1c40b2e06d894a06af9e58285892ea8d P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:51.823616 1169 ts_tablet_manager.cc:1403] T 1c40b2e06d894a06af9e58285892ea8d P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20251025 07:52:51.823635 1166 raft_consensus.cc:359] T 421db850a64048218eecb9de65e2bdbf P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.823797 1166 raft_consensus.cc:385] T 421db850a64048218eecb9de65e2bdbf P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:51.823858 1166 raft_consensus.cc:740] T 421db850a64048218eecb9de65e2bdbf P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Initialized, Role: FOLLOWER
I20251025 07:52:51.824077 1166 consensus_queue.cc:260] T 421db850a64048218eecb9de65e2bdbf P 3aea2869ff814cc7b142012faa274382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.824651 1166 ts_tablet_manager.cc:1434] T 421db850a64048218eecb9de65e2bdbf P 3aea2869ff814cc7b142012faa274382: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251025 07:52:51.824898 1166 tablet_bootstrap.cc:492] T bbe1d40d67594ad389ccc97bb9e2ac3b P 3aea2869ff814cc7b142012faa274382: Bootstrap starting.
I20251025 07:52:51.824301 1169 raft_consensus.cc:359] T 1c40b2e06d894a06af9e58285892ea8d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.826215 1169 raft_consensus.cc:385] T 1c40b2e06d894a06af9e58285892ea8d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:51.826277 1169 raft_consensus.cc:740] T 1c40b2e06d894a06af9e58285892ea8d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:51.826431 1166 tablet_bootstrap.cc:654] T bbe1d40d67594ad389ccc97bb9e2ac3b P 3aea2869ff814cc7b142012faa274382: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:51.826475 1169 consensus_queue.cc:260] T 1c40b2e06d894a06af9e58285892ea8d P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.827231 2146 ts_tablet_manager.cc:1929] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251025 07:52:51.827334 2146 log.cc:1199] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/ee1c8ea694284d80a30f9c05be4cbf49
I20251025 07:52:51.827646 2146 ts_tablet_manager.cc:1950] T ee1c8ea694284d80a30f9c05be4cbf49 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:51.828331 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 393) took 1935 ms. Trace:
I20251025 07:52:51.828416 2146 rpcz_store.cc:276] 1025 07:52:49.892434 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.892469 (+ 35us) service_pool.cc:225] Handling call
1025 07:52:51.803476 (+1911007us) ts_tablet_manager.cc:1151] Deleting tablet ee1c8ea694284d80a30f9c05be4cbf49
1025 07:52:51.803484 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.808053 (+ 4569us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.827141 (+ 19088us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.828315 (+ 1174us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1910884}
I20251025 07:52:51.828754 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 394) took 1936 ms. Trace:
I20251025 07:52:51.828826 2146 rpcz_store.cc:276] 1025 07:52:49.892740 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.892780 (+ 40us) service_pool.cc:225] Handling call
1025 07:52:51.828671 (+1935891us) ts_tablet_manager.cc:1151] Deleting tablet 6dcab53a020847a1aa27930265ba0205
1025 07:52:51.828679 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.828742 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1935763}
I20251025 07:52:51.829083 2146 tablet_replica.cc:333] T 72dc084f8efc45478771c247e07a16f4 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:51.829293 2146 raft_consensus.cc:2243] T 72dc084f8efc45478771c247e07a16f4 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:51.829425 2146 raft_consensus.cc:2272] T 72dc084f8efc45478771c247e07a16f4 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:51.829545 1169 ts_tablet_manager.cc:1434] T 1c40b2e06d894a06af9e58285892ea8d P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251025 07:52:51.829803 1169 tablet_bootstrap.cc:492] T df9db9dd31d840e0b1394db04d662a62 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:51.829833 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet ee1c8ea694284d80a30f9c05be4cbf49 (table default.Table_e8d761a95cdf469b9cba89359c58a5a4 [id=b369bec99e4d48c1847a2a327b1d6ac8]) successfully deleted
I20251025 07:52:51.829890 2146 ts_tablet_manager.cc:1916] T 72dc084f8efc45478771c247e07a16f4 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251025 07:52:51.830113 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 6dcab53a020847a1aa27930265ba0205 because the tablet was not found. No further retry: Not found: Tablet not found: 6dcab53a020847a1aa27930265ba0205
I20251025 07:52:51.831276 1169 tablet_bootstrap.cc:654] T df9db9dd31d840e0b1394db04d662a62 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:51.836999 1167 tablet_bootstrap.cc:492] T 0063a0776b2247afaa8be82f6eff26f1 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:51.837141 1167 ts_tablet_manager.cc:1403] T 0063a0776b2247afaa8be82f6eff26f1 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.142s user 0.002s sys 0.001s
I20251025 07:52:51.837757 1167 raft_consensus.cc:359] T 0063a0776b2247afaa8be82f6eff26f1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.837913 1167 raft_consensus.cc:385] T 0063a0776b2247afaa8be82f6eff26f1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:51.837960 1167 raft_consensus.cc:740] T 0063a0776b2247afaa8be82f6eff26f1 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:51.836040 2146 ts_tablet_manager.cc:1929] T 72dc084f8efc45478771c247e07a16f4 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:51.838224 2146 log.cc:1199] T 72dc084f8efc45478771c247e07a16f4 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/72dc084f8efc45478771c247e07a16f4
I20251025 07:52:51.838502 2146 ts_tablet_manager.cc:1950] T 72dc084f8efc45478771c247e07a16f4 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:51.839152 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 395) took 1946 ms. Trace:
I20251025 07:52:51.839249 2146 rpcz_store.cc:276] 1025 07:52:49.893052 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.893090 (+ 38us) service_pool.cc:225] Handling call
1025 07:52:51.829041 (+1935951us) ts_tablet_manager.cc:1151] Deleting tablet 72dc084f8efc45478771c247e07a16f4
1025 07:52:51.829049 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.832959 (+ 3910us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.835966 (+ 3007us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.839134 (+ 3168us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1935826}
I20251025 07:52:51.839653 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet 72dc084f8efc45478771c247e07a16f4 (table default.Table_5a8143ecfd74418f928ceb9ae9cd50ba [id=23d0aa66d6f545d4b579cfd63b7e7143]) successfully deleted
I20251025 07:52:51.840471 1169 tablet_bootstrap.cc:492] T df9db9dd31d840e0b1394db04d662a62 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:51.840598 1169 ts_tablet_manager.cc:1403] T df9db9dd31d840e0b1394db04d662a62 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20251025 07:52:51.841167 1169 raft_consensus.cc:359] T df9db9dd31d840e0b1394db04d662a62 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.841329 1169 raft_consensus.cc:385] T df9db9dd31d840e0b1394db04d662a62 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:51.841395 1169 raft_consensus.cc:740] T df9db9dd31d840e0b1394db04d662a62 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:51.841610 1169 consensus_queue.cc:260] T df9db9dd31d840e0b1394db04d662a62 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.842053 1169 ts_tablet_manager.cc:1434] T df9db9dd31d840e0b1394db04d662a62 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251025 07:52:51.842319 1169 tablet_bootstrap.cc:492] T af743ee344e543a0aa6da3cd7a30e467 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:51.843896 1169 tablet_bootstrap.cc:654] T af743ee344e543a0aa6da3cd7a30e467 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:51.845166 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 333) took 1998 ms. Trace:
I20251025 07:52:51.845255 2166 rpcz_store.cc:276] 1025 07:52:49.847174 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.847226 (+ 52us) service_pool.cc:225] Handling call
1025 07:52:51.796842 (+1949616us) ts_tablet_manager.cc:1151] Deleting tablet f3c87a83082b421cb4be546ac4ce7ad8
1025 07:52:51.845044 (+ 48202us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.845132 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1949454}
I20251025 07:52:51.845602 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 334) took 1997 ms. Trace:
I20251025 07:52:51.845674 2166 rpcz_store.cc:276] 1025 07:52:49.847700 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.847749 (+ 49us) service_pool.cc:225] Handling call
1025 07:52:51.845525 (+1997776us) ts_tablet_manager.cc:1151] Deleting tablet 9d6a9937d10a4cc2813671dd1bfa1664
1025 07:52:51.845535 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.845588 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1997648}
I20251025 07:52:51.845918 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 335) took 1997 ms. Trace:
I20251025 07:52:51.845986 2166 rpcz_store.cc:276] 1025 07:52:49.848380 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.848432 (+ 52us) service_pool.cc:225] Handling call
1025 07:52:51.845858 (+1997426us) ts_tablet_manager.cc:1151] Deleting tablet e4da89f863cc43f3bf6df908bb9dc6bf
1025 07:52:51.845866 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.845907 (+ 41us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1997291}
I20251025 07:52:51.846378 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05b675d7b06e4f07a37cfede382fc3b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:51.850091 1167 consensus_queue.cc:260] T 0063a0776b2247afaa8be82f6eff26f1 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.850664 1167 ts_tablet_manager.cc:1434] T 0063a0776b2247afaa8be82f6eff26f1 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20251025 07:52:51.851200 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 396) took 1957 ms. Trace:
I20251025 07:52:51.851280 2146 rpcz_store.cc:276] 1025 07:52:49.893418 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.893455 (+ 37us) service_pool.cc:225] Handling call
1025 07:52:51.851093 (+1957638us) ts_tablet_manager.cc:1151] Deleting tablet 50ae9bca6bed4f749d54191ac1543410
1025 07:52:51.851104 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.851180 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1957520}
I20251025 07:52:51.851823 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50cc270e0ebb4fd1a10f3bf2d627cb46. 1 dirs total, 0 dirs full, 0 dirs failed
W20251025 07:52:51.851820 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 50ae9bca6bed4f749d54191ac1543410 because the tablet was not found. No further retry: Not found: Tablet not found: 50ae9bca6bed4f749d54191ac1543410
I20251025 07:52:51.852195 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 397) took 1958 ms. Trace:
I20251025 07:52:51.852353 2146 rpcz_store.cc:276] 1025 07:52:49.893773 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.893811 (+ 38us) service_pool.cc:225] Handling call
1025 07:52:51.852111 (+1958300us) ts_tablet_manager.cc:1151] Deleting tablet fd265cdbf6a8440393a2b5d8240d1872
1025 07:52:51.852120 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.852176 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1958176}
W20251025 07:52:51.852816 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet fd265cdbf6a8440393a2b5d8240d1872 because the tablet was not found. No further retry: Not found: Tablet not found: fd265cdbf6a8440393a2b5d8240d1872
I20251025 07:52:51.853219 1167 tablet_bootstrap.cc:492] T cc407c7eb12e4eadb41fb21385611c94 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:51.854058 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 398) took 1960 ms. Trace:
I20251025 07:52:51.854163 2146 rpcz_store.cc:276] 1025 07:52:49.894076 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.894113 (+ 37us) service_pool.cc:225] Handling call
1025 07:52:51.853974 (+1959861us) ts_tablet_manager.cc:1151] Deleting tablet 1b5ac7e845a24b2eb5b2429d47361232
1025 07:52:51.853984 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.854042 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1959693}
W20251025 07:52:51.854605 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 1b5ac7e845a24b2eb5b2429d47361232 because the tablet was not found. No further retry: Not found: Tablet not found: 1b5ac7e845a24b2eb5b2429d47361232
I20251025 07:52:51.854725 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a52f80e7c6a147c6a17ad7d67fc66490. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:51.854894 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 399) took 1960 ms. Trace:
I20251025 07:52:51.856040 1167 tablet_bootstrap.cc:654] T cc407c7eb12e4eadb41fb21385611c94 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:51.854969 2146 rpcz_store.cc:276] 1025 07:52:49.894452 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.894491 (+ 39us) service_pool.cc:225] Handling call
1025 07:52:51.854822 (+1960331us) ts_tablet_manager.cc:1151] Deleting tablet a7ab3150a35b4caca9119a1374d3f2f7
1025 07:52:51.854831 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.854880 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1960197}
W20251025 07:52:51.856536 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet a7ab3150a35b4caca9119a1374d3f2f7 because the tablet was not found. No further retry: Not found: Tablet not found: a7ab3150a35b4caca9119a1374d3f2f7
I20251025 07:52:51.856856 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 402) took 1913 ms. Trace:
I20251025 07:52:51.856932 2146 rpcz_store.cc:276] 1025 07:52:49.943922 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.944069 (+ 147us) service_pool.cc:225] Handling call
1025 07:52:51.856786 (+1912717us) ts_tablet_manager.cc:1151] Deleting tablet 2b594d4718344aafbdaec34040695589
1025 07:52:51.856795 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.856843 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1912529}
W20251025 07:52:51.857296 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 2b594d4718344aafbdaec34040695589 because the tablet was not found. No further retry: Not found: Tablet not found: 2b594d4718344aafbdaec34040695589
I20251025 07:52:51.857849 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 403) took 1911 ms. Trace:
I20251025 07:52:51.857923 2146 rpcz_store.cc:276] 1025 07:52:49.946047 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.946269 (+ 222us) service_pool.cc:225] Handling call
1025 07:52:51.857780 (+1911511us) ts_tablet_manager.cc:1151] Deleting tablet fa998b9ca6724c4cb1eb68a3eecb0b11
1025 07:52:51.857788 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.857836 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1911350}
W20251025 07:52:51.858332 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet fa998b9ca6724c4cb1eb68a3eecb0b11 because the tablet was not found. No further retry: Not found: Tablet not found: fa998b9ca6724c4cb1eb68a3eecb0b11
I20251025 07:52:51.858973 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 406) took 1902 ms. Trace:
I20251025 07:52:51.859041 2146 rpcz_store.cc:276] 1025 07:52:49.956617 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.956670 (+ 53us) service_pool.cc:225] Handling call
1025 07:52:51.858905 (+1902235us) ts_tablet_manager.cc:1151] Deleting tablet d66ad049d5484228a268b0facb00ecf1
1025 07:52:51.858914 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.858960 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1902014}
W20251025 07:52:51.859433 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet d66ad049d5484228a268b0facb00ecf1 because the tablet was not found. No further retry: Not found: Tablet not found: d66ad049d5484228a268b0facb00ecf1
I20251025 07:52:51.859676 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 336) took 2010 ms. Trace:
I20251025 07:52:51.859766 2166 rpcz_store.cc:276] 1025 07:52:49.848820 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.848869 (+ 49us) service_pool.cc:225] Handling call
1025 07:52:51.859589 (+2010720us) ts_tablet_manager.cc:1151] Deleting tablet d77704b4fe6c4456a2ba40ffec0c60fc
1025 07:52:51.859599 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.859660 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2010578}
I20251025 07:52:51.860070 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 337) took 2010 ms. Trace:
I20251025 07:52:51.860145 2166 rpcz_store.cc:276] 1025 07:52:49.849986 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.850038 (+ 52us) service_pool.cc:225] Handling call
1025 07:52:51.859997 (+2009959us) ts_tablet_manager.cc:1151] Deleting tablet 6897405bbc1c47a4ab0aa3af80e60e3d
1025 07:52:51.860005 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.860057 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2009802}
I20251025 07:52:51.860296 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 405) took 1904 ms. Trace:
I20251025 07:52:51.860412 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 339) took 1964 ms. Trace:
I20251025 07:52:51.860373 2146 rpcz_store.cc:276] 1025 07:52:49.956194 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.957127 (+ 933us) service_pool.cc:225] Handling call
1025 07:52:51.860225 (+1903098us) ts_tablet_manager.cc:1151] Deleting tablet 4b28215f61574cf48c447dd01955ac77
1025 07:52:51.860233 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.860282 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1902933}
I20251025 07:52:51.860482 2166 rpcz_store.cc:276] 1025 07:52:49.896273 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.896320 (+ 47us) service_pool.cc:225] Handling call
1025 07:52:51.860340 (+1964020us) ts_tablet_manager.cc:1151] Deleting tablet c59f6e6eb9594a8e83f109a09b034045
1025 07:52:51.860350 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.860399 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1963841}
I20251025 07:52:51.860786 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 340) took 1963 ms. Trace:
W20251025 07:52:51.860778 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 4b28215f61574cf48c447dd01955ac77 because the tablet was not found. No further retry: Not found: Tablet not found: 4b28215f61574cf48c447dd01955ac77
I20251025 07:52:51.860859 2166 rpcz_store.cc:276] 1025 07:52:49.896969 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.897016 (+ 47us) service_pool.cc:225] Handling call
1025 07:52:51.860694 (+1963678us) ts_tablet_manager.cc:1151] Deleting tablet b02f5c0ab0b24288bb50a3ba0e6bd78a
1025 07:52:51.860701 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.860771 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1963526}
I20251025 07:52:51.861111 2166 tablet_replica.cc:333] T 3c4970c8a491460e8b4b1c29fde83e4c P 23eeffd3de254ce18db5febfde971c6b: stopping tablet replica
I20251025 07:52:51.861307 2166 raft_consensus.cc:2243] T 3c4970c8a491460e8b4b1c29fde83e4c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:51.861325 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 407) took 1875 ms. Trace:
I20251025 07:52:51.861455 2166 raft_consensus.cc:2272] T 3c4970c8a491460e8b4b1c29fde83e4c P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:51.861389 2146 rpcz_store.cc:276] 1025 07:52:49.986355 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.986430 (+ 75us) service_pool.cc:225] Handling call
1025 07:52:51.861257 (+1874827us) ts_tablet_manager.cc:1151] Deleting tablet 3abfd52327a04ba88625f391a8658fbb
1025 07:52:51.861265 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.861312 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1854988}
I20251025 07:52:51.861495 1169 tablet_bootstrap.cc:492] T af743ee344e543a0aa6da3cd7a30e467 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:51.861727 1169 ts_tablet_manager.cc:1403] T af743ee344e543a0aa6da3cd7a30e467 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
W20251025 07:52:51.861754 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 3abfd52327a04ba88625f391a8658fbb because the tablet was not found. No further retry: Not found: Tablet not found: 3abfd52327a04ba88625f391a8658fbb
I20251025 07:52:51.861967 2166 ts_tablet_manager.cc:1916] T 3c4970c8a491460e8b4b1c29fde83e4c P 23eeffd3de254ce18db5febfde971c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:51.862071 2146 tablet_replica.cc:333] T af743ee344e543a0aa6da3cd7a30e467 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
I20251025 07:52:51.862318 2146 raft_consensus.cc:2243] T af743ee344e543a0aa6da3cd7a30e467 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251025 07:52:51.862495 2146 raft_consensus.cc:2272] T af743ee344e543a0aa6da3cd7a30e467 P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:51.862520 1169 raft_consensus.cc:359] T af743ee344e543a0aa6da3cd7a30e467 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.862695 1169 raft_consensus.cc:385] T af743ee344e543a0aa6da3cd7a30e467 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:51.862766 1169 raft_consensus.cc:740] T af743ee344e543a0aa6da3cd7a30e467 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:51.863106 2146 ts_tablet_manager.cc:1916] T af743ee344e543a0aa6da3cd7a30e467 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:51.863067 1169 consensus_queue.cc:260] T af743ee344e543a0aa6da3cd7a30e467 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.863785 3203 raft_consensus.cc:493] T a6587f4be6af44dd944e6b6f9599dfda P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:51.863917 3203 raft_consensus.cc:515] T a6587f4be6af44dd944e6b6f9599dfda P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.864184 1169 ts_tablet_manager.cc:1434] T af743ee344e543a0aa6da3cd7a30e467 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251025 07:52:51.864445 3203 leader_election.cc:290] T a6587f4be6af44dd944e6b6f9599dfda P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:51.864471 1169 tablet_bootstrap.cc:492] T a6ae1315b28940d49b667e3c80347021 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:51.864462 3204 consensus_queue.cc:1048] T ab68e5a89b234064b27aa7cae9467e14 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:51.866015 1169 tablet_bootstrap.cc:654] T a6ae1315b28940d49b667e3c80347021 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:51.866354 764 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6587f4be6af44dd944e6b6f9599dfda" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:51.866532 764 raft_consensus.cc:2468] T a6587f4be6af44dd944e6b6f9599dfda P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 0.
I20251025 07:52:51.867430 847 leader_election.cc:304] T a6587f4be6af44dd944e6b6f9599dfda P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:51.867679 3204 raft_consensus.cc:2804] T a6587f4be6af44dd944e6b6f9599dfda P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251025 07:52:51.867746 3204 raft_consensus.cc:493] T a6587f4be6af44dd944e6b6f9599dfda P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251025 07:52:51.867805 3204 raft_consensus.cc:3060] T a6587f4be6af44dd944e6b6f9599dfda P 3aea2869ff814cc7b142012faa274382 [term 0 FOLLOWER]: Advancing to term 1
W20251025 07:52:51.871402 710 consensus_peers.cc:597] T 32a143a715cb485cb4501f4e28c68252 P 23eeffd3de254ce18db5febfde971c6b -> Peer 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): Couldn't send request to peer 3aea2869ff814cc7b142012faa274382. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32a143a715cb485cb4501f4e28c68252. This is attempt 1: this message will repeat every 5th retry.
I20251025 07:52:51.872107 2990 consensus_queue.cc:1048] T e429b94326cc4827a03968fd92de38e0 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:51.872385 3204 raft_consensus.cc:515] T a6587f4be6af44dd944e6b6f9599dfda P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.872958 3204 leader_election.cc:290] T a6587f4be6af44dd944e6b6f9599dfda P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Requested vote from peers db42be7ad3c741a0aa427f27e245993b (127.0.104.3:45417), 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231)
I20251025 07:52:51.873674 764 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a6587f4be6af44dd944e6b6f9599dfda" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b"
I20251025 07:52:51.873831 764 raft_consensus.cc:3060] T a6587f4be6af44dd944e6b6f9599dfda P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:51.878554 1167 tablet_bootstrap.cc:492] T cc407c7eb12e4eadb41fb21385611c94 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:51.878692 1167 ts_tablet_manager.cc:1403] T cc407c7eb12e4eadb41fb21385611c94 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.001s
I20251025 07:52:51.879364 1167 raft_consensus.cc:359] T cc407c7eb12e4eadb41fb21385611c94 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.879516 1167 raft_consensus.cc:385] T cc407c7eb12e4eadb41fb21385611c94 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:51.879561 1167 raft_consensus.cc:740] T cc407c7eb12e4eadb41fb21385611c94 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:51.879761 1167 consensus_queue.cc:260] T cc407c7eb12e4eadb41fb21385611c94 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.880456 1167 ts_tablet_manager.cc:1434] T cc407c7eb12e4eadb41fb21385611c94 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:51.880704 1167 tablet_bootstrap.cc:492] T 0540d6a10cf04fa89756623e9aaaeb68 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:51.882462 1167 tablet_bootstrap.cc:654] T 0540d6a10cf04fa89756623e9aaaeb68 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:51.884055 2146 ts_tablet_manager.cc:1929] T af743ee344e543a0aa6da3cd7a30e467 P 3aea2869ff814cc7b142012faa274382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:51.884182 2146 log.cc:1199] T af743ee344e543a0aa6da3cd7a30e467 P 3aea2869ff814cc7b142012faa274382: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-1/wal/wals/af743ee344e543a0aa6da3cd7a30e467
I20251025 07:52:51.884487 2146 ts_tablet_manager.cc:1950] T af743ee344e543a0aa6da3cd7a30e467 P 3aea2869ff814cc7b142012faa274382: Deleting consensus metadata
I20251025 07:52:51.885224 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 408) took 1898 ms. Trace:
I20251025 07:52:51.885310 2146 rpcz_store.cc:276] 1025 07:52:49.986502 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.986549 (+ 47us) service_pool.cc:225] Handling call
1025 07:52:51.861991 (+1875442us) ts_tablet_manager.cc:1151] Deleting tablet af743ee344e543a0aa6da3cd7a30e467
1025 07:52:51.861998 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.874059 (+ 12061us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.883958 (+ 9899us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.885206 (+ 1248us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1855394}
I20251025 07:52:51.885780 440 catalog_manager.cc:4985] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): tablet af743ee344e543a0aa6da3cd7a30e467 (table default.Table_843881ca867542619b4fff840d2879ca [id=d324f310a01a4ae5abbff6ddafffca7d]) successfully deleted
I20251025 07:52:51.890949 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 409) took 1904 ms. Trace:
I20251025 07:52:51.891155 2146 rpcz_store.cc:276] 1025 07:52:49.986594 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.986642 (+ 48us) service_pool.cc:225] Handling call
1025 07:52:51.890796 (+1904154us) ts_tablet_manager.cc:1151] Deleting tablet 34315f64b89148efbf1f20a43d5c5efb
1025 07:52:51.890808 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.890910 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1884120}
W20251025 07:52:51.891783 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 34315f64b89148efbf1f20a43d5c5efb because the tablet was not found. No further retry: Not found: Tablet not found: 34315f64b89148efbf1f20a43d5c5efb
I20251025 07:52:51.892244 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 410) took 1905 ms. Trace:
I20251025 07:52:51.892391 2146 rpcz_store.cc:276] 1025 07:52:49.986679 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.986787 (+ 108us) service_pool.cc:225] Handling call
1025 07:52:51.892129 (+1905342us) ts_tablet_manager.cc:1151] Deleting tablet a4e4b99be0434baaa75d988281797846
1025 07:52:51.892138 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.892212 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1885406}
W20251025 07:52:51.892851 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet a4e4b99be0434baaa75d988281797846 because the tablet was not found. No further retry: Not found: Tablet not found: a4e4b99be0434baaa75d988281797846
I20251025 07:52:51.892846 764 raft_consensus.cc:2468] T a6587f4be6af44dd944e6b6f9599dfda P 23eeffd3de254ce18db5febfde971c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aea2869ff814cc7b142012faa274382 in term 1.
I20251025 07:52:51.893241 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 411) took 1906 ms. Trace:
I20251025 07:52:51.893404 847 leader_election.cc:304] T a6587f4be6af44dd944e6b6f9599dfda P 3aea2869ff814cc7b142012faa274382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382; no voters:
I20251025 07:52:51.893860 3204 raft_consensus.cc:2804] T a6587f4be6af44dd944e6b6f9599dfda P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Leader election won for term 1
I20251025 07:52:51.894021 3204 raft_consensus.cc:697] T a6587f4be6af44dd944e6b6f9599dfda P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Becoming Leader. State: Replica: 3aea2869ff814cc7b142012faa274382, State: Running, Role: LEADER
I20251025 07:52:51.894335 3204 consensus_queue.cc:237] T a6587f4be6af44dd944e6b6f9599dfda P 3aea2869ff814cc7b142012faa274382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.898245 3011 raft_consensus.cc:493] T df9db9dd31d840e0b1394db04d662a62 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251025 07:52:51.893422 2146 rpcz_store.cc:276] 1025 07:52:49.986761 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.987244 (+ 483us) service_pool.cc:225] Handling call
1025 07:52:51.893125 (+1905881us) ts_tablet_manager.cc:1151] Deleting tablet 92aa605fabbc4257a6a204c087e7cc7f
1025 07:52:51.893134 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.893198 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1886352}
I20251025 07:52:51.898409 3011 raft_consensus.cc:515] T df9db9dd31d840e0b1394db04d662a62 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.898622 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 412) took 1911 ms. Trace:
W20251025 07:52:51.898756 440 catalog_manager.cc:4952] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 92aa605fabbc4257a6a204c087e7cc7f because the tablet was not found. No further retry: Not found: Tablet not found: 92aa605fabbc4257a6a204c087e7cc7f
I20251025 07:52:51.898729 2146 rpcz_store.cc:276] 1025 07:52:49.987470 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.987651 (+ 181us) service_pool.cc:225] Handling call
1025 07:52:51.898463 (+1910812us) ts_tablet_manager.cc:1151] Deleting tablet ff3daed6c9af47e3976b82b4b9a41086
1025 07:52:51.898473 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.898577 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1891644}
I20251025 07:52:51.899044 3011 leader_election.cc:290] T df9db9dd31d840e0b1394db04d662a62 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23eeffd3de254ce18db5febfde971c6b (127.0.104.1:46231), 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563)
I20251025 07:52:51.899063 2146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59616 (request call id 421) took 1060 ms. Trace:
I20251025 07:52:51.899173 2146 rpcz_store.cc:276] 1025 07:52:50.838563 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:50.839062 (+ 499us) service_pool.cc:225] Handling call
1025 07:52:51.898939 (+1059877us) ts_tablet_manager.cc:1151] Deleting tablet 0063a0776b2247afaa8be82f6eff26f1
1025 07:52:51.898947 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.899017 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1059675}
I20251025 07:52:51.899451 2146 tablet_replica.cc:333] T 9979cae195e849908677d78c6c33be71 P 3aea2869ff814cc7b142012faa274382: stopping tablet replica
W20251025 07:52:51.900063 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet ff3daed6c9af47e3976b82b4b9a41086 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff3daed6c9af47e3976b82b4b9a41086 already in progress: creating tablet
I20251025 07:52:51.900046 905 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df9db9dd31d840e0b1394db04d662a62" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aea2869ff814cc7b142012faa274382" is_pre_election: true
W20251025 07:52:51.900312 440 catalog_manager.cc:4965] TS 3aea2869ff814cc7b142012faa274382 (127.0.104.2:43563): delete failed for tablet 0063a0776b2247afaa8be82f6eff26f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0063a0776b2247afaa8be82f6eff26f1 already in progress: creating tablet
I20251025 07:52:51.901263 1169 tablet_bootstrap.cc:492] T a6ae1315b28940d49b667e3c80347021 P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:51.901407 1169 ts_tablet_manager.cc:1403] T a6ae1315b28940d49b667e3c80347021 P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.037s user 0.013s sys 0.002s
I20251025 07:52:51.902108 1169 raft_consensus.cc:359] T a6ae1315b28940d49b667e3c80347021 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.902302 1169 raft_consensus.cc:385] T a6ae1315b28940d49b667e3c80347021 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:51.902349 1169 raft_consensus.cc:740] T a6ae1315b28940d49b667e3c80347021 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:51.902578 1169 consensus_queue.cc:260] T a6ae1315b28940d49b667e3c80347021 P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.903385 2146 raft_consensus.cc:2243] T 9979cae195e849908677d78c6c33be71 P 3aea2869ff814cc7b142012faa274382 [term 1 LEADER]: Raft consensus shutting down.
I20251025 07:52:51.902691 764 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df9db9dd31d840e0b1394db04d662a62" candidate_uuid: "db42be7ad3c741a0aa427f27e245993b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eeffd3de254ce18db5febfde971c6b" is_pre_election: true
I20251025 07:52:51.903682 2146 raft_consensus.cc:2272] T 9979cae195e849908677d78c6c33be71 P 3aea2869ff814cc7b142012faa274382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251025 07:52:51.903893 982 leader_election.cc:304] T df9db9dd31d840e0b1394db04d662a62 P db42be7ad3c741a0aa427f27e245993b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db42be7ad3c741a0aa427f27e245993b; no voters: 23eeffd3de254ce18db5febfde971c6b, 3aea2869ff814cc7b142012faa274382
I20251025 07:52:51.904237 2146 ts_tablet_manager.cc:1916] T 9979cae195e849908677d78c6c33be71 P 3aea2869ff814cc7b142012faa274382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251025 07:52:51.904343 3011 raft_consensus.cc:2749] T df9db9dd31d840e0b1394db04d662a62 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251025 07:52:51.908098 3325 consensus_queue.cc:1048] T 1b22bd172e0e4f86896e9ecdef666b42 P 23eeffd3de254ce18db5febfde971c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:51.910349 1169 ts_tablet_manager.cc:1434] T a6ae1315b28940d49b667e3c80347021 P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20251025 07:52:51.921689 1169 tablet_bootstrap.cc:492] T e4da89f863cc43f3bf6df908bb9dc6bf P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:51.923646 1169 tablet_bootstrap.cc:654] T e4da89f863cc43f3bf6df908bb9dc6bf P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:51.929306 1039 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9903e80a26fd469a8744dc855534ba75" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:51.932498 1169 tablet_bootstrap.cc:492] T e4da89f863cc43f3bf6df908bb9dc6bf P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:51.932637 1169 ts_tablet_manager.cc:1403] T e4da89f863cc43f3bf6df908bb9dc6bf P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20251025 07:52:51.933244 2166 ts_tablet_manager.cc:1929] T 3c4970c8a491460e8b4b1c29fde83e4c P 23eeffd3de254ce18db5febfde971c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251025 07:52:51.933363 2166 log.cc:1199] T 3c4970c8a491460e8b4b1c29fde83e4c P 23eeffd3de254ce18db5febfde971c6b: Deleting WAL directory at /tmp/dist-test-taskRBHlYt/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761378740604055-416-0/minicluster-data/ts-0/wal/wals/3c4970c8a491460e8b4b1c29fde83e4c
I20251025 07:52:51.933669 2166 ts_tablet_manager.cc:1950] T 3c4970c8a491460e8b4b1c29fde83e4c P 23eeffd3de254ce18db5febfde971c6b: Deleting consensus metadata
I20251025 07:52:51.934988 1039 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc407c7eb12e4eadb41fb21385611c94" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:51.938675 1169 raft_consensus.cc:359] T e4da89f863cc43f3bf6df908bb9dc6bf P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.938854 1169 raft_consensus.cc:385] T e4da89f863cc43f3bf6df908bb9dc6bf P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:51.938911 1169 raft_consensus.cc:740] T e4da89f863cc43f3bf6df908bb9dc6bf P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:51.939102 1169 consensus_queue.cc:260] T e4da89f863cc43f3bf6df908bb9dc6bf P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.939600 1169 ts_tablet_manager.cc:1434] T e4da89f863cc43f3bf6df908bb9dc6bf P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20251025 07:52:51.940922 2686 consensus_queue.cc:1048] T 66134661be834f7b811d7927ab629812 P 3aea2869ff814cc7b142012faa274382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251025 07:52:51.942474 1039 raft_consensus.cc:3060] T 5dab439bc50d4e869548154e1d222ad6 P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:51.947068 1167 tablet_bootstrap.cc:492] T 0540d6a10cf04fa89756623e9aaaeb68 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:51.947207 1167 ts_tablet_manager.cc:1403] T 0540d6a10cf04fa89756623e9aaaeb68 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.067s user 0.010s sys 0.001s
I20251025 07:52:51.947599 878 tablet_service.cc:1505] Processing CreateTablet for tablet eced77e6a53447db9fb2df3b670684f2 (DEFAULT_TABLE table=default.Table_4e81dfac4c7742c7986eb24e6d54a124 [id=b87e43a3dc3c4cd98ee3f44964e8092f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251025 07:52:51.948133 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eced77e6a53447db9fb2df3b670684f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251025 07:52:51.948072 1167 raft_consensus.cc:359] T 0540d6a10cf04fa89756623e9aaaeb68 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.948283 1167 raft_consensus.cc:385] T 0540d6a10cf04fa89756623e9aaaeb68 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:51.948361 1167 raft_consensus.cc:740] T 0540d6a10cf04fa89756623e9aaaeb68 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:51.948659 1167 consensus_queue.cc:260] T 0540d6a10cf04fa89756623e9aaaeb68 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.949095 1169 tablet_bootstrap.cc:492] T 6ce62847e80f48e8807fdbbbdc50803c P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:51.949198 1167 ts_tablet_manager.cc:1434] T 0540d6a10cf04fa89756623e9aaaeb68 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251025 07:52:51.949481 1167 tablet_bootstrap.cc:492] T 0b47d1b0e6a342888135d7e4e5fa1f43 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:51.950999 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 341) took 2052 ms. Trace:
I20251025 07:52:51.951094 2166 rpcz_store.cc:276] 1025 07:52:49.898100 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.898182 (+ 82us) service_pool.cc:225] Handling call
1025 07:52:51.861075 (+1962893us) ts_tablet_manager.cc:1151] Deleting tablet 3c4970c8a491460e8b4b1c29fde83e4c
1025 07:52:51.861083 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.902671 (+ 41588us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.933127 (+ 30456us) tablet_metadata.cc:641] Metadata flushed
1025 07:52:51.950981 (+ 17854us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1962724}
I20251025 07:52:51.951844 1169 tablet_bootstrap.cc:654] T 6ce62847e80f48e8807fdbbbdc50803c P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:51.952212 1167 tablet_bootstrap.cc:654] T 0b47d1b0e6a342888135d7e4e5fa1f43 P 23eeffd3de254ce18db5febfde971c6b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:51.954860 1169 tablet_bootstrap.cc:492] T 6ce62847e80f48e8807fdbbbdc50803c P db42be7ad3c741a0aa427f27e245993b: No bootstrap required, opened a new log
I20251025 07:52:51.955016 1169 ts_tablet_manager.cc:1403] T 6ce62847e80f48e8807fdbbbdc50803c P db42be7ad3c741a0aa427f27e245993b: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20251025 07:52:51.955819 1169 raft_consensus.cc:359] T 6ce62847e80f48e8807fdbbbdc50803c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.956024 1169 raft_consensus.cc:385] T 6ce62847e80f48e8807fdbbbdc50803c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:51.956084 1169 raft_consensus.cc:740] T 6ce62847e80f48e8807fdbbbdc50803c P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db42be7ad3c741a0aa427f27e245993b, State: Initialized, Role: FOLLOWER
I20251025 07:52:51.956292 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 342) took 2057 ms. Trace:
I20251025 07:52:51.956375 2166 rpcz_store.cc:276] 1025 07:52:49.899258 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.899312 (+ 54us) service_pool.cc:225] Handling call
1025 07:52:51.956183 (+2056871us) ts_tablet_manager.cc:1151] Deleting tablet fd265cdbf6a8440393a2b5d8240d1872
1025 07:52:51.956195 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.956274 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2056692}
I20251025 07:52:51.956347 1169 consensus_queue.cc:260] T 6ce62847e80f48e8807fdbbbdc50803c P db42be7ad3c741a0aa427f27e245993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.956856 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 343) took 2056 ms. Trace:
I20251025 07:52:51.956961 2166 rpcz_store.cc:276] 1025 07:52:49.900378 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.900450 (+ 72us) service_pool.cc:225] Handling call
1025 07:52:51.956764 (+2056314us) ts_tablet_manager.cc:1151] Deleting tablet 7c7a9d7a4fb3449bb57eafa83230a839
1025 07:52:51.956773 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.956840 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2056134}
I20251025 07:52:51.957497 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 344) took 2019 ms. Trace:
I20251025 07:52:51.957578 2166 rpcz_store.cc:276] 1025 07:52:49.938541 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.938601 (+ 60us) service_pool.cc:225] Handling call
1025 07:52:51.957357 (+2018756us) ts_tablet_manager.cc:1151] Deleting tablet 943ec703e65f4fd3b0c690e08c0bd226
1025 07:52:51.957369 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.957481 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2018514}
I20251025 07:52:51.957736 1169 ts_tablet_manager.cc:1434] T 6ce62847e80f48e8807fdbbbdc50803c P db42be7ad3c741a0aa427f27e245993b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251025 07:52:51.957958 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 345) took 2017 ms. Trace:
I20251025 07:52:51.958052 2166 rpcz_store.cc:276] 1025 07:52:49.940194 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.940250 (+ 56us) service_pool.cc:225] Handling call
1025 07:52:51.957872 (+2017622us) ts_tablet_manager.cc:1151] Deleting tablet c88c46e022ff41a58b02538f2151e260
1025 07:52:51.957880 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.957944 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2017406}
I20251025 07:52:51.958525 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 346) took 2017 ms. Trace:
I20251025 07:52:51.958607 2166 rpcz_store.cc:276] 1025 07:52:49.940912 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.940960 (+ 48us) service_pool.cc:225] Handling call
1025 07:52:51.958440 (+2017480us) ts_tablet_manager.cc:1151] Deleting tablet fa998b9ca6724c4cb1eb68a3eecb0b11
1025 07:52:51.958448 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.958511 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2017316}
I20251025 07:52:51.958730 1169 tablet_bootstrap.cc:492] T 3b940fa5ae26427dafbcfd26372754b3 P db42be7ad3c741a0aa427f27e245993b: Bootstrap starting.
I20251025 07:52:51.959000 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 347) took 2017 ms. Trace:
I20251025 07:52:51.959084 2166 rpcz_store.cc:276] 1025 07:52:49.941505 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.941554 (+ 49us) service_pool.cc:225] Handling call
1025 07:52:51.958925 (+2017371us) ts_tablet_manager.cc:1151] Deleting tablet 70fe9aa2f58d4abea3b3b71066a29653
1025 07:52:51.958934 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.958987 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2017199}
I20251025 07:52:51.959527 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 348) took 2017 ms. Trace:
I20251025 07:52:51.959614 2166 rpcz_store.cc:276] 1025 07:52:49.942158 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.942200 (+ 42us) service_pool.cc:225] Handling call
1025 07:52:51.959442 (+2017242us) ts_tablet_manager.cc:1151] Deleting tablet f4578ce69a0d4322944e5ce7c8a59831
1025 07:52:51.959450 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.959511 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2017087}
I20251025 07:52:51.960661 1169 tablet_bootstrap.cc:654] T 3b940fa5ae26427dafbcfd26372754b3 P db42be7ad3c741a0aa427f27e245993b: Neither blocks nor log segments found. Creating new log.
I20251025 07:52:51.962112 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c4970c8a491460e8b4b1c29fde83e4c" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"
I20251025 07:52:51.962217 1041 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c4789b9d89f42cb844e6cb595e687db" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b" is_pre_election: true
I20251025 07:52:51.962370 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 349) took 2019 ms. Trace:
I20251025 07:52:51.962443 2166 rpcz_store.cc:276] 1025 07:52:49.942700 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:49.942740 (+ 40us) service_pool.cc:225] Handling call
1025 07:52:51.962265 (+2019525us) ts_tablet_manager.cc:1151] Deleting tablet ff3daed6c9af47e3976b82b4b9a41086
1025 07:52:51.962274 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.962355 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2019339}
I20251025 07:52:51.962998 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 351) took 1942 ms. Trace:
I20251025 07:52:51.963073 2166 rpcz_store.cc:276] 1025 07:52:50.020705 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:50.020761 (+ 56us) service_pool.cc:225] Handling call
1025 07:52:51.962920 (+1942159us) ts_tablet_manager.cc:1151] Deleting tablet e47c4f741f354a19b49777bd06e54764
1025 07:52:51.962927 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.962985 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1941995}
I20251025 07:52:51.963449 2166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58412 (request call id 352) took 1940 ms. Trace:
I20251025 07:52:51.963519 2166 rpcz_store.cc:276] 1025 07:52:50.023127 (+ 0us) service_pool.cc:168] Inserting onto call queue
1025 07:52:50.023173 (+ 46us) service_pool.cc:225] Handling call
1025 07:52:51.963372 (+1940199us) ts_tablet_manager.cc:1151] Deleting tablet f656945454fe4921ab819c772a089eb3
1025 07:52:51.963379 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1025 07:52:51.963436 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1940019}
I20251025 07:52:51.963774 1041 raft_consensus.cc:3060] T 792f32cbd66449ca98813c669b2a261d P db42be7ad3c741a0aa427f27e245993b [term 0 FOLLOWER]: Advancing to term 1
I20251025 07:52:51.964277 1095 heartbeater.cc:499] Master 127.0.104.62:37421 was elected leader, sending a full tablet report...
I20251025 07:52:51.964483 1167 tablet_bootstrap.cc:492] T 0b47d1b0e6a342888135d7e4e5fa1f43 P 23eeffd3de254ce18db5febfde971c6b: No bootstrap required, opened a new log
I20251025 07:52:51.964591 1167 ts_tablet_manager.cc:1403] T 0b47d1b0e6a342888135d7e4e5fa1f43 P 23eeffd3de254ce18db5febfde971c6b: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20251025 07:52:51.965248 1167 raft_consensus.cc:359] T 0b47d1b0e6a342888135d7e4e5fa1f43 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.965440 1167 raft_consensus.cc:385] T 0b47d1b0e6a342888135d7e4e5fa1f43 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251025 07:52:51.965490 1167 raft_consensus.cc:740] T 0b47d1b0e6a342888135d7e4e5fa1f43 P 23eeffd3de254ce18db5febfde971c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eeffd3de254ce18db5febfde971c6b, State: Initialized, Role: FOLLOWER
I20251025 07:52:51.965705 1167 consensus_queue.cc:260] T 0b47d1b0e6a342888135d7e4e5fa1f43 P 23eeffd3de254ce18db5febfde971c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23eeffd3de254ce18db5febfde971c6b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46231 } } peers { permanent_uuid: "db42be7ad3c741a0aa427f27e245993b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45417 } } peers { permanent_uuid: "3aea2869ff814cc7b142012faa274382" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43563 } }
I20251025 07:52:51.966563 1167 ts_tablet_manager.cc:1434] T 0b47d1b0e6a342888135d7e4e5fa1f43 P 23eeffd3de254ce18db5febfde971c6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251025 07:52:51.966821 1167 tablet_bootstrap.cc:492] T 165575a8b5b54d3c9dbcc1cc93044623 P 23eeffd3de254ce18db5febfde971c6b: Bootstrap starting.
I20251025 07:52:51.968055 1041 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e18611cbc1348eca927162fec70e36f" candidate_uuid: "3aea2869ff814cc7b142012faa274382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db42be7ad3c741a0aa427f27e245993b"